Reusing deep nets features

Recent studies have pointed that using the features computed with deep nets in machine learning algorithms like Support Vector Machines or Random Forests, offers state of the art results. This section provides an insight to classification using classic Machine Learning techniques on deep learning features. Here deep learning is used merely as feature engineering rather that sole classifier.

Principle¶

We will use algorithms that are already implemented in the Orfeo ToolBox with features coming from our deep net. The whole workflow is summarized in the following figure.

flowchart LR

img((Input image)) --> TensorflowModelServe
dn((Deep net)) -.-> TensorflowModelServe
tt((Terrain truth)) --> TrainImageClassifier
TensorflowModelServe --> TrainImageClassifier -.-> c([Random Forest Classifier])
c -.-> ImageClassifier --> map((Land-cover map))
TensorflowModelServe -- "Deep net features" --> ImageClassifier

In previous sections, we have described the model guts. In particular, before computing the softmax on the last neurons, our model computes a number of features, which are named feats.

This enables us to use the TensorflowModelServe to produce the feats output tensor for every pixel of the image, and the resulting pixels are then used as features for training a Random Forest (with the TrainImageClassifier application). Once the RF rule is produced, it can be used to classify the entire image, from the deep features computed on each pixels.

OTB machine learning framework¶

In OTB, composite applications are multiple applications connected together. Typically, one application input can come from another application output, instead of being read from file. The major benefit is that temporary files read and write are avoided, which (i) saves some disk space, and (ii) helps reducing processing time when I/O is a bottleneck. In OTB, the streaming mechanism enables to process large images, in splitting the requested parts of the images in multiple regions. Composite applications that involve applications with streaming support, also benefit from this mechanism.

In there is currently two experimental composite applications for reusing the features from a deep net.

• TrainClassifierFromDeepFeatures, which reuses our model serving application (TensorflowModelServe) as input of the TrainImagesClassifier application to train a machine learning algorithm based on the features of the deep net given in the first application.

• ImageClassifierFromDeepFeatures uses TensorflowModelServe as input of the ImagesClassifier application to classify the input image.

Since the classifier is based on the features computed from the network, the deep net used in ImageClassifierFromDeepFeatures must be the same as the deep net used in TrainClassifierFromDeepFeatures to generate the classification rule! If not, the classifier would perform the image classification task using the features vectors coming from the wrong model, and the classification would be irrelevant.

This exercise will require an slightly more important amount of computing power, because the whole input image will be processed with the deep net twice. The first time: for the classifier training (using the TrainClassifierFromDeepFeatures application). The second time: during the classification map generation (using the ImageClassifierFromDeepFeatures application).

Training a random forest classifier¶

Our goal is to train a random forest classifier that performs on the features of the deep net.

flowchart LR

img((Input image)) --> TensorflowModelServe
dn((Deep net)) -.-> TensorflowModelServe
tt((Terrain truth)) --> TrainImageClassifier
subgraph TrainClassifierFromDeepFeatures
  TensorflowModelServe --> TrainImageClassifier
end
TrainImageClassifier -.-> c([Random Forest Classifier])

import argparse
import pyotb

parser = argparse.ArgumentParser(
    description="Train a classifier from deep net features"
)
parser.add_argument("--savedmodel", required=True, help="savedmodel directory")
params = parser.parse_args()


pyotb.TrainClassifierFromDeepFeatures(
    source1_il="/data/s2_tokyo_10m.tif",
    source1_rfieldx=16,
    source1_rfieldy=16,
    source1_placeholder="input",
    model_dir=params.savedmodel,
    model_fullyconv=True,
    output_names="features",
    vd="/data/pos_a.geojson",
    valid="/data/pos_b.geojson",
    sample_vfn="class",
    sample_bm=0,
    classifier="rf",
    optim_tilesizex=10980,  # Force the model to be applied on a rectangular
    optim_tilesizey=256,  # region, like expected by the classifier application  out=out,
    ram=4096,
    out="/data/randomforest_from_deep_model.yaml",
)

Once the training has been finished, you should have a .yaml file that serializes the random forest classification rule.

Classification from deep net features¶

We use the ImageClassifierFromDeepFeatures to generate the entire classification map. This composite application first computes the features from the deep net, them applies the previously trained classifier over them to generate the land-cover map.

flowchart LR

img((Input image)) --> TensorflowModelServe
dn((Deep net)) -.-> TensorflowModelServe
c([Random Forest Classifier]) -.-> ImageClassifier --> map((Land-cover map))
subgraph ImageClassifierFromDeepFeatures
  TensorflowModelServe -- "Deep net features" --> ImageClassifier
end

import argparse
import pyotb

parser = argparse.ArgumentParser(
    description="Train a classifier from deep net features"
)
parser.add_argument("--savedmodel", required=True, help="savedmodel directory")
params = parser.parse_args()

# Generate a map using the Random Forest classifier
# performing on the features coming from the deep net
infer = pyotb.ImageClassifierFromDeepFeatures(
    source1_il="/data/s2_tokyo_10m.tif",
    source1_rfieldx=16,
    source1_rfieldy=16,
    source1_placeholder="input",
    deepmodel_dir=params.savedmodel,
    deepmodel_fullyconv=True,
    output_names="features",
    model="/data/randomforest_from_deep_model.yaml",
)

infer.write(
    "/data/map_rf_from_feats.tif",
    pixel_type="uint8",
    ext_fname="box=4000:4000:1000:1000",
)