Faster RCNN

Region Proposal:

  • Anchors: Different size bounding box of different aspect ratio and sizes

  • Task to be done by RPN:

  • Does this anchor contain a relevant object?

  • How would we adjust this anchor to better fit the relevant object?

  • It’s important to understand that even though anchors are defined based on the convolutional feature map, the final anchors reference the original image.

  • Mathematically, if the image was w×hw × hw×h, the feature map will end up w/r×h/rw/r × h/rw/r×h/r where rrr is called subsampling ratio.

  • If we define one anchor per spatial position of the feature map, the final image will end up with a bunch of anchors separated by r pixels.

  • In order to choose the set of anchors we usually define a set of sizes (e.g. 64px, 128px, 256px) and a set of ratios between width and height of boxes (e.g. 0.5, 1, 1.5) and use all the possible combinations of sizes and ratios

  • The RPN uses all the anchors selected for the mini batch to calculate the classification loss using binary cross entropy.

  • Then, it uses only those minibatch anchors marked as foreground to calculate the regression loss. For calculating the targets for the regression,

  • we use the foreground anchor and the closest ground truth object and calculate the correct Δ\DeltaΔ needed to transform the anchor into the object.

  • Two different output for anchors:

    • Objectness score: Probability that the anchor is an object

    • Bounding box regression: For correcting bounding box to fully engulf the object.

RPN architecture:

Take as input feature map from base layer

2 output heads for objectness and bounding box regression

For the classification layer, we output two predictions per anchor: Background vs Foreground logit

For regression: 4 delta values

  • Network:

    Convolutional layer with 512 channels and 3x3 kernel size, pad 1 and then we have two parallel convolution layers using a 1*1 kernel, whose number of channels depends on the number of anchors per point.

  • Training:

    Categorization of anchors:

    Those that overlap a ground-truth object with an Intersection over Union (IoU) bigger than 0.5 are considered “foreground”

    those that don’t overlap any ground truth object or have less than 0.1 IoU with ground-truth objects are considered “background”.

    • Random sampling:

      From all anchors to form a mini batch of size 256 making balance between foreground and backgnd

    • Binary cross entropy:

      The RPN uses all the anchors selected for the mini batch to calculate the classification loss using binary cross entropy.

      Regression loss:

      Then, it uses only those minibatch anchors marked as foreground to calculate the regression loss.

    • Regularization Used:

      Smooth L1 loss

      Smooth L1 is basically L1, but when the L1 error is small enough, defined by a certain σ\sigmaσ, the error is considered almost correct and the loss diminishes at a faster rate.

      • Problem:

        Maintaining a perfect ratio between foreground and background in a batch size due to lesser foreground anchors

      • Solution:

        Turn to the anchors with biggest IOU to ground truth boxes.

  • Post Processing:

    NMS(Non maximum suppression)

    Duplicate proposal regions over same object need to be removed.

    NMS takes the list of proposals sorted by score and iterates over the sorted list, discarding those proposals that have an IoU larger than some predefined threshold with a proposal that has a higher score.

    It means similar proposals with very high degree of similarity are removed using IOU.

    Too low and you may end up missing proposals for objects; too high and you could end up with too many proposals for the same object. A value commonly used is 0.6.

  • How many proposals to use then?

    Based on paper, after NMS, let N=2000 but sometimes even 50 will work fine.

  • Region of Interest Pooling:

    Calculating category for each region proposals

    can be done by passing the image patch of region after scaling or crop and pass through a pre trained base network.

    But it is inefficient and slow.

    Faster RCNN tries to mitigate using existing feature convolution map.

    This is done by extracting fixed-sized feature maps for each proposal using region of interest pooling.

    We extract features for each of those proposals (via RoI Pooling) and use these features for classification

RCNN module:

Classify the content in the bounding box (or discard it, using “background” as a label).

Adjust the bounding box coordinates (so it better fits the object).

Makes sure the regions are converted to same shapes using ROI pooling layer.

Another method than ROI pooling to make conv feature maps the same sized:

Crop the convolutional feature map using each proposal and then resize each crop to a fixed sized 14×14×convdepth using interpolation (usually bilinear).

After cropping, use maxpooling with 2* 2 kernel to get final 7* 7 * convdepth feature map for each proposal.

Full Training:

The four different losses are combined using a weighted sum. Two for RPN and Two for RCNN.

Also there is the base network but we can finetune it or not.

The parts were trained independently and merged the trained wts before full final training.

Joint end to end training was found to have better results.

This is because we may want to give classification losses more weight relative to regression ones, or maybe give R-CNN losses more power over the RPNs’.