3D data representations:

  1. Point Cloud(PointNet): – More concretely, given points in a point cloud, the network needs to learn unique features that are invariant to the permutations of this input data, since the ordering of points fed into a network doesn’t impact the underlying geometry. – network should be robust to transformations of the point cloud — rotations, translations and scaling should not impact prediction. – To ensure invariance across input ordering, the key insight behind PointNet is using a simple symmetric function that produces a consistent output for any ordering of the input (examples in this class of functions include addition and multiplication) – So, we need to have a symmetric function ‘f’ that converts input points into k-dim vector f = gamma * g(h(x1),…. h(xn)) where h is hidden MLP conversion of input points g is symmetric function too The max-pooling operations serves as the symmetric function ‘g’, aggregating learned features into a global descriptor for the point cloud. gamma is also a MLP which predicts the output

    – PointNet employs a mini-network, called T-Net for geometric invariance – overall PointNet architecture integrates the vanilla approach and T-Net with multiple MLP layers that create feature representations for the point clouds. – Also enables semantic segmentation of scenes and objects – To accomplish this, the architecture combines the global feature vector from the max pooling symmetric function with per-point features learned after the input data is passed through a few MLPs. By concatenating these two vectors, each point is aware of both its global semantics and its local features, enabling the network to learn additional, more meaningful features that help with segmentation.

1.2. PointNet++: – primary drawbacks is that the architecture fails to capture the underlying local structure within neighborhoods of points —- an idea that is similar to extracting features from increasing receptive field sizes in images using CNNs.

– The basis behind this method is a hierarchical feature learning layer that has three key steps. It (1) samples points to serve as centroids for the local region, (2) groups neighboring points in these local regions based on distance from the centroid, and (3) encodes features for these regions using a mini-PointNet.

1.infinity:

  1. Dynamic Graph CNNs : Using GNNs to operate on point cloud. (2018) – The idea is that PointNet and PointNet++ fail to capture the geometric relationships among individual points because these methods need to maintain invariance to different input permutations. – EdgeConv, an operator that generates unique features across points in the data

  2. SPLATNet: – Convolution operator that can directly operate on point clouds – What’s especially fascinating is that SPLATNet can project features extracted from multi-view images into 3D space, fusing this 2D data with the original point cloud in an end-to-end learnable architecture.

  3. Frustum PointNets: – Fuse RGB images with point cloud – Contributions: – Bounding box recognition by simply swiping through entire point space is intractable (almost), so this paper proposes: – Bounding box from 2D image by convolution or window sliding – extrude a 3D frustum corresponding to the region of the point cloud in which the detected object could reside – Perform search over only this frustum to locate the object

    – instead of performing the typical sliding window classification to during the bounding box search process, design a novel PointNet-based architecture that can directly perform instance segmentation (segmenting the point cloud into individual objects) and bounding box estimation over the entire 3D frustum in one pass, making their method both fast and robust to occlusions and sparsity.

  1. Voxels(Voxnet) – Problems with rotation at first – Lose resolution compared to point clouds since multiple 3d points may be translated to same voxel – Can lead to high memory usage in sparse environment settings
  1. Multi View Representation(Multi view CNN) – Dimensionality reduction can never make actual repr of underlying data distribution: there are losses – You need to first compute full 3D render and then create views: not so efficient.