In Summer 2020, I worked at MIT SparkLab in the AI Innovation Accelerator (AIIA) search-and-rescue intelligence branch as a lab assistant under Professor Luca Carlone.

The initial objective of my summer research was to leverage MaskRCNN for transfer learning in Kimera (an open-source real-time SLAM software) to generate semantic labels from the RGB images simulated through Flightgoggles. Eventually, the aim was to be able to replace Flightgoggles with real world images and get relatively accurate metric-semantic reconstruction.

This evolved into speed-testing and latency reduction of Kimera, while also testing different neural networks to find one whoose trainig data best aligns it for use on Flightgoggles simulations.

My action items were:

• Setting up Kimera (Kimera-VIO-ROS + Kimera-Semantics + dependencies) and running Kimera on test datasets
• Getting ground truth per-pixel semantic labelling from Flightgoggles
• Testing neural networks (eg MaskRCNN) to generate semantic labels from the RGB image
• Adding throttle nodes and reducing resolution of RGB input into Kimera to reduce the time it takes to get a reconstruction
• Integrating Kimera into the broader RL system- involves generalizing topic names and modifying broadcasting rates to account for different input/output buffers