Object Detection

Publish the camera feed into a topic

Use your laptop webcam or a pluggable USB camera ro publish the camera feed into a topic. The following command will publish the camera feed into the /camera/image_raw topic.

You can watch this video to learn how to do so and you can also consult the description in this book to understand the publish subscribe concepts.

Create assets and embed them in the maze

All objects should be placed on a low height bench so that the robot can see them. The benches should touch the wall and you can create 3-4 benches with 3D objects from the COCO classes that are suitable for a bench. Each bench should have a different set of objects.

Object detection for the simulated camera sensor

You can enable the camera sensor that is included in the robot and repeat the exercise that you did with the USB camera. You need to make the robot navigate the maze in such a way that it can see the objects on the benches. Write a custom ROS node that subscribes to the camera feed and detects the objects using pytorch. If you have a GPU ensure that you enable it. The detector should be pretrained in the COCO dataset and we advise you to follow the Pytorch Object Detection Tutorial to get started. Most people borrow the detectron2 mask-rcnn model and fine-tune it on their own dataset. No finetuning is needed here provided that the objects are detectable.

Vector database and Semantic Localization

When an object is detected, its representation should be extracted from its bounding box and stored in the Qdrant database. Metadata should also be stored in a Qdrant database in such a way that the database can be queried and respond with the approximate location of the object in the maze.