Segment Anything Model Finetuning for Remote Sensing Applications#
In this project you will be asked to segment sidewalks from satellite imagery. The importance of remote sensing and its associated job market is growing rapidly. Remote sensing powers, weather prediction, agriculture, urban planning, defense and many other applications.
Lately, we have observed the advent of very large foundational models in NLP applications - sometimes these models result in hundreds of billions of parameters - for example the GPT-3 model that has 175 billion parameters. Can we do the same for computer vision applications ? An example of a positive answer to this problem is the Segmenting Anything Model (SAM).
Background#
Read and experiment with the SAM implementation and the SAM paper.
Milestone 1: Get to know Geographical Information Systems (GIS) and set up your development environment#
See this tutorial that is best to be followed in a desktop installation of QGIS.
Use the example docker container if you have access to a local NVIDIA GPU. Note that you need to do the project in Pytorch and you need to edit the devcontainer.json file to point to the right container for PyTorch.
In addition to python scripts you will also need python notebooks to experiment and visualize the SAM model outputs as well as geospatial data. The suggested container ships with Jupyter and if you use Google Colab you are obviously using notebooks.
Milestone 2: Replicate SAM implementation for satellite imagery#
The SAM implementation for remote sensing applications is based on this package. Go over the following video and replicate the workflow in your environment for the imagery used by the author.
The notebook based environment makes it very easy to experiment with the SAM model and the workflow.
This desktop QGIS plugin is also available to help you experiment with the SAM model.
Milestone 3: Finetune the SAM model for the sidewalks dataset#
Use the sidewalk dataset (this is currently being prepared and will be released soon) and finetune the SAM model for this dataset. Follow these instructions to do so.
Use loss functions and segementation quality metrics that are suitable for roads and sidewalks. You can see this reference for guidance. Avoid going into a rabbit hole of developing your own metrics - there are many metrics that are suitable for this task and you just need to be aware of them and use them. Note that sidewalks are many times more difficult to segment than roads due to their narrow width and the fact that they are often occluded by trees and other objects.
These include:
Milesone 4: Hugging Face App and Video Demo#
Using the reacticve app framework Shiny for Python and the Hugging Face Shiny spaces to create a web app that allows the user to upload an image of a tile and get the sidewalk segmentation. You can use this dashboard as a starting point.
Produce a 2min video showcasing the problem statement and your demo. The video can be uploaded to your Youtube channel or linked as an mp4 file in your Github repo.