Assignment 2b
Points:
- Anomaly Detection: 50 points equally distributed between the tasks quoted. Please note that the Similarity Search only applies to the specific sections - CS370-Honors and CS-GY-6613.
Anomaly Detection using the MVTec-AD Dataset
The MVTec AD (MVTec Anomaly Detection) dataset is widely used in the anomaly detection community, particularly for benchmarking algorithms in computer vision tasks such as unsupervised anomaly detection where the goal is to detect anomalies without prior knowledge of what constitutes a “defect.” The dataset provides labeled data for testing purposes but encourages approaches that do not require extensive labeled training data.
Main Characteristics of MVTec-AD:
Diverse Object Categories: The dataset contains 15 different object and texture categories, including items like bottles, capsules, leather, pillows, and screws. Each category includes both defect-free (normal) samples and samples with various defects.
Types of Defects: Defects in the dataset include structural anomalies like scratches, dents, or missing parts, as well as textural anomalies, such as discoloration or irregular surface patterns. This mix makes the dataset useful for evaluating models on both texture-based and structure-based anomaly detection tasks.
Dataset Size and Variations: Each object category contains around 70-300 images for training and a similarly large number of test images with a variety of defects. The defects range from subtle imperfections to large, visible anomalies, providing a wide range of difficulty levels for detection.
Benchmark Performance: The MVTec-AD dataset has been used to benchmark several state-of-the-art methods especially methods using self-supervised learning techniques, have shown promising results on this dataset.
Class Imbalance: Like many real-world anomaly detection problems, the dataset reflects an inherent class imbalance, with many more normal (non-defective) samples than defective ones. This mirrors real-world industrial settings, where defects are relatively rare but critical to detect.
It is worth noting that the dataset is challenging for several reasons:
- High Intra-Class Variability: The normal images have a high degree of variability within each class, making it challenging for models to learn a robust representation of “normality.”
- Subtle Anomalies: Some defects are extremely subtle, making detection difficult for unsupervised models that rely on distinguishing between normal and abnormal instances.
- Multiple Types of Anomalies: The variety of both structural and textural defects requires models that can adapt to different anomaly characteristics.
The anomalib library
You will use the anomalib library on the MVTec-AD dataset for only the categories that are flat surfaces (tile, leather and grid). The anomalib library provides a simple interface to load the dataset, visualize images, and evaluate anomaly detection models. The library also includes utility functions for data preprocessing, model training, and evaluation.
You will exercise two models on the dataset:
Results
Report AUROC scores for both models on the MVTec-AD dataset. You need to report AUROC at product category level and the average across categories. You can use pretrained models.
Paper
Your grade will depend on how you document your work, visualize the results, and interpret the results - we aim in presenting a tutorial-like paper that can be used by others to understand how to use the anomalib library, evaluate anomaly detection models on the MVTec-AD dataset and obviously how the models achieve that. To give you an example, one of the algorithms is using coresets - your report should explain what they are.
Similarity Search
You will extract features from the models and use them to perform similarity search on the anomalous cases your model predicts. You will use the qdrant vector database to store the features. Your code should accept an image and return the top 5 similar images. If an image is an anomalous case, the similar images should also be anomalous cases.