Helping Hertz’s AD System - Part 1

Anomaly detection in manufacturing and other industrial settings is a common application of computer vision systems. Hertz has started using AI to inspect returned rental cars for damages, scratches, and other anomalies. As shown in the next video, customers started to complain that they were unfairly charged for damages that were either not present when they returned the cars or the damages were so minor that they can be considered as normal wear and tear.

You are called to solve the issue and improve on the credibility of such systems and more specifically you need to test the following hypothesis: explainability signals can be helpful in the reasoning decision that should only charge extra only when the customer did cause a more significant than just a dent or two damage to the car.

MVTec AD Dataset

The MVTec AD (MVTec Anomaly Detection) dataset is widely used in the anomaly detection community, particularly for benchmarking algorithms in computer vision tasks such as unsupervised anomaly detection where the goal is to detect anomalies without prior knowledge of what constitutes a “defect.” The dataset provides labeled data for testing purposes but encourages approaches that do not require extensive labeled training data.

Main Characteristics of MVTec-AD:

  1. Diverse Object Categories: The dataset contains 15 different object and texture categories, including items like bottles, capsules, leather, pillows, and screws. Each category includes both defect-free (normal) samples and samples with various defects.

  2. Types of Defects: Defects in the dataset include structural anomalies like scratches, dents, or missing parts, as well as textural anomalies, such as discoloration or irregular surface patterns. This mix makes the dataset useful for evaluating models on both texture-based and structure-based anomaly detection tasks.

  3. Dataset Size and Variations: Each object category contains around 70-300 images for training and a similarly large number of test images with a variety of defects. The defects range from subtle imperfections to large, visible anomalies, providing a wide range of difficulty levels for detection.

  4. Benchmark Performance: The MVTec-AD dataset has been used to benchmark several state-of-the-art methods especially methods using self-supervised learning techniques, have shown promising results on this dataset.

  5. Class Imbalance: Like many real-world anomaly detection problems, the dataset reflects an inherent class imbalance, with many more normal (non-defective) samples than defective ones. This mirrors real-world industrial settings, where defects are relatively rare but critical to detect.

It is worth noting that the dataset is challenging for several reasons:

  • High Intra-Class Variability: The normal images have a high degree of variability within each class, making it challenging for models to learn a robust representation of “normality.”
  • Subtle Anomalies: Some defects are extremely subtle, making detection difficult for unsupervised models that rely on distinguishing between normal and abnormal instances.
  • Multiple Types of Anomalies: The variety of both structural and textural defects requires models that can adapt to different anomaly characteristics.

The anomalib library

You will use the anomalib library on the MVTec-AD dataset for only the categories that are flat surfaces (tile, leather and grid). The anomalib library provides a simple interface to load the dataset, visualize images, and evaluate anomaly detection models. The library also includes utility functions for data preprocessing, model training, and evaluation.

You will exercise the PatchCore AD method pioneered by Amazon to detect

Results

Report AUROC scores. You need to report AUROC at product category level and the average across all flat surface categories. You can use pretrained models.

Paper

Your grade will depend on how you document your work, visualize the results, and interpret the results - we aim in presenting a tutorial-like paper that can be used by others to understand how to use the anomalib library, evaluate anomaly detection models on the MVTec-AD dataset and obviously how the models achieve that. To give you an example, we use coresets - your report should explain what they are.