Classifying rare events

Baseline (20 points)

You will start with this notebook and run it (you need GPU acceleration) to get baseline results for the train and test datasets after you address overfitting.

  1. Plot the ROC curves
  2. Plot the Recall-Precision curves
  3. Confusion matrix for 50% threshold

Use Optuna to find the best hyperparameters for the baseline model.

Please note that you can use Pytorch for this and subsequent steps of this assignment. You need to recode the model in Pytorch though, however it is typically not very time consuming.

Cause rare events (5 points)

Produce another dataset with the same number of examples but with the rare class being 10% and 1% of the total. You can do this by sampling from the original dataset and keeping only 10% of the rare class (cats) and 100% of the common class (dogs).

Store such datasets in the data directory.

Re-train and re-evaluate (15 points)

Re-train the model on the rare class datasets and evaluate it by reoplicating the results of the baseline case above. Comment on your observations as to what is the performance of the rare class.

Improve (15 points)

Improve the performance of the model on the rare class by using two of the techniques outlined in section 3 of this paper.

What you will submit

You can submit the rare event improvement as a separate notebook. Both must exists in your private Github repo we asked you to create for all your assignments under a suitably chosen directory name that we can easily identify as this assignment.