Discriminative vs Generative Classification Methods

There are three broad classes of methods for determining the parameters \(\mathbf{w}\) of a linear classifier cite

  1. Discriminative Models, which form a discriminant function that maps directly test data \(\mathbf{x}\) to classes \(\mathcal{C}_k\). In this case, probabilities play no role. Examples include the Perceptron and Support Vector Machines (SVMs).

  2. Probabilistic Discrimitative Models, First solve the inference problem of determining the posterior class probabilities \(p(\mathcal{C}_k|\mathbf{x})\) and then subsequently assign each new \(\mathbf{x}\) to one of the classes. Approaches that model the posterior probabilities directly are called discriminative models. Examples of discriminative training of linear classifiers include:

    • Logistic regression—maximum likelihood estimation of \(\mathbf{w}\) assuming that the observed training set was generated by a binomial model that depends on the output of the classifier.

  3. Probabilistic Generative Models, which infer the posterior \(p(\mathcal{C}_k|\mathbf{x})\) using Bayessian approach and we therefore generate the class-conditional density \(p(\mathbf{x}|\mathcal{C}_k)\) and the prior \(p(\mathcal{C}_k)\). Examples of such algorithms include:

    • Linear Discriminant Analysis (or Fisher’s linear discriminant) (LDA)—assumes Gaussian conditional density models
    • Naive Bayes classifier with multinomial or multivariate Bernoulli event models.