In statistical classification (also the last layer of neural network, also potentially clustering), two main approaches are called the generative approach and the discriminative approach.

Definition

Two major types can be distinguished:

• A generative model is a statistical model of the joint probability distribution $p(x, y|\theta)$ on given observable variable $x$ and target variable $y$;
  • Standard examples of linear classifiers, which are a fundamental category of machine learning models, include generative classifiers and discriminative models. Generative classifiers encompass algorithms like the Naive Bayes Classifier and Gaussian Discriminant Analysis (GDA) along with its variations such as Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA). These methods are primarily focused on modeling the probability distributions of the different classes in the dataset and make predictions based on these learned distributions. Generative classifiers are particularly useful when the underlying data generation process is known or can be modeled effectively.
  • A generative model for images might capture correlations like "things that look like boats are probably going to appear near things that look like water" and "eyes are unlikely to appear on foreheads." These are very complicated distributions.
• A discriminative model is a model of the conditional probability $P(y\mid x, \theta)$ of the target $y$, given an observation $x$; and Classifiers computed without using a probability model like Support Vector Machine (SVM) are also referred to loosely as "discriminative".
  • On the other hand, discriminative models are another category of linear classifiers. Examples of discriminative models include Linear Regression, Logistic Regression, and Support Vector Machines (SVM). These models concentrate on directly learning the decision boundary that separates different classes in the data. Unlike generative classifiers, discriminative models do not explicitly model the underlying data distributions but rather aim to find the optimal decision boundary that best separates the data points of various classes. Each of these linear classifiers has its unique characteristics and applications, making them essential tools in various machine learning and data classification tasks.
  • In contrast, a discriminative model might learn the difference between "sailboat" or "not sailboat" by just looking for a few tell-tale patterns. It could ignore many of the correlations that the generative model must get right.