Skip to main content

Model representation and approach

Instruction and application
Complete

Another fundamental dimension for classifying machine learning models is their underlying approach to learning from data. At a high level, models can be categorised as either generative or discriminative, each representing a fundamentally different perspective on the learning problem.

Generative models​

Generative models learn the joint probability distribution of the input features and the target variable. In essence, these models attempt to understand how the data is generated, including the relationships between all variables in the system. Generative models provide deeper insights into the data structure.

Generative models illustration

Key characteristics of generative models​

  • Data generation capability: These models can generate new, synthetic data points that resemble the training data.
  • Class-conditional learning: They learn the distribution of features for each class separately.
  • Prior probability modelling: They explicitly model the prior probability of each class.
  • Handling missing data: They can naturally handle missing features by marginalising over the unknown values.When should I use a generative model? Choose a generative model when you need to understand the underlying data distribution, generate synthetic examples, handle missing data naturally or work with limited labelled data but abundant unlabelled data.

Credit scoring with generative models​

In the financial industry, generative models can be used to understand the distribution of financial behaviours among different customer segments. For each credit score category (excellent, good, fair, poor), the model learns the typical distribution of features like income, debt-to-income ratio and payment history.

When a new applicant comes in, the model can determine which credit score category the applicant most likely belongs to based on how well their profile matches the learnt distributions.

Discriminative models​

Discriminative models, in contrast, learn the conditional probability, or the probability of the output given the input. These models focus exclusively on the decision boundary between classes rather than modelling how the data was generated. Discriminative models often achieve higher predictive accuracy.

Discriminative models illustration

Key characteristics of discriminative models​

  • Decision boundary focus: They concentrate on finding the optimal boundary between classes.
  • Conditional probability modelling: They model how outputs depend on inputs without modelling the input distribution itself.
  • Efficiency: They often require less data to achieve good performance since they solve a more direct problem.
  • Performance: They typically achieve better predictive accuracy when sufficient labelled data is available.When should I use a discriminative model? Choose a discriminative model when prediction accuracy is the primary goal, you have abundant labelled training data or you're more interested in the decision boundary than understanding the data generation process.

Credit scoring with discriminative models​

Let's return to our credit scoring example — this time, with a discriminative model. A discriminative model can be used to directly predict credit scores from customer features. Unlike the generative approach, this model doesn't learn what a ‘typical’ customer in each category looks like. Instead, it learns the precise boundaries that separate customers of different creditworthiness.

The model excels at accurately classifying new applicants into the correct credit score category, but it doesn't provide insights into the underlying distribution of customer profiles within each category.