What are Discriminative and Generative Models in AI?

In this article we aim to discuss two essential types of machine learning models - discriminative and generative algorithms. To simplify this complex topic, we'll start with a story. Then, we'll get to the details and examine these models with some popular algorithms as examples.

Let's get started!

Unfolding The Tale of Two Aliens

Imagine two alien visitors on Earth who have never seen apples and bananas before. They're both intrigued by these earthly fruits and want to learn how to tell them apart.

The first alien chooses to understand these fruits by sketching them, carefully observing their shape, color, and texture. It creates visual representations, or models, of each fruit to use as a reference. This approach mirrors what we call a 'Generative Algorithm' in machine learning.

The second alien, however, adopts a different strategy. Instead of drawing, it starts comparing the features of the fruits. It notices that apples are generally round and red, while bananas are long and yellow. When presented with a new fruit, it checks which fruit features the new one more closely resembles and identifies it based on this comparison. This approach resonates with what we term a 'Discriminative Algorithm' in machine learning.

Discriminative vs Generative Classification

Just like the first alien, a 'Generative Classification' model learns to generate a representation of each class. It's akin to learning how an apple or a banana looks and using that knowledge to identify future instances.

Conversely, a 'Discriminative Classification' model, like our second alien, learns to distinguish between classes based on their features. Instead of learning what an apple or banana looks like, it learns the differences between them and uses these differences to classify new fruits.

To add context, let's consider two specific algorithms: Logistic Regression for 'Discriminative Classification' and the Naive Bayes algorithm for 'Generative Classification'.

Logistic Regression vs Naive Bayes:
In a Logistic Regression model, the algorithm uses features (like color, length, and weight of fruits) to learn patterns and make decisions. It might learn that if a fruit is yellow and longer than five inches, it's likely to be a banana.

On the other hand, the Naive Bayes algorithm, a generative classification model, strives to understand the distribution of each class in the feature space. It learns how each class— apples and bananas—distributes in the data, rather than just identifying their differences.

Common Examples of Discriminative and Generative Algorithms

Here are some popular examples of both types:
Discriminative Algorithms:

• Logistic Regression
• Support Vector Machines (SVM)
• Decision Trees
• Random Forest
• Gradient Boosting Machines (GBM)

Generative Algorithms:

• Naive Bayes
• Gaussian Mixture Models (GMM)
• Hidden Markov Models (HMM)
• Latent Dirichlet Allocation (LDA)
• Generative Adversarial Networks (GAN)

Important Training and Data Considerations

Let's now answer some important questions about generative and discriminative models. Understanding the differences between these two models can help you appreciate the capabilities and flexibility of generative AI systems. It guides you in deciding which type of AI system is best suited to your project or use case.

Which model needs more data for training?

Discriminative models generally require more data for training as they need a rich set of examples to effectively identify differences between classes.

Which one gets affected by missing data?

Both models can be affected, but generative models might be more sensitive because they're trying to capture the overall distribution of the data. Any missing information could skew this distribution.

Which model gets impacted by outliers?

Outliers can affect both models, but discriminative models might be more susceptible because these models focus on boundaries between classes.

Which requires more computation?

In terms of computations, generative models often require more because they involve estimating the distribution of data.

Which one tends to overfit?

Both models can overfit, but discriminative models are generally more prone to it.

By understanding these differences, you can better appreciate the potential of generative AI systems and make informed decisions when choosing an AI system for your project or use case.