Explain Model Predictions with SHAP

Machine learning models often act like black boxes. They make predictions, but they rarely tell you why. SHAP changes that.

SHAP stands for SHapley Additive exPlanations. It is a game theory method that assigns each feature a fair share of the prediction. This article shows you how to use SHAP in Python, step by step.

Why Model Explainability Matters

A model that scores well on a test set can still be wrong in the real world. It may learn spurious patterns. It may rely on a feature that should be ignored.

Explainability helps you catch these problems. It also builds trust with stakeholders, supports debugging, and can be required by regulations.

SHAP gives both global and local explanations. Global views show which features matter overall. Local views show why one specific prediction was made.

How SHAP Works

SHAP borrows the idea of Shapley values from cooperative game theory. Each feature is treated as a player. The prediction is the payout.

The method measures how much each feature contributes by comparing predictions with and without that feature, across many feature combinations.

The result is a set of additive values. The base value plus all SHAP values equals the model output.

prediction = base_value + sum(shap_values)

This additivity makes SHAP easy to reason about. Positive values push the prediction up. Negative values push it down.

Installing SHAP

SHAP is a pure Python package. Install it with pip.

pip install shap

You will also need a model and data. Scikit-learn is a good starting point. If you plan to serve explanations behind an API, you might later wire this up alongside OpenAI function calling in Python so an assistant can call your explanation endpoint.

Training a Simple Model

Let us train a small model on the built-in breast cancer dataset. It is a binary classification problem with 30 features.

import shap
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

# Load data
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train a random forest
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

The model is now trained. Next, we build an explainer.

Creating a SHAP Explainer

SHAP offers several explainers. The right one depends on your model type.

shap.TreeExplainer is fast and exact for tree-based models like random forests and gradient boosting. For other models, use shap.Explainer, which picks a sensible default.

# TreeExplainer is optimized for tree models
explainer = shap.TreeExplainer(model)

# Compute SHAP values for the test set
shap_values = explainer.shap_values(X_test)

For a binary classifier, shap_values may come back as a list with one array per class. Select the positive class to interpret.

# For binary output, take the positive class
shap_values_pos = shap_values[1]
print(shap_values_pos.shape)

(114, 30)

The shape tells us there are 114 test samples and 30 features. Each cell is one feature's contribution to one prediction.

Explaining a Single Prediction

Local explanations answer the question: why did the model make this prediction?

shap.force_plot draws a single prediction as a tug-of-war. Red features push the score higher. Blue features push it lower.

# Explain the first test sample
shap.initjs()
shap.force_plot(
    explainer.expected_value[1],
    shap_values_pos[0],
    X_test.iloc[0]
)

In a notebook, this renders an interactive chart. In a report, use matplotlib=True to get a static image.

Local explanations are powerful for customer-facing decisions. You can tell a user exactly which factors moved their score.

Global Feature Importance

Global explanations answer a different question: which features matter most across all predictions?

shap.summary_plot gives you two views. The bar version ranks features by mean absolute SHAP value. The dot version also shows direction and spread.

# Bar chart of mean absolute SHAP values
shap.summary_plot(shap_values_pos, X_test, plot_type="bar")

# Beeswarm plot with direction and spread
shap.summary_plot(shap_values_pos, X_test)

The beeswarm plot is especially useful. Each dot is one prediction. Color shows the feature value. Horizontal position shows the impact.

You can quickly see whether high values of a feature raise or lower the prediction. This is often the first insight a data team needs.

Dependence Plots

A dependence plot shows how one feature affects predictions across the dataset.

shap.dependence_plot plots the feature value against its SHAP value. It also colors points by an interacting feature.

# How does 'worst radius' affect predictions?
shap.dependence_plot(
    "worst radius",
    shap_values_pos,
    X_test
)

Look for trends, thresholds, and clusters. A sharp jump often signals a decision boundary the model learned.

These plots are great for feature engineering. They reveal where a simple transform might help.

Working with Text and Tabular Pipelines

SHAP is not limited to numeric tabular data. It also supports text models, image models, and pipelines.

If your input is raw text, you still need to clean and split it first. A common approach is to use tokenization with NLTK in Python before feeding text into a classifier you later explain.

For production systems, explanations often travel as serialized messages. Teams frequently define their schema with compiled .proto files in Python so services agree on the payload format.

This keeps the explanation service decoupled from the model service. Both sides stay simple and testable.

Performance Tips

SHAP can be slow on large datasets. Use these tips to keep it fast.

First, explain a sample, not the whole dataset. A few hundred rows usually show the same patterns.

Second, prefer TreeExplainer for tree models. It is far faster than model-agnostic methods.

Third, cache the explainer. Building it once and reusing it saves time in repeated calls.

# Explain a sample for speed
sample = X_test.sample(100, random_state=42)
shap_values_sample = explainer.shap_values(sample)

Common Pitfalls

Be careful with correlated features. SHAP splits credit between them, which can look confusing.

Also check the class index. For binary models, using the wrong class flips all signs.

Finally, do not treat SHAP values as causal effects. They explain the model, not the world.

A feature can have a large SHAP value and still have no causal link to the outcome. Keep that distinction clear when you report results.

Putting It All Together

A practical workflow looks like this. Train the model. Build the explainer. Check global importance. Inspect a few local predictions. Then share the plots with stakeholders.

This workflow is quick and repeatable. It fits well into a notebook or a scheduled report.

If you later move explanations into a data platform, you may log results to a database. For example, teams that query Oracle Database from Python often store per-prediction SHAP values for audits.

Conclusion

SHAP turns black-box predictions into clear, additive explanations. It supports both global and local views. It works with many model types.

Start with TreeExplainer if you use tree models. Use summary plots for the big picture. Use force plots for individual cases.

Keep an eye on performance and correlation. And always remember that SHAP explains the model, not causality.

With a few lines of code, you can make your models more transparent, more trustworthy, and easier to debug.