In plain words
Modern AI models are often called black boxes. Information goes in, an answer comes out, and the reasoning in between is spread across billions of numbers that no person can read directly.
That is a problem when the answer matters. If a bank's model turns down your loan, you, the bank and the regulator all want to know why. Was it your missed payments, or something unfair like your postcode?
Explainability is the set of methods for answering that. Some show which inputs pushed the decision one way or the other. Others, called interpretability research, open the model up and trace what its internal parts actually do. Both are still far from perfect, especially for very large models.
Everyday example
Asking the doctor why
A doctor says you need a scan. You ask why. She points to three things: your symptoms, your blood test and your family history, and says the blood test weighed most. You can now check those facts and ask for a second opinion. Explainability tries to make an AI answer the same question: which facts, and how much did each count.
Where the example breaks: The doctor's reasons are how she really decided. An AI's explanation is often an approximation produced afterwards by a separate method, so it can be neat and still not match how the model truly worked.
How it actually works
Methods fall into two broad families: explaining a decision from the outside, and understanding the mechanism from the inside.
Feature attribution
Methods like LIME (2016) and SHAP (2017) test how the output changes when inputs are changed or removed, then give each input (each feature) a score for how much it pushed the result. This works with almost any model and is common in banking and healthcare.
Saliency for images and text
For image models, heat maps highlight the pixels that most affected a classification. For language models, similar scores highlight influential words. These are useful hints but can be unstable: small changes can shift the map a lot.
Looking inside: mechanistic interpretability
Researchers try to map what individual parts of a neural network represent. In May 2024, Anthropic reported finding millions of "features" inside a Claude model, including one for the Golden Gate Bridge. Turning it up made the model talk about the bridge constantly, even claiming to be it.
Tracing a whole answer
In March 2025, Anthropic traced the steps inside a model: for example, answering the capital of the state containing Dallas by first representing Texas, then Austin. It also found that a model's written explanation of its reasoning did not always match what happened inside.
Limits
These tools capture only a fraction of what a large model computes, and analysing even a short prompt can take hours of expert work. Explanations can also mislead if they sound convincing but are wrong. Chain-of-thought text from a model is not a guaranteed explanation either.
- Also called
- XAI (explainable AI), interpretability
- Well-known methods
- LIME (2016), SHAP (2017), saliency maps, sparse dictionary learning (2023 onwards)
- Why it matters
- Spotting bias, debugging errors, building trust, and checking alignment in large models
- In India
- "Understandable by Design" is one of the seven principles in India's AI Governance Guidelines (November 2025)
Should high-stakes decisions use black-box models with explanations?
One view
Explain the powerful model
The most accurate models are often complex. Explanation tools let us keep that accuracy while checking decisions, and they keep improving.
The other view
Use simple, readable models instead
Computer scientist Cynthia Rudin argued in 2018 that explaining black boxes for decisions like bail or loans can "perpetuate bad practices", and that models people can read directly often do just as well.
Most likely: It depends on the task. For tabular decisions like credit, simple interpretable models are often competitive. For language and images, complex models have no simple rival, so explanation and inside-the-model research are the main route.
Questions people ask
What is the difference between explainability and interpretability?
They are often used as synonyms. When separated, explainability means giving reasons for a particular output, often after the fact, while interpretability means understanding how the model itself works inside. Mechanistic interpretability is the inside-the-model research.
Why is AI called a black box?
Because its decision is spread across millions or billions of learned numbers (parameters) rather than written rules. Even its makers cannot read off why one answer was chosen without special tools.
Can ChatGPT explain its own answers?
It can write an explanation, and it is often useful. But that text is generated like any other reply, and research shows it does not always match the model's actual internal process. Treat it as a helpful account, not proof.
What does SHAP mean?
SHapley Additive exPlanations, from a 2017 paper. It borrows Shapley values from game theory, a fair way to split a team's reward among players, to split a prediction among the input features that contributed to it.
Further reading
- Mapping the Mind of a Large Language ModelAnthropic, 2024
- Tracing the thoughts of a large language modelAnthropic, 2025
- A Unified Approach to Interpreting Model Predictions (SHAP)arXiv, 2017
- "Why Should I Trust You?": Explaining the Predictions of Any Classifier (LIME)arXiv, 2016
- Stop Explaining Black Box Machine Learning Models for High Stakes DecisionsarXiv, 2018