What a missing input is worth

A model looks at a house and says it is worth 180. It was given three facts — the house is large, it has a garden, it has many bedrooms — and it combined them in some way to produce the number. Somebody reading the number wants to know which of the facts it came from: how much of the 180 is the size, how much the garden, how much the bedrooms.

That is a division problem, and there is a rule for division problems. Make each fact a player, reveal the facts to the model one at a time in some order, record how much each revelation moves the prediction, and average over every order. Four conditions — the credits add up to the total, interchangeable inputs get equal credit, an input that never matters gets none, and credits for a sum of models are the sums of the credits — leave no other rule. Since Erik Štrumbelj and Igor Kononenko proposed it in 2010, and Scott Lundberg and Su-In Lee popularised it in 2017 under the name SHAP, this has become one of the most widely used ways of explaining what a model did.

But the rule needs a game: a number for every coalition. Here a coalition is a set of facts the model has been told, and a model given only some of its inputs does not produce a number at all — it needs all three. What a coalition is worth depends on what the model is assumed to see in place of the facts it was not told, and that choice is not made by the four conditions. It is made before they apply, and it changes the answer completely.

The houses and the model #

The model is deliberately simple. It adds 60 for a large house and another 20 if a large house also has a garden, and it never reads the number of bedrooms. That is the one thing about it anybody could say with certainty, and an explanation that gets it wrong is getting wrong the only thing that can be checked.

The data it is used on is ten houses of five kinds. In nine of the ten, a house has many bedrooms exactly when it is large — which is what bedrooms and size usually do. The house being explained is large, has a garden and has many bedrooms, so the model prices it at 180. The question is how to divide that 180, or rather the part of it that needs explaining.

Every input is 0 or 1 and every group of houses is counted, so that each value below is an exact fraction and each table can be checked by hand. Nothing about the argument depends on that smallness. A model with a hundred continuous inputs faces the same three choices at every coalition; it simply cannot print the tables that show which choice was made.

Leaving an input out by setting it to zero #

The simplest answer to “what does the model say without the garden” is: whatever it says with the garden set to 0. Each coalition is valued by keeping its inputs at the house’s own values and setting every other input to a fixed reference, here 0 for all three. With nothing known the model sees a small house with no garden and few bedrooms and says 100; with everything known it says 180.

Walking the inputs in in each of the six orders, size always adds 60 when it arrives without the garden already present and 80 when the garden is already there, the garden adds 0 or 20 depending on whether size has arrived, and the bedrooms always add 0. Averaging over orders gives 70 to the size, 10 to the garden and 0 to the bedrooms, which add up to the 80 between the reference and the prediction.

That is a coherent answer to a coherent question — how does this house differ from a small house with no garden? — but it depends entirely on the reference. Choose a different one and every credit moves, and the total being explained moves with it: set the reference to a large house with no garden and the size gets nothing, because it no longer differs from the reference.

Leaving an input out by drawing it from the data #

The second answer takes the reference from the houses themselves. A coalition’s value is the model’s average price when its inputs are fixed at the house’s values and the others are taken from each of the ten houses in turn. With nothing known that is the average price, 136; with everything known it is 180.

The table lists all eight coalitions. Knowing the size alone lifts the expected price from 136 to 170, because half the houses have gardens and a large house earns the garden bonus half the time. Knowing the garden alone lifts it only to 140, because half the houses are small and the garden does nothing for them. The bedrooms never move anything, since the model never reads them and the unknown inputs are drawn without regard to them.

The average over orders gives 37 to the size, 7 to the garden and 0 to the bedrooms. The bedrooms get nothing for exactly the reason the dummy condition exists: in every coalition, adding the bedrooms changes the value by nothing. This is the answer most implementations compute when they are handed a sample of background data, and it is sometimes called interventional, because it treats each unknown input as something the analyst sets, independently of the others.

Leaving an input out by drawing from matching houses #

The third answer is the one that sounds most like what the question means. What does the model say if all that is known is that the house has many bedrooms? Then it is presumably one of the houses in the data with many bedrooms, and the fair expectation is the average price over those houses — not over all ten with the bedrooms overwritten. Each coalition is valued by the average price over the houses that match the house being explained on the known inputs, a conditional expectation.

Now knowing the bedrooms is almost knowing the size. The houses with many bedrooms are four of the five large ones, so the expected price given only the bedrooms is 170, nearly as much as given the size. In orders where the bedrooms arrive first they collect most of what the size would have collected, and averaged over the six orders the credits come to about 17.3 for the size, 10.3 for the garden and 16.3 for the bedrooms.

The bedrooms take almost as much credit as the size, for a model that does not read them. Nothing has gone wrong in the arithmetic: the game is different, and in this game the bedrooms are not a dummy, because revealing them genuinely changes what can be expected of the price. The dummy condition protects an input that never matters to the game, and whether an input matters to the game is exactly what the choice of fill decided.

One house, one model, three answers #

Side by side the three look like disagreement, and they are not. Each is the unique answer to its own question. The first says how this house differs from a reference house. The second says how the model’s own calculation depends on each input, averaging over the inputs it was not told as though they varied freely. The third says what each fact tells an observer about the price, given how the facts go together in the world.

Hugh Chen and co-authors called the choice, in 2020, one between being true to the model and true to the data. If the aim is to audit a model — to find out whether it uses something it should not — then the second is the right answer, and the third is actively misleading: it would report that the model relies on bedrooms, which it does not, because bedrooms are a proxy for something it does rely on. If the aim is to understand what a fact means about the outcome, then the third is the right answer and the second invents houses that do not exist, such as a small house with many bedrooms, and asks the model about them.

The four conditions cannot decide between these, because they are conditions on a division of a given game, and the three games are different games. That is the whole of the difficulty, and it lies in a step that happens before the Shapley value is computed at all — which is why two explanations of the same prediction, from two pieces of software that both say they compute the Shapley value, can disagree about which input mattered most.

What the starting point is #

The credits always add up to something, and what they add up to is also a choice. The first fill explains $180 - 100$, the difference from a reference house; the other two explain $180 - 136$, the difference from the average price. Neither total is the prediction itself, and neither could be: the rule divides a difference between two values of the game, and the empty coalition must be worth something.

It is tempting to replace the average price with the price of the average house, but the two are different numbers here, as they are for any model that is not a straight line. The average house is half large and half gardened and the model prices it at $100 + 30 + 5 = 135$, not 136, because the value of the average and the average of the values part company as soon as the model multiplies its inputs together. The difference is one unit here and can be anything in a model with more interaction in it.

So an explanation that reports its credits without its starting point has reported half of itself. Credits that add to 44 and credits that add to 80 are answers to different questions about the same 180, and a reader comparing explanations of two houses must first check that both were measured from the same place.

An interaction is split by surprise #

The model above has one interaction — the garden is worth something only on a large house — and the three fills already treat it differently. It is cleaner to see on its own.

Here the model pays 40 for size and garden together and nothing for either alone. The two inputs enter the formula symmetrically, and a natural first guess is that the credit for the 40 is split in half. With the unknowns set to zero it is: each input adds 0 when it arrives first and 40 when it arrives second, so each gets 20.

Drawn from the houses, it is not. Three houses in four are large and one in four has a garden, so learning that a house is large barely changes the expected price — it was probably large anyway — while learning that it has a garden changes it a great deal. The size gets 6.25 and the garden 26.25, four times as much for the same factor of a product. The average over orders does not split an interaction in two; it splits it according to how much each part of the interaction was already expected.

That is right by its own lights, and it is worth knowing before reading any explanation of a model with interactions in it. The credit an input gets depends on the population as much as on the model, and an input that is rare in the data will be credited with interactions that an input common in the data shares equally.

Computing it for more than three #

Three inputs give eight coalitions and six orders. A model with fifty inputs has more than $10^{15}$ coalitions, and for each the value is itself an expectation over the data. The general method is the one already measured for this rule: sample orders, and let the error fall like the reciprocal square root of the number drawn, as averages always do. The conditional fill adds a second difficulty, since the average over matching cases needs matching cases, and with fifty inputs almost no two cases in any real dataset match on more than a few of them; it has to be estimated by a model of the data, which adds its own error to the explanation.

For some models there is a shortcut. Lundberg and co-authors found in 2018 that for decision trees the interventional credits can be computed exactly in time polynomial in the size of the tree, by following each path once rather than each order — the same move as the cost-sharing rule with a closed form, where the structure of the game collapses the average over orders into something that can be written down.

What the figures cannot show #

Every model and every population is invented and tiny. Three inputs that are each 0 or 1 and ten or sixteen houses make every coalition’s value an exact fraction that can be printed in a table. Real explanations run on models with many continuous inputs, where every value is an estimate and the estimates have error of their own that nothing here measures.

The fill that is “right” is not decided here, and cannot be. The essay describes what each of the three answers means; which question a given reader is asking is not a mathematical fact, and the figures take no side.

The fills drawn are the three common ones, not all of them. Causal versions, which fill in the unknowns according to a model of how the inputs cause one another, sit between the second and third and need a causal model the data alone cannot supply; they are named here and not computed.

And nothing here says the explanations are useful. Whether the credits help a person understand, trust or correct a model is an empirical question about people, studied separately, and not touched by any count of orders.

Coalitions a network allows #

The same average applies to any situation that can be cast as a game, and one of the most natural is the case where the coalitions are restricted rather than the values uncertain: players who can only cooperate along the links of a network, so that a group is worth only what its connected pieces can earn. There the average over orders turns out to be the unique rule under which breaking a link costs both of its ends the same.

The two applications make the same point from opposite sides. Here the game was uncertain and had to be chosen before the rule could act; there the game is given, and the network changes it by removing coalitions that cannot form. In both cases the rule itself is untouched — the same average over the same orders — and everything interesting happens in the step that builds the game it is averaged over. Every question asked of that rule has been answered inside a game somebody built first, and the lesson of these two is that the rule was never where the disagreement lived, and never where the work of applying it well is done.

The game comes first #

The average over orders explains a prediction only after somebody has said what the model sees when an input is missing, and that decision carries most of the answer. Filled with a reference, the credits describe a difference from that reference. Filled independently from the data, they describe the model’s own use of each input, and an input it never reads gets nothing. Filled from matching cases, they describe what each fact reveals about the outcome, and a proxy takes credit for what it stands in for. The four conditions hold in all three, and they are satisfied by three different numbers for the bedrooms of one house.

None of the three is wrong and none is the explanation. Software that reports one of them without saying which has hidden a choice that matters more than any digit it prints — and a reader who asks “which fill?” before “which input?” has asked the question that decides the answer.

When a rule is the unique answer to a set of conditions, check which game it was handed — the uniqueness is inside the game, and the choice of game is outside every condition that makes the rule unique.