Machine Learning · TreeSHAP · Part 4 of 5

Choose What a Hidden Feature Means

October 2026 · 6 min read

The last chapter used stored branch counts whenever an input was hidden. We will now compare that rule with filling hidden inputs from the ten customer rows. Maya remains Returning, Premium, High usage, and the tree still predicts 90.

Keep the branch-count rule in view

When A is known, we follow Maya into the Returning branch. Usage C is still hidden. Of the six Returning customers, two have Low usage and four have High usage, so:

$$v_{\mathrm{path}}(\{A\})= \frac26(50)+\frac46(90)=76\tfrac23.$$

This uses the counts at the usage split inside the Returning branch. The rule is: follow a known split, and average by its stored child-to-parent counts at an unknown split. It is called tree-path-dependent in TreeExplainer.

Fill hidden inputs from the original customer rows

A second rule begins with a copy of each of the ten customer rows. Overwrite the known columns with Maya's values, keep the other columns from that row, run the tree, and average the ten predictions.

For “only A known,” set every row's customer group to Returning. Leave its plan and usage unchanged. All rows now reach the Returning branch, where only usage matters:

The same usage column, counted in two different populations.
Usage	Among the 6 Returning customers	Among all 10 customers	Prediction after setting A to Returning
Low	2	4	50
High	4	6	90

$$v_{\mathrm{ref}}(\{A\})= \frac4{10}(50)+\frac6{10}(90)=74.$$

The difference is entirely in the weights. The path rule uses the 2 Low and 4 High customers already inside the Returning branch. Replacement uses the usage values from all ten rows, including the four originally New customers.

For any known group S, the replacement rule is:

$$v_{\mathrm{ref}}(S)= \frac1{10}\sum_{i=1}^{10}f(x_S,r_{\bar S}^{(i)}).$$

Here $x_S$ supplies Maya's known columns. The remaining columns, $r_{\bar S}^{(i)}$, come together from reference row i. We average the model's predictions on these completed rows. TreeExplainer calls this setting interventional.

Change the group values, then use the same Shapley average

With no columns overwritten, both rules give the original model average, 52. With every column overwritten, every completed row is Maya and predicts 90. For the groups in between:

All eight groups for the same tree and Maya. Only the A and A,B rows differ.
Known group	Stored path counts	Replace from 10 rows
None	52	52
A	76⅔	74
B	58	58
C	60	60
A, B	76⅔	74
A, C	90	90
B, C	66	66
A, B, C	90	90

Knowing B in addition to A does not change the result: plan is never tested in the Returning branch. The difference between the two A rows therefore carries over to the A,B rows.

To calculate A's replacement contribution, use the same reveal-order weights as before. With three features, each position has probability 1/3. If A is in the middle, either B or C can precede it, giving each middle case weight 1/6:

Known before A	Change when A is revealed	Weight
None	74 − 52 = 22	1/3
B	74 − 58 = 16	1/6
C	90 − 60 = 30	1/6
B, C	90 − 66 = 24	1/3

$$\phi_A=\frac{22}{3}+\frac{16}{6}+\frac{30}{6}+\frac{24}{3}=23.$$

Applying that same average to B and C gives:

Rule	A	B	C
Stored path counts	24⅓	3	10⅔
Replace from 10 rows	23	3	12

Both rows sum to 38 and reconstruct 52 + 38 = 90. A receives 1⅓ less under replacement and C receives 1⅓ more. The prediction has not changed; the averages used while inputs are hidden have changed.

Why we keep all ten reference rows

Replacement overwrites known columns in every reference row. Filtering the dataset down to customers who already match Maya would define a different average.

For instance, the four Premium customers are Dev, Farah, Hari, and Jay. Their model predictions are 30, 50, 90, and 90, averaging 65. That is the empirical conditional value for “B known.” Both rules in our table instead give 58. Stored branch counts and background replacement should therefore not be described as filtering on all known features.

Use the rule that defines the comparison you want

For our original explanation, use feature_perturbation="tree_path_dependent". To compare Maya with completed inputs formed from these ten customer rows, supply those rows as data and use feature_perturbation="interventional". Keep the feature order A, B, C in both cases.

We now know the expected answer for each setting. The final chapter passes this exact tree to TreeExplainer and checks its results against these calculations.

The TreeExplainer API documents both settings. The customer counts and group values above are calculated from the same ten rows used throughout this series.