- Notifications
You must be signed in to change notification settings - Fork3.4k
Open
Description
Issue Description
A related but separate issue to#3948; you can end up with additivity check errors if your number of samples * number of features * number of outputs is greater than the maxuint32 value 4,294,967,295.
These are the dimensions ofphihttps://github.com/shap/shap/blob/master/shap/explainers/_tree.py#L589 which is converted to the linear arrayout_contribs used inshap/cext/tree_shap.h.
out_contibs is indexed in multiple places with a highest precision ofuint32 such ashttps://github.com/shap/shap/blob/master/shap/cext/tree_shap.h#L531
This means a not so unreasonable combination of number of samples and features could result in this error.
Minimal Reproducible Example
importnumpyasnpfromsklearn.ensembleimportRandomForestClassifierimportshapseed=0n_rows=2_000rng=np.random.default_rng(seed)X=rng.integers(low=0,high=2,size=(n_rows,1_100_000)).astype(np.float64)y=rng.integers(low=0,high=2,size=n_rows)clf=RandomForestClassifier(random_state=seed)clf.fit(X,y)clf.predict_proba(X)exp=shap.Explainer(clf,algorithm='tree',feature_perturbation='tree_path_dependent')exp(X)
Traceback
shap.utils._exceptions.ExplainerError: Additivity check failedin TreeExplainer! Please ensure the data matrix you passed to the explainer is the same shape that the model was trained on. If your data shape is correctthen please report this on GitHub. Consider retrying with the feature_perturbation='interventional' option. This check failed becausefor one of the samples the sum of the SHAP values was 1.039915,while the model output was 0.180000. If this difference is acceptable you canset check_additivity=False to disable this check.
Expected Behavior
No response
Bug report checklist
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on thelatest release of shap.
- I have confirmed this bug exists on themaster branch of shap.
- I'd be interested in making a PR to fix this bug
Installed Versions
0.46.1.dev107