Introduction#

Captum’s approach to model interpretability is in terms ofattributions. There are three kinds of attributions available inCaptum:

• Feature Attribution seeks to explain a particular output in termsof features of the input that generated it. Explaining whether amovie review was positive or negative in terms of certain words inthe review is an example of feature attribution.

• Layer Attribution examines the activity of a model’s hidden layersubsequent to a particular input. Examining the spatially-mappedoutput of a convolutional layer in response to an input image in anexample of layer attribution.

• Neuron Attribution is analagous to layer attribution, but focuseson the activity of a single neuron.

In this interactive notebook, we’ll look at Feature Attribution andLayer Attribution.

Each of the three attribution types has multipleattributionalgorithms associated with it. Many attribution algorithms fall intotwo broad categories:

• Gradient-based algorithms calculate the backward gradients of amodel output, layer output, or neuron activation with respect to theinput.Integrated Gradients (for features),Layer Gradient *Activation, andNeuron Conductance are all gradient-basedalgorithms.

• Perturbation-based algorithms examine the changes in the outputof a model, layer, or neuron in response to changes in the input. Theinput perturbations may be directed or random.Occlusion,Feature Ablation, andFeature Permutation are allperturbation-based algorithms.

We’ll be examining algorithms of both types below.

Especially where large models are involved, it can be valuable tovisualize attribution data in ways that relate it easily to the inputfeatures being examined. While it is certainly possible to create yourown visualizations with Matplotlib, Plotly, or similar tools, Captumoffers enhanced tools specific to its attributions:

• Thecaptum.attr.visualization module (imported below asviz)provides helpful functions for visualizing attributions related toimages.

• Captum Insights is an easy-to-use API on top of Captum thatprovides a visualization widget with ready-made visualizations forimage, text, and arbitrary model types.

Both of these visualization toolsets will be demonstrated in thisnotebook. The first few examples will focus on computer vision usecases, but the Captum Insights section at the end will demonstratevisualization of attributions in a multi-model, visualquestion-and-answer model.

Installation#

Before you get started, you need to have a Python environment with:

• Python version 3.6 or higher

• For the Captum Insights example, Flask 1.1 or higher and Flask-Compress(the latest version is recommended)

• PyTorch version 1.2 or higher (the latest version is recommended)

• TorchVision version 0.6 or higher (the latest version is recommended)

• Captum (the latest version is recommended)

• Matplotlib version 3.3.4, since Captum currently uses a Matplotlibfunction whose arguments have been renamed in later versions

To install Captum in an Anaconda or pip virtual environment, use theappropriate command for your environment below:

Withconda:

condainstallpytorchtorchvisioncaptumflask-compressmatplotlib=3.3.4-cpytorch

Withpip:

pipinstalltorchtorchvisioncaptummatplotlib==3.3.4Flask-Compress

Restart this notebook in the environment you set up, and you’re ready togo!

A First Example#

To start, let’s take a simple, visual example. We’ll start with a ResNetmodel pretrained on the ImageNet dataset. We’ll get a test input, anduse differentFeature Attribution algorithms to examine how theinput images affect the output, and see a helpful visualization of thisinput attribution map for some test images.

First, some imports:

importtorchimporttorch.nn.functionalasFimporttorchvision.transformsastransformsimporttorchvision.modelsasmodelsimportcaptumfromcaptum.attrimportIntegratedGradients,Occlusion,LayerGradCam,LayerAttributionfromcaptum.attrimportvisualizationasvizimportos,sysimportjsonimportnumpyasnpfromPILimportImageimportmatplotlib.pyplotaspltfrommatplotlib.colorsimportLinearSegmentedColormap

Now we’ll use the TorchVision model library to download a pretrainedResNet. Since we’re not training, we’ll place it in evaluation mode fornow.

model=models.resnet18(weights='IMAGENET1K_V1')model=model.eval()

The place where you got this interactive notebook should also have animg folder with a filecat.jpg in it.

test_img=Image.open('img/cat.jpg')test_img_data=np.asarray(test_img)plt.imshow(test_img_data)plt.show()

Our ResNet model was trained on the ImageNet dataset, and expects imagesto be of a certain size, with the channel data normalized to a specificrange of values. We’ll also pull in the list of human-readable labelsfor the categories our model recognizes - that should be in theimgfolder as well.

# model expects 224x224 3-color imagetransform=transforms.Compose([transforms.Resize(224),transforms.CenterCrop(224),transforms.ToTensor()])# standard ImageNet normalizationtransform_normalize=transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225])transformed_img=transform(test_img)input_img=transform_normalize(transformed_img)input_img=input_img.unsqueeze(0)# the model requires a dummy batch dimensionlabels_path='img/imagenet_class_index.json'withopen(labels_path)asjson_data:idx_to_labels=json.load(json_data)

Now, we can ask the question: What does our model think this imagerepresents?

output=model(input_img)output=F.softmax(output,dim=1)prediction_score,pred_label_idx=torch.topk(output,1)pred_label_idx.squeeze_()predicted_label=idx_to_labels[str(pred_label_idx.item())][1]print('Predicted:',predicted_label,'(',prediction_score.squeeze().item(),')')

We’ve confirmed that ResNet thinks our image of a cat is, in fact, acat. Butwhy does the model think this is an image of a cat?

For the answer to that, we turn to Captum.

Feature Attribution with Integrated Gradients#

Feature attribution attributes a particular output to features ofthe input. It uses a specific input - here, our test image - to generatea map of the relative importance of each input feature to a particularoutput feature.

IntegratedGradients is one ofthe feature attribution algorithms available in Captum. IntegratedGradients assigns an importance score to each input feature byapproximating the integral of the gradients of the model’s output withrespect to the inputs.

In our case, we’re going to be taking a specific element of the outputvector - that is, the one indicating the model’s confidence in itschosen category - and use Integrated Gradients to understand what partsof the input image contributed to this output.

Once we have the importance map from Integrated Gradients, we’ll use thevisualization tools in Captum to give a helpful representation of theimportance map. Captum’svisualize_image_attr() function provides avariety of options for customizing display of your attribution data.Here, we pass in a custom Matplotlib color map.

Running the cell with theintegrated_gradients.attribute() call willusually take a minute or two.

# Initialize the attribution algorithm with the modelintegrated_gradients=IntegratedGradients(model)# Ask the algorithm to attribute our output target toattributions_ig=integrated_gradients.attribute(input_img,target=pred_label_idx,n_steps=200)# Show the original image for comparison_=viz.visualize_image_attr(None,np.transpose(transformed_img.squeeze().cpu().detach().numpy(),(1,2,0)),method="original_image",title="Original Image")default_cmap=LinearSegmentedColormap.from_list('custom blue',[(0,'#ffffff'),(0.25,'#0000ff'),(1,'#0000ff')],N=256)_=viz.visualize_image_attr(np.transpose(attributions_ig.squeeze().cpu().detach().numpy(),(1,2,0)),np.transpose(transformed_img.squeeze().cpu().detach().numpy(),(1,2,0)),method='heat_map',cmap=default_cmap,show_colorbar=True,sign='positive',title='Integrated Gradients')

In the image above, you should see that Integrated Gradients gives usthe strongest signal around the cat’s location in the image.

Feature Attribution with Occlusion#

Gradient-based attribution methods help to understand the model in termsof directly computing out the output changes with respect to the input.Perturbation-based attribution methods approach this more directly, byintroducing changes to the input to measure the effect on the output.Occlusion is one such method.It involves replacing sections of the input image, and examining theeffect on the output signal.

Below, we set up Occlusion attribution. Similarly to configuring aconvolutional neural network, you can specify the size of the targetregion, and a stride length to determine the spacing of individualmeasurements. We’ll visualize the output of our Occlusion attributionwithvisualize_image_attr_multiple(), showing heat maps of bothpositive and negative attribution by region, and by masking the originalimage with the positive attribution regions. The masking gives a veryinstructive view of what regions of our cat photo the model found to bemost “cat-like”.

occlusion=Occlusion(model)attributions_occ=occlusion.attribute(input_img,target=pred_label_idx,strides=(3,8,8),sliding_window_shapes=(3,15,15),baselines=0)_=viz.visualize_image_attr_multiple(np.transpose(attributions_occ.squeeze().cpu().detach().numpy(),(1,2,0)),np.transpose(transformed_img.squeeze().cpu().detach().numpy(),(1,2,0)),["original_image","heat_map","heat_map","masked_image"],["all","positive","negative","positive"],show_colorbar=True,titles=["Original","Positive Attribution","Negative Attribution","Masked"],fig_size=(18,6))

Again, we see greater significance placed on the region of the imagethat contains the cat.

Layer Attribution with Layer GradCAM#

Layer Attribution allows you to attribute the activity of hiddenlayers within your model to features of your input. Below, we’ll use alayer attribution algorithm to examine the activity of one of theconvolutional layers within our model.

GradCAM computes the gradients of the target output with respect to thegiven layer, averages for each output channel (dimension 2 of output),and multiplies the average gradient for each channel by the layeractivations. The results are summed over all channels. GradCAM isdesigned for convnets; since the activity of convolutional layers oftenmaps spatially to the input, GradCAM attributions are often upsampledand used to mask the input.

Layer attribution is set up similarly to input attribution, except thatin addition to the model, you must specify a hidden layer within themodel that you wish to examine. As above, when we callattribute(),we specify the target class of interest.

layer_gradcam=LayerGradCam(model,model.layer3[1].conv2)attributions_lgc=layer_gradcam.attribute(input_img,target=pred_label_idx)_=viz.visualize_image_attr(attributions_lgc[0].cpu().permute(1,2,0).detach().numpy(),sign="all",title="Layer 3 Block 1 Conv 2")

We’ll use the convenience methodinterpolate() in theLayerAttributionbase class to upsample this attribution data for comparison to the inputimage.

upsamp_attr_lgc=LayerAttribution.interpolate(attributions_lgc,input_img.shape[2:])print(attributions_lgc.shape)print(upsamp_attr_lgc.shape)print(input_img.shape)_=viz.visualize_image_attr_multiple(upsamp_attr_lgc[0].cpu().permute(1,2,0).detach().numpy(),transformed_img.permute(1,2,0).numpy(),["original_image","blended_heat_map","masked_image"],["all","positive","positive"],show_colorbar=True,titles=["Original","Positive Attribution","Masked"],fig_size=(18,6))

Visualizations such as this can give you novel insights into how yourhidden layers respond to your input.

Visualization with Captum Insights#

Captum Insights is an interpretability visualization widget built on topof Captum to facilitate model understanding. Captum Insights worksacross images, text, and other features to help users understand featureattribution. It allows you to visualize attribution for multipleinput/output pairs, and provides visualization tools for image, text,and arbitrary data.

In this section of the notebook, we’ll visualize multiple imageclassification inferences with Captum Insights.

First, let’s gather some image and see what the model thinks of them.For variety, we’ll take our cat, a teapot, and a trilobite fossil:

imgs=['img/cat.jpg','img/teapot.jpg','img/trilobite.jpg']forimginimgs:img=Image.open(img)transformed_img=transform(img)input_img=transform_normalize(transformed_img)input_img=input_img.unsqueeze(0)# the model requires a dummy batch dimensionoutput=model(input_img)output=F.softmax(output,dim=1)prediction_score,pred_label_idx=torch.topk(output,1)pred_label_idx.squeeze_()predicted_label=idx_to_labels[str(pred_label_idx.item())][1]print('Predicted:',predicted_label,'/',pred_label_idx.item(),' (',prediction_score.squeeze().item(),')')

…and it looks like our model is identifying them all correctly - but ofcourse, we want to dig deeper. For that we’ll use the Captum Insightswidget, which we configure with anAttributionVisualizer object,imported below. TheAttributionVisualizer expects batches of data,so we’ll bring in Captum’sBatch helper class. And we’ll be lookingat images specifically, so well also importImageFeature.

We configure theAttributionVisualizer with the following arguments:

• An array of models to be examined (in our case, just the one)

• A scoring function, which allows Captum Insights to pull out thetop-k predictions from a model

• An ordered, human-readable list of classes our model is trained on

• A list of features to look for - in our case, anImageFeature

• A dataset, which is an iterable object returning batches of inputsand labels - just like you’d use for training

fromcaptum.insightsimportAttributionVisualizer,Batchfromcaptum.insights.attr_vis.featuresimportImageFeature# Baseline is all-zeros input - this may differ depending on your datadefbaseline_func(input):returninput*0# merging our image transforms from abovedeffull_img_transform(input):i=Image.open(input)i=transform(i)i=transform_normalize(i)i=i.unsqueeze(0)returniinput_imgs=torch.cat(list(map(lambdai:full_img_transform(i),imgs)),0)visualizer=AttributionVisualizer(models=[model],score_func=lambdao:torch.nn.functional.softmax(o,1),classes=list(map(lambdak:idx_to_labels[k][1],idx_to_labels.keys())),features=[ImageFeature("Photo",baseline_transforms=[baseline_func],input_transforms=[],)],dataset=[Batch(input_imgs,labels=[282,849,69])])

Note that running the cell above didn’t take much time at all, unlikeour attributions above. That’s because Captum Insights lets youconfigure different attribution algorithms in a visual widget, afterwhich it will compute and display the attributions.That process willtake a few minutes.

Running the cell below will render the Captum Insights widget. You canthen choose attributions methods and their arguments, filter modelresponses based on predicted class or prediction correctness, see themodel’s predictions with associated probabilities, and view heatmaps ofthe attribution compared with the original image.

visualizer.render()