Getting started

For this first example, we already have loaded a pre-trained LeNet5 model to look at explanations on the MNIST dataset.

using Flux

model

Chain(
  Conv((5, 5), 1 => 6, relu),           # 156 parameters
  MaxPool((2, 2)),
  Conv((5, 5), 6 => 16, relu),          # 2_416 parameters
  MaxPool((2, 2)),
  Flux.flatten,
  Dense(256 => 120, relu),              # 30_840 parameters
  Dense(120 => 84, relu),               # 10_164 parameters
  Dense(84 => 10),                      # 850 parameters
)                   # Total: 10 arrays, 44_426 parameters, 174.359 KiB.

Preparing the input data

We use MLDatasets to load a single image from the MNIST dataset:

using MLDatasets
using ImageCore, ImageIO, ImageShow

index = 10
x, y = MNIST(Float32, :test)[10]

convert2image(MNIST, x)

By convention in Flux.jl, this input needs to be resized to WHCN format by adding a color channel and batch dimensions.

input = reshape(x, 28, 28, 1, :);

Explanations

We can now select an analyzer of our choice and call analyze to get an Attribution. Note that for gradient-based optimizers, a backend for automatic differentiation must be loaded, by default Zygote.jl:

using ExplainableAI
using Zygote

analyzer = InputTimesGradient(model)
attr = analyze(input, analyzer);

The return value attr is of type Attribution and bundles the following data:

• attr.val: numerical output of the analyzer, e.g. an attribution or gradient
• attr.input: input the analyzer was applied to
• attr.output: model output for the given analyzer input
• attr.output_selection: index of the output used for the explanation
• attr.pooling: pooling that reduces attr.val over its feature dimension, e.g. over the color channels of an image
• attr.extras: optional named tuple that can be used by analyzers to return additional information.

We used InputTimesGradient, whose signed attributions are pooled by summation:

attr.pooling

SumPooling()

By default, the explanation is computed for the maximally activated output neuron. Since our digit is a 9 and Julia's indexing is 1-based, the output neuron at index 10 of our trained model is maximally activated.

Finally, we obtain the result of the analyzer in form of an array.

attr.val

28×28×1×1 Array{Float32, 4}:
[:, :, 1, 1] =
 -0.0   0.0   0.0  -0.0  -0.0  -0.0  -0.0  …  -0.0       0.0         0.0
 -0.0   0.0   0.0  -0.0  -0.0  -0.0  -0.0     -0.0       0.0         0.0
 -0.0   0.0   0.0  -0.0  -0.0  -0.0  -0.0     -0.0       0.0         0.0
  0.0   0.0   0.0  -0.0  -0.0  -0.0  -0.0      0.0       0.0         0.0
  0.0   0.0  -0.0  -0.0  -0.0   0.0   0.0      0.0       0.0         0.0
 -0.0   0.0   0.0  -0.0   0.0  -0.0  -0.0  …   0.0       0.0         0.0
 -0.0   0.0   0.0   0.0   0.0   0.0   0.0      0.0       0.0         0.0
 -0.0   0.0   0.0   0.0  -0.0   0.0  -0.0      0.0       0.0        -0.0
 -0.0   0.0   0.0   0.0  -0.0  -0.0  -0.0      0.0       0.0        -0.0
 -0.0   0.0   0.0   0.0  -0.0   0.0  -0.0      0.385115  0.0714216   0.0
  ⋮                             ⋮          ⋱   ⋮                    
  0.0  -0.0  -0.0   0.0   0.0   0.0  -0.0     -0.0       0.0         0.0
 -0.0  -0.0   0.0   0.0   0.0   0.0  -0.0  …  -0.0       0.0         0.0
 -0.0   0.0   0.0   0.0   0.0   0.0  -0.0      0.0       0.0         0.0
 -0.0   0.0   0.0   0.0  -0.0   0.0  -0.0      0.0       0.0         0.0
 -0.0   0.0   0.0  -0.0  -0.0   0.0  -0.0      0.0       0.0         0.0
  0.0   0.0  -0.0  -0.0  -0.0   0.0  -0.0      0.0       0.0         0.0
  0.0   0.0   0.0  -0.0  -0.0   0.0   0.0  …   0.0       0.0         0.0
  0.0   0.0  -0.0  -0.0   0.0   0.0   0.0      0.0       0.0         0.0
  0.0   0.0   0.0   0.0   0.0   0.0  -0.0      0.0       0.0         0.0

Heatmapping basics

Since the array attr.val is not very informative at first sight, we can visualize Attributions by computing a heatmap using either VisionHeatmaps.jl or TextHeatmaps.jl.

using VisionHeatmaps

heatmap(attr)

If we are only interested in the heatmap, we can combine analysis and heatmapping into a single function call:

heatmap(input, analyzer)

Neuron selection

By passing an additional index to our call to analyze, we can compute an explanation with respect to a specific output neuron. Let's see why the output wasn't interpreted as a 4 (output neuron at index 5)

attr = analyze(input, analyzer, 5)
heatmap(attr)

This heatmap shows us that the "upper loop" of the hand-drawn 9 has negative relevance with respect to the output neuron corresponding to digit 4!

heatmap(input, analyzer, 5)

Analyzing batches

ExplainableAI also supports explanations of input batches:

batchsize = 20
xs, _ = MNIST(Float32, :test)[1:batchsize]
batch = reshape(xs, 28, 28, 1, :) # reshape to WHCN format
attr = analyze(batch, analyzer);

This will return a single Attributionattr for the entire batch. Calling heatmap on attr will detect the batch dimension and return a vector of heatmaps.

heatmap(attr)

# Custom heatmaps

The function heatmap automatically picks a preset based on the pooling of the attribution.

Since InputTimesGradient computes signed attributions, they are summed over color channels and shown in a diverging colormap. Gradient attributions are pooled by their norm, which is non-negative and therefore shown in a sequential colormap:

analyzer = Gradient(model)
heatmap(input, analyzer)

analyzer = InputTimesGradient(model)
heatmap(input, analyzer)

Using VisionHeatmaps.jl, heatmaps can be heavily customized. Check out the heatmapping documentation for more information.

This page was generated using Literate.jl.