API Reference

Abstract super type of all XAI methods.

XAI methods are expected to implement call_analyzer, returning an Attribution:

call_analyzer(input, method::MyMethod, output_selector::AbstractOutputSelector; kwargs...)

If this function is implemented, XAIBase provides the analyze functionality. The resulting attributions can be visualized using VisionHeatmaps.jl or TextHeatmaps.jl.

analyze(input, method)
analyze(input, method, output_selection)

Apply the analyzer method for the given input, returning an Attribution. If output_selection is specified, the attribution will be calculated for that output. Otherwise, the output with the highest activation is automatically chosen.

See also Attribution.

Attribution(val, input, output, output_selection, pooling; extras)

Return type of feature-attribution methods when calling analyze.

Fields

• val: numerical output of the analyzer, e.g. an attribution or gradient
• input: input for the given analyzer
• output: model output for the given analyzer input
• output_selection: index of the output the attribution was computed for
• pooling: an AbstractPooling function that reduces val over its feature dimension when projecting the attribution onto a human-interpretable space, e.g. for visualization. Downstream packages such as VisionHeatmaps.jl call it as pooling(val, dims), choosing the reduced dimension dims themselves. Defaults to NormPooling.
• extras: optional named tuple that can be used by analyzers to return additional information. Keyword argument, defaults to nothing.

pooling is an optional positional argument, defaulting to NormPooling.

Abstract super type of all feature selectors in XAIBase.

Feature selectors are expected to be callable and to return a vector of CartesianIndices for each selected feature.

Note

The XAIBase interface currently assumes that features have either 2 or 4 dimensions ((features, batchsize) or (width, height, features, batchsize)).

It also assumes that the batch dimension is the last dimension of the feature.

IndexedFeatures(indices...)

Select features by indices.

For outputs of convolutional layers, the index refers to a feature dimension.

See also TopNFeatures.

Note

The XAIBase interface currently assumes that features have either 2 or 4 dimensions ((features, batchsize) or (width, height, features, batchsize)).

It also assumes that the batch dimension is the last dimension of the feature.

Example

julia> feature_selector = IndexedFeatures(2, 3)
 IndexedFeatures(2, 3)

julia> feature = rand(3, 3, 3, 2);

julia> feature_selector(feature)
2-element Vector{Vector{CartesianIndices{4, NTuple{4, UnitRange{Int64}}}}}:
 [CartesianIndices((1:3, 1:3, 2:2, 1:1)), CartesianIndices((1:3, 1:3, 2:2, 2:2))]
 [CartesianIndices((1:3, 1:3, 3:3, 1:1)), CartesianIndices((1:3, 1:3, 3:3, 2:2))]

julia> feature = rand(3, 2);

julia> feature_selector(feature)
 1-element Vector{Vector{CartesianIndices{2, Tuple{UnitRange{Int64}, UnitRange{Int64}}}}}:
  [CartesianIndices((2:2, 1:1)), CartesianIndices((2:2, 2:2))]

TopNFeatures(n)

Select top-n features.

For outputs of convolutional layers, the relevance is summed across height and width channels for each feature.

See also IndexedFeatures.

Note

The XAIBase interface currently assumes that features have either 2 or 4 dimensions ((features, batchsize) or (width, height, features, batchsize)).

It also assumes that the batch dimension is the last dimension of the feature.

Example

julia> feature_selector = TopNFeatures(2)
 TopNFeatures(2)

julia> feature = rand(3, 2)
3×2 Matrix{Float64}:
 0.265312  0.953689
 0.674377  0.172154
 0.649722  0.570809

julia> feature_selector(feature)
2-element Vector{Vector{CartesianIndices{2, Tuple{UnitRange{Int64}, UnitRange{Int64}}}}}:
 [CartesianIndices((2:2, 1:1)), CartesianIndices((1:1, 2:2))]
 [CartesianIndices((3:3, 1:1)), CartesianIndices((3:3, 2:2))]

julia> feature = rand(3, 3, 3, 2);

julia> feature_selector(feature)
2-element Vector{Vector{CartesianIndices{4, NTuple{4, UnitRange{Int64}}}}}:
 [CartesianIndices((1:3, 1:3, 2:2, 1:1)), CartesianIndices((1:3, 1:3, 1:1, 2:2))]
 [CartesianIndices((1:3, 1:3, 1:1, 1:1)), CartesianIndices((1:3, 1:3, 3:3, 2:2))]

Abstract super type of all output selectors in XAIBase.

Output selectors are expected to be callable and to return a vector of CartesianIndex of the selected outputs.

Note

XAIBase assumes that the output of a model is a matrix of logits and that the batch dimension is the last dimension of the output.

MaxActivationSelector()

Output selector that picks the output with the highest activation.

Note

XAIBase assumes that the output of a model is a matrix of logits and that the batch dimension is the last dimension of the output.

Example

julia> output = rand(3, 3)
3×3 Matrix{Float64}:
 0.411871  0.313366  0.13402
 0.885562  0.136938  0.465622
 0.498235  0.627209  0.298911

julia> output_selector = MaxActivationSelector()
 MaxActivationSelector()

julia> output_selector(output)
 3-element Vector{CartesianIndex{2}}:
  CartesianIndex(2, 1)
  CartesianIndex(3, 2)
  CartesianIndex(2, 3)

IndexSelector(index)

Output selector that picks the output at the given index. If multiple indices are provided, outputs of a batch are selected separately.

Note

XAIBase assumes that the output of a model is a matrix of logits and that the batch dimension is the last dimension of the output.

Example

julia> output = rand(3, 3)
3×3 Matrix{Float64}:
 0.411871  0.313366  0.13402
 0.885562  0.136938  0.465622
 0.498235  0.627209  0.298911

julia> output_selector = IndexSelector(1)
 IndexSelector{Int64}(1)

julia> output_selector(output)
 3-element Vector{CartesianIndex{2}}:
  CartesianIndex(1, 1)
  CartesianIndex(1, 2)
  CartesianIndex(1, 3)

julia> output_selector = IndexSelector((1, 1, 2));

julia> output_selector(output)
3-element Vector{CartesianIndex{2}}:
 CartesianIndex(1, 1)
 CartesianIndex(1, 2)
 CartesianIndex(2, 3)

Batch(A; dims = ndims(A))

Wrap an array A to mark it as a batch of samples along the batch dimension dims, which defaults to the last dimension of A. Unwrapped arrays are treated as single samples.

Downstream packages use Batch to distinguish transforms that are applied to each sample individually (the default, see XAIBase.mapsamples) from batch-aware transforms like BatchedNormalization.

eachsample(batch)

Iterate over the samples in a XAIBase.Batch, which are views into the wrapped array.

mapsamples(f, batch)
mapsamples(f, batch, batches...)

Apply f to each sample in a XAIBase.Batch and stack the results into a new Batch. When called with several batches, f is called on corresponding samples, like map.

All results of f need to have the same size. If f changes the number of dimensions of the samples, a batch dimension that was the last dimension of batch remains the last dimension. Otherwise, the batch dimension keeps its index.

AbstractTransform

Abstract super type of all transforms of attributions, e.g. feature-attribution pooling functions (AbstractPooling) and normalization functions (AbstractNormalization).

Transforms can be composed into a Pipeline using |>. XAIBase only defines the composition of transforms: how transforms are applied is defined by downstream packages, e.g. VisionHeatmaps.jl and TextHeatmaps.jl.

Pipeline(transforms...)

Sequentially apply transforms of type AbstractTransform. Pipelines are usually created by composing transforms using |>:

pipeline = NormPooling() |> ExtremaNormalization()

compose(transforms...)

Compose transforms of type AbstractTransform to create a Pipeline. Nested pipelines are flattened. t1 |> t2 is an alias for compose(t1, t2).

NormPooling()

$\ell^2$-norm pooling $\sqrt{\sum_i a_i^2}$ over the feature dimension. Returns non-negative values.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

SumPooling()

Sum-pooling $\sum_i a_i$ over the feature dimension. Returns signed values.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

MaxPooling()

Max-pooling $\max_i a_i$ over the feature dimension. Returns signed values.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

SumAbsPooling()

Sum-of-absolute-values pooling $\sum_i |a_i|$ ($\ell^1$-norm) over the feature dimension. Returns non-negative values.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

AbsSumPooling()

Absolute-value-of-sum pooling $\left| \sum_i a_i \right|$ over the feature dimension. Returns non-negative values.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

MaxAbsPooling()

Maximum-absolute-value pooling $\max_i |a_i|$ ($\ell^\infty$-norm) over the feature dimension. Returns non-negative values.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

SquaredNormPooling()

Squared $\ell^2$-norm pooling $\sum_i a_i^2$ over the feature dimension. Returns non-negative values.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

pool(pooling, A, dims)

Reduce array A over the feature dimension dims using the attribution pooling function pooling, an AbstractPooling.

For convenience, pooling(A, dims) is equivalent to pool(pooling, A, dims).

A can also be a XAIBase.Batch, whose batch dimension is updated to account for the dropped dimension. The batch dimension itself can't be pooled.

Custom pooling functions implement pool(pooling, A::AbstractArray, dims), which has to drop the pooled dimension.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

Abstract super type of all attribution pooling functions in XAIBase.

Pooling functions are fieldless structs that reduce an array over a feature dimension via pool. They are subtypes of either UnsignedPooling or SignedPooling, depending on whether their output is non-negative or signed.

Note

Attribution pooling projects a high-dimensional attribution (e.g. RGB values per pixel) onto a lower-dimensional, human-interpretable space (e.g. a single value per pixel) by reducing over a feature dimension dims (typically the color-channel dimension 3 of a (width, height, channels, batch) array).

The pooled dimension is dropped, e.g. an array of size (W, H, C, N) is reduced to size (W, H, N).

Pooling functions are documented by their action on a single slice $a = (a_1, \ldots, a_n)$ along the pooled dimension dims.

Abstract super type of attribution pooling functions with unsigned output, e.g. SumAbsPooling, AbsSumPooling, MaxAbsPooling, NormPooling and SquaredNormPooling.

Unsigned attributions are best visualized using a sequential colormap.

Abstract super type of attribution pooling functions with signed output, e.g. SumPooling and MaxPooling.

Signed attributions are best visualized using a diverging colormap.

SignedNoPooling()

Identity pooling that leaves values unchanged and only drops the feature dimension dims, which has to be a singleton.

Use SignedNoPooling for attributions that are already reduced along the feature dimension and therefore require no pooling. SignedNoPooling subtypes SignedPooling and is therefore visualized using a diverging colormap.

For attributions that are guaranteed to be non-negative, use UnsignedNoPooling.

UnsignedNoPooling()

Identity pooling that leaves values unchanged and only drops the feature dimension dims, which has to be a singleton.

Use UnsignedNoPooling for attributions that are already reduced along the feature dimension and guaranteed to be non-negative. UnsignedNoPooling subtypes UnsignedPooling and is therefore visualized using a sequential colormap.

For attributions of unknown sign, use SignedNoPooling.

normalize(normalization, A)
normalize(normalization, A, bounds)

Rescale the values of array A linearly onto the unit interval [0, 1] using the normalization function normalization, an AbstractNormalization. The value range bounds = (lo, hi) is mapped onto [0, 1] and values outside of it are clamped. If bounds is not provided, it is computed from A via normalization_bounds.

For convenience, normalization(A) is equivalent to normalize(normalization, A).

Passing precomputed bounds normalizes A to a shared value range, e.g. to make heatmaps comparable across all samples in a batch:

bounds = normalization_bounds(normalization, batch)
slices = [normalize(normalization, x, bounds) for x in eachslice(batch; dims = 4)]

Note

Normalization is the step between feature-attribution pooling and colormapping: it linearly rescales pooled attribution values onto the unit interval [0, 1], which colormaps expect as input.

The appropriate normalization depends on the sign of the pooled values and is therefore determined by the pooling function's supertype (see default_normalization).

Abstract super type of all normalization functions in XAIBase.

Normalization functions are fieldless structs that linearly rescale arrays of pooled attribution values onto the unit interval [0, 1] via XAIBase.normalize. The value range that is mapped onto [0, 1] is computed by normalization_bounds.

Note

Normalization is the step between feature-attribution pooling and colormapping: it linearly rescales pooled attribution values onto the unit interval [0, 1], which colormaps expect as input.

The appropriate normalization depends on the sign of the pooled values and is therefore determined by the pooling function's supertype (see default_normalization).

ExtremaNormalization()

Normalization that linearly maps the value range (minimum(A), maximum(A)) onto the unit interval [0, 1].

ExtremaNormalization is a suitable normalization for unsigned attributions (UnsignedPooling), which are visualized with a sequential colormap.

Note

Normalization is the step between feature-attribution pooling and colormapping: it linearly rescales pooled attribution values onto the unit interval [0, 1], which colormaps expect as input.

The appropriate normalization depends on the sign of the pooled values and is therefore determined by the pooling function's supertype (see default_normalization).

CenteredNormalization()

Normalization that linearly maps the symmetric value range (-maximum(abs, A), maximum(abs, A)) onto the unit interval [0, 1], mapping the value zero onto the midpoint 0.5.

CenteredNormalization is the natural normalization for signed attributions (SignedPooling), which are visualized with a diverging colormap whose neutral midpoint then corresponds to zero attribution.

Note

Normalization is the step between feature-attribution pooling and colormapping: it linearly rescales pooled attribution values onto the unit interval [0, 1], which colormaps expect as input.

The appropriate normalization depends on the sign of the pooled values and is therefore determined by the pooling function's supertype (see default_normalization).

BatchedNormalization(normalization)

Normalize a whole XAIBase.Batch at once, computing a shared value range over all samples via normalization_bounds. This makes heatmaps comparable across the samples in a batch.

By default, normalization functions normalize each sample in a batch separately. On single samples, BatchedNormalization(normalization) behaves like normalization.

Note

Normalization is the step between feature-attribution pooling and colormapping: it linearly rescales pooled attribution values onto the unit interval [0, 1], which colormaps expect as input.

The appropriate normalization depends on the sign of the pooled values and is therefore determined by the pooling function's supertype (see default_normalization).

normalization_bounds(normalization, A)

Compute the value range (lo, hi) of A that XAIBase.normalize maps onto the unit interval [0, 1].

A can be an array or any iterator of real values. This allows bounds to be computed across a whole batch of arrays, e.g. via Iterators.flatten, for normalization to a shared value range.

Note

Normalization is the step between feature-attribution pooling and colormapping: it linearly rescales pooled attribution values onto the unit interval [0, 1], which colormaps expect as input.

The appropriate normalization depends on the sign of the pooled values and is therefore determined by the pooling function's supertype (see default_normalization).

default_normalization(pooling)

Return the natural AbstractNormalization for the given feature-attribution pooling function, determined by the sign of the pooling's output:

• UnsignedPooling → ExtremaNormalization (sequential colormap)
• SignedPooling → CenteredNormalization (diverging colormap)

issigned(pooling)
issigned(normalization)

Return true if a pooling function returns signed values or if a normalization function is meant to be applied to signed values. Return false for unsigned values and nothing if the sign is unknown, which is the default for custom normalization functions and for pooling functions that subtype neither SignedPooling nor UnsignedPooling.

Composing a pooling function with a normalization function of a different sign emits a warning. Downstream packages can use issigned to select a suitable colormap: diverging for signed values, sequential for unsigned values.

test_interface(method, input; output_selection = 1)

Test that method, an AbstractXAIMethod, correctly implements the XAIBase interface for the given input, returning true if it does.

The returned Attribution is checked by calling analyze, asserting that

• analyze returns an Attribution,
• the batch dimension (the last dimension) of val, input and output is consistent,
• output_selection is a Vector{<:CartesianIndex} with one entry per batch sample, and
• pooling is an AbstractPooling.

Both the automatic output selection (analyze(input, method)) and the explicit selection analyze(input, method, output_selection) are exercised. output_selection is passed to an IndexSelector and defaults to 1.

The first violated invariant throws an InterfaceError describing the problem.

using XAIBase, Test

@testset "XAIBase interface" begin
    @test XAIBase.test_interface(MyMethod(), rand(Float32, 32, 32, 3, 2))
end

Note

XAIBase assumes that the output of a model is a matrix of logits and that the batch dimension is the last dimension of the output.