Getting started

Let's assume you took the following image img, reshaped it to WHCN format (width, height, color channels, batch dimension) and ran it through a vision model:

using Images
img = load(joinpath(asset_dir, "img1.png")) # load image file
Example block output

You might use an input space attribution method (for example from ExplainableAI.jl) to determine which parts of the input contributed most to the "saxophone" class.

Let's load such an attribution val in WHCN format:

val = load(data_heatmap, "x") # load precomputed array from file
typeof(val)
Array{Float32, 4}
size(val)
(224, 224, 3, 1)

To make this attribution more interpretable, we can visualize it as a heatmap:

using VisionHeatmaps
heatmap(val)
Example block output

Custom color schemes

We can partially or fully override presets by passing keyword arguments to heatmap. For example, we can use a custom color scheme from ColorSchemes.jl using the keyword argument cs:

using ColorSchemes
heatmap(val; colorscheme=ColorSchemes.jet)
Example block output
heatmap(val; colorscheme=ColorSchemes.inferno)
Example block output

Refer to the ColorSchemes.jl catalogue for a gallery of available color schemes.

Custom color channel reduction

For arrays with multiple color channels, the channels need to be reduced to a single scalar value for each pixel, which is later mapped onto a color scheme.

The following presets are available for this purpose:

  • :sum: sum up color channels (default setting)
  • :norm: compute 2-norm over color channels
  • :maxabs: compute maximum(abs, x) over the color channels
heatmap(val; reduce=:sum)
Example block output
heatmap(val; reduce=:norm)
Example block output
heatmap(val; reduce=:maxabs)
Example block output

Using the default reduce=:sum visibly leaves more negative values in the heatmap, highlighting only the saxophone.

Mapping reduced values onto a color scheme

To map the now color-channel-reduced array onto a color scheme, we first need to normalize all values to the range $[0, 1]$.

For this purpose, two presets are available through the rangescale keyword argument:

  • :extrema: normalize to the minimum and maximum value in the array.
  • :centered: normalize to the maximum absolute value of the array. Values of zero will be mapped to the center of the color scheme.

Depending on the color scheme, one of these presets may be more suitable than the other. The default color scheme, seismic, is centered around zero, making :centered a good choice:

heatmap(val; rangescale=:centered)
Example block output

With centered color schemes such as seismic, :extrema should be avoided, as it leads to visual artifacts:

heatmap(val; rangescale=:extrema)
Example block output

However, for the inferno color scheme, which is not centered around zero, :extrema can lead to a heatmap with higher contrast.

heatmap(val; colorscheme=ColorSchemes.inferno, rangescale=:centered)
Example block output
heatmap(val; colorscheme=ColorSchemes.inferno, rangescale=:extrema)
Example block output

For the full list of heatmap keyword arguments, refer to the heatmap documentation.

Heatmapping batches

heatmap can also be used to visualize input batches. Let's assume we computed an input space attribution val_batch for the following images:

imgs = [load(joinpath(asset_dir, f)) for f in ("img1.png", "img2.png", "img3.png", "img4.png", "img5.png")] # load image files
(a vector displayed as a row to save space)

Once again, we assume that val_batch is in WHCN format:

val_batch = load(data_heatmaps, "x") # load precomputed array from file
typeof(val_batch)
Array{Float32, 4}
size(val_batch)
(224, 224, 3, 5)

Calling heatmap will automatically return an vector of images:

heatmap(val_batch)
(a vector displayed as a row to save space)

These heatmaps can be customized as usual:

heatmap(val_batch; colorscheme=ColorSchemes.inferno, rangescale=:extrema)
(a vector displayed as a row to save space)

Processing batches

The normalization when mapping values onto a color scheme can optionally be computed for a batch. Using the example of rangescale=:extrema, this means that the minimum and maximum value will be computed over all images in the batch, instead of individually for each image.

Note that this will lead to different heatmaps for each image, based on other images in the batch.

heatmap(val_batch; process_batch=true)
(a vector displayed as a row to save space)
heatmap(val_batch; colorscheme=ColorSchemes.inferno, rangescale=:extrema, process_batch=true)
(a vector displayed as a row to save space)

Consistent output types

As we have seen, calling heatmap on an array of size (W, H, C, 1) will return a single heatmap image, while calling it on an array of size (W, H, C, N) will return a vector of heatmap. This is due to the fact that VisionHeatmaps.jl will automatically "unpack" singleton vectors of heatmaps.

If this behavior is not desired, the keyword argument unpack_singleton can be set to false:

heatmap(val; unpack_singleton=false)
(a vector displayed as a row to save space)