Getting started
Let's assume you took the following image img
, reshaped it to WHCN format (width, height, color channels, batch dimension) and ran it through a vision model:
using Images
img = load(joinpath(asset_dir, "img1.png")) # load image file
You might use an input space attribution method (for example from ExplainableAI.jl) to determine which parts of the input contributed most to the "saxophone" class.
Let's load such an attribution val
in WHCN format:
val = load(data_heatmap, "x") # load precomputed array from file
typeof(val)
Array{Float32, 4}
size(val)
(224, 224, 3, 1)
To make this attribution more interpretable, we can visualize it as a heatmap:
using VisionHeatmaps
heatmap(val)
Custom color schemes
We can partially or fully override presets by passing keyword arguments to heatmap
. For example, we can use a custom color scheme from ColorSchemes.jl using the keyword argument cs
:
using ColorSchemes
heatmap(val; colorscheme=ColorSchemes.jet)
heatmap(val; colorscheme=ColorSchemes.inferno)
Refer to the ColorSchemes.jl catalogue for a gallery of available color schemes.
Custom color channel reduction
For arrays with multiple color channels, the channels need to be reduced to a single scalar value for each pixel, which is later mapped onto a color scheme.
The following presets are available for this purpose:
:sum
: sum up color channels (default setting):norm
: compute 2-norm over color channels:maxabs
: computemaximum(abs, x)
over the color channels
heatmap(val; reduce=:sum)
heatmap(val; reduce=:norm)
heatmap(val; reduce=:maxabs)
Using the default reduce=:sum
visibly leaves more negative values in the heatmap, highlighting only the saxophone.
Mapping reduced values onto a color scheme
To map the now color-channel-reduced array onto a color scheme, we first need to normalize all values to the range $[0, 1]$.
For this purpose, two presets are available through the rangescale
keyword argument:
:extrema
: normalize to the minimum and maximum value in the array.:centered
: normalize to the maximum absolute value of the array. Values of zero will be mapped to the center of the color scheme.
Depending on the color scheme, one of these presets may be more suitable than the other. The default color scheme, seismic
, is centered around zero, making :centered
a good choice:
heatmap(val; rangescale=:centered)
With centered color schemes such as seismic
, :extrema
should be avoided, as it leads to visual artifacts:
heatmap(val; rangescale=:extrema)
However, for the inferno
color scheme, which is not centered around zero, :extrema
can lead to a heatmap with higher contrast.
heatmap(val; colorscheme=ColorSchemes.inferno, rangescale=:centered)
heatmap(val; colorscheme=ColorSchemes.inferno, rangescale=:extrema)
For the full list of heatmap
keyword arguments, refer to the heatmap
documentation.
Heatmapping batches
heatmap
can also be used to visualize input batches. Let's assume we computed an input space attribution val_batch
for the following images:
imgs = [load(joinpath(asset_dir, f)) for f in ("img1.png", "img2.png", "img3.png", "img4.png", "img5.png")] # load image files
Once again, we assume that val_batch
is in WHCN format:
val_batch = load(data_heatmaps, "x") # load precomputed array from file
typeof(val_batch)
Array{Float32, 4}
size(val_batch)
(224, 224, 3, 5)
Calling heatmap
will automatically return an vector of images:
heatmap(val_batch)
These heatmaps can be customized as usual:
heatmap(val_batch; colorscheme=ColorSchemes.inferno, rangescale=:extrema)
Processing batches
The normalization when mapping values onto a color scheme can optionally be computed for a batch. Using the example of rangescale=:extrema
, this means that the minimum and maximum value will be computed over all images in the batch, instead of individually for each image.
Note that this will lead to different heatmaps for each image, based on other images in the batch.
heatmap(val_batch; process_batch=true)
heatmap(val_batch; colorscheme=ColorSchemes.inferno, rangescale=:extrema, process_batch=true)
Consistent output types
As we have seen, calling heatmap
on an array of size (W, H, C, 1)
will return a single heatmap image, while calling it on an array of size (W, H, C, N)
will return a vector of heatmap. This is due to the fact that VisionHeatmaps.jl will automatically "unpack" singleton vectors of heatmaps.
If this behavior is not desired, the keyword argument unpack_singleton
can be set to false
:
heatmap(val; unpack_singleton=false)