animal2vec workshop · MPIABDay 2 · 14:15–15:00

Beyond the label

What an embedding is—and what it is not.

Julian C. Schäfer-Zimmermann

Max Planck Institute of Animal Behavior
Department for the Ecology of Animal Societies
Communication and Collective Movement (CoCoMo) Group

animal2vec workshop · MPIABDay 2 · 14:15–15:00

A representation is not a prediction

Embeddings

[T, D]
T time positions; D learned features.
Useful for comparison and downstream analysis.

Label scores

[T, C]
The same T positions; C trained labels.
Useful for the supervised detection task.

The baseline has 1,024 feature dimensions. So, for 1s of audio, using the default settings, we get 200 1,024 dimensional vectors. This is a very high-dimensional space to explore.

animal2vec workshop · MPIABDay 2 · Introduction to embeddings

What is an embedding?

Audio over time

Current segment: noise 0–5 ms · frame 1 / 90

2D view of embedding space

One point per frame

noisecall_A call_Bcurrent frame

Interpretation

An embedding is a learned vector representation of a sound in context.

Current 5 ms bin

Regionnoise Point(0.0, 0.0)

This frame is in the noise region.

Animation controls

Slowed for teaching

Jump to

Synthetic illustration, not model output.
Real embeddings are high-dimensional.

This is the layout-corrected version of the existing 90-frame, 450 ms teaching animation. The synthetic points, three call/noise clusters and 5 ms frame mapping have been preserved. Play advances one frame every 170 ms of presentation time, deliberately slower than real time. It is a silent waveform illustration, not a playable WAV file or an animal2vec inference result. An embedding is a learned representation, not a class probability or a hand-measured acoustic feature. The displayed clusters are intentionally clean, invented 2D coordinates; they are not a computed projection of model embeddings. Similarity may support retrieval or downstream analysis, but clusters do not automatically identify biological categories. The colors are supplied teaching labels, not the model assigning labels to this plot. For this slide, each point represents one 5 ms frame; real contextual embeddings also depend on surrounding audio. The whole composition scales uniformly inside reveal.js, and the SVG graphics preserve their aspect ratios. Use Play/Pause, Step, Reset or the jump controls. Autoplay can be disabled with data-autoplay="false".

animal2vec workshop · MPIABDay 2 · Introduction to embeddings

Pretrained vs fine-tuned embeddings

Audio over time

Current segment: noise 0–5 ms · frame 1 / 90

2D view of embedding space

Pretrained

noisecall_A call_Bcurrent frame

Animation controls

PlaybackSlowed for teaching

Jump to

Current 5 ms bin

Example labelnoise Plotted point(0.0, 0.0)

Switch models to compare this same frame.

Synthetic comparison, not measured embeddings.
Clean clustering is illustrative, not guaranteed.

Use the same time position in both views. Pause on call_A or call_B, then toggle Pretrained and Finetuned: there is one plot, with fixed limits, so neither plot size nor the current frame changes. The points move between two illustrative layouts. The animation also permits switching while the time scan continues.

The starting layout represents useful but broader pretrained groups. The Finetuned layout is deliberately more compact for the displayed labels. Coordinates and their interpolation are synthetic teaching devices, not animal2vec outputs, PCA/UMAP results, or measured trajectories through fine-tuning. Noise is a teaching category here, not a claim that the classifier must have a dedicated noise output. Colors are example annotations.

Scientific context: the supplied supplemental information, “Linear evaluation of frozen embeddings,” Table S2 (pp. 3–4), compares pretrained and fine-tuned representations through a linear classifier. It supports improved linear-evaluation performance after fine-tuning, not a universal reduction in within-class variance. That distinction is why the tighter clouds here are explicitly labeled as an illustration.

Do not infer that two independently trained checkpoints share aligned coordinates. The smooth correspondence is chosen only for teaching. These axes are arbitrary visualization coordinates, not named acoustic measurements or class probabilities.

This preserves the earlier silent waveform illustration: 90 frames, each representing 5 ms (450 ms total). Play advances a frame every 170 ms of presentation time. It does not play a WAV file. Contextualized embeddings can also depend on surrounding audio.

Reset returns to the first frame without changing the selected model. Model toggles preserve Play/Pause state. At the final frame playback stops. A new Play starts from the beginning. Reduced-motion preferences disable automatic playback and point morphing. There are no external assets, plotting libraries, or model dependencies.

animal2vec workshop · MPIABDay 2 · Introduction to embeddings

What should one point represent?

Audio over time

Current segment: call_A

Frame vectors and their summaries

Temporal mean

noisecall_Acall_Bsummary

Choose the unit

90 frame vectors[90, D]

Jump to

Synthetic 2D vectors; real data have D features.
One mean per interval, not a new model prediction.

Purpose: choose the observation unit before choosing a pooling operator. Suggested demonstration: inspect Frames, switch to Events, then 50 ms windows and Recording. Play or step through the same time positions. All summaries on this slide use the temporal arithmetic mean, so only interval membership changes.

The 90-frame silent synthetic example is reused from the preceding slides. Frames 25–40 (120–200 ms) are the known call_A event; frames 53–69 (260–345 ms) are the known call_B event. Intervals are half-open. Two event vectors exclude the 57 background frames. The event grouping is supplied as teaching annotation; no detection algorithm runs here. Fixed non-overlapping windows contain ten frames each, giving nine 50 ms vectors. A recording summary covers all 450 ms. A point represents an entire feature vector, not an average across features.

The drawing uses actual invented two-dimensional vectors, obtained by re-expressing the last slide’s fine-tuned display coordinates. Numerical temporal means are calculated from those vectors. This is not a PCA/UMAP projection of animal2vec outputs. For real data, compute summaries in the original D-dimensional embedding space, and project only for display. Pool only valid frames; retain the original vectors and metadata.

The moving dots illustrate a reduction of stored vectors; weights, source embeddings and waveform do not change. One summary vector is compact and can be useful for retrieval, clustering or a downstream predictor, but loses explicit within-interval ordering and individual frame values. These are analysis choices, not performance guarantees. Longer background intervals contribute more rows to the recording mean. Keep source recording, start/end times, duration and frame count alongside every summary. Distinct calls of the same label should each receive their own event vector.

animal2vec connection: In the supplied workshop code, InferenceRunner returns result.embeddings [T,D] and result.timestamps [T] after averaging selected layer outputs at each frame. Temporal pooling is a separate post-extraction operation; it is not the layer averaging of the classifier input and not probability smoothing used for event boundaries. Select real rows by the returned frame-center timestamps, e.g. start <= t < end, rather than dividing recording duration by T. Frame vectors can retain context from outside the selected interval; selecting rows does not rerun the model on an isolated call.

Sources: Schäfer-Zimmermann et al. (2025), animal2vec and MeerKAT, doi:10.1111/2041-210x.70218, section 2.2.2; supplemental “Finetuning animal2vec,” p. 8, for averaging across Transformer layers; supplemental pp. 7–8 for 200 Hz output spacing. Workshop archive a2v2/workflows.py, InferenceRunner.run_tensor, for aligned returned arrays. Temporal grouping here is a workshop illustration, not a temporal pooling benchmark reported in the paper. NumPy mean documents the arithmetic operation.

Controls: Play/Pause advances one 5 ms model frame every 170 ms of presentation time. Reset preserves the unit and returns to frame 1. Mode switches preserve the scan position. This does not play a WAV. Automatic playback is off. Reduced motion skips the convergence animation; manual playback remains available.

animal2vec workshop · MPIABDay 2 · Introduction to embeddings

Many frames, one summary vector

Audio over time

2D toy embedding space

noisecall_Acall_Bsummary

Choose the aggregation

Perturb one frame

Purpose: after choosing the temporal interval, compare what a fixed-length summary keeps and discards. Suggested sequence: call_A + Mean; Median; perturb one frame; compare Mean and Median; choose Recording and inspect Max/Min. Then show Medoid as a distinct option that retains a real input vector.

The result is calculated column-wise over the selected [N,D] matrix, producing a vector of D features. It does not average the dimensions with one another, average probability outputs, or identify onsets/offsets. The demonstration is intentionally two-dimensional with fixed axes; for actual representations, apply the operation in the full D-dimensional space before projection. Coordinate-wise extrema and medians in a PCA/UMAP visualization are not generally the projection of pooling in the original feature space.

Mean: equal arithmetic weight for every selected frame. It can summarize central tendency, but every background or atypical frame also contributes. The recording example has 57 background frames out of 90. Do not describe the resulting vector as a call count or class probability.

Median: sort each coordinate independently; use its middle value, averaging the middle two for even N (NumPy convention). Coordinate donors can differ; the result is not the geometric median or necessarily an observed frame. It is less sensitive to an isolated extreme in a coordinate, but can suppress brief minority structure.

Max / Min: take the largest / smallest signed value in each coordinate. The outlined source frames and dashed lines show which coordinates supply the result. These operations can create a combination that never occurred in any individual frame. They do not choose the vector of maximum/minimum norm, the loudest/quietest frame, or the most/least confident call. An embedding coordinate is not a probability. They are sensitive to extremes and to the coordinate basis. No method here is universally best.

Medoid: choose an observed frame vector minimizing the sum of Euclidean distances to all selected frame vectors; equal costs are resolved by the smallest original frame index. It is not “the frame nearest to the mean” and not a coordinate-wise median. The direct implementation is quadratic in the number of frames and is appropriate for this small demonstration, not an unmodified large-scale pipeline.

Trim 10%: independently sort each coordinate, discard floor(0.1*N) values at each tail, and average what remains. The floor convention makes short intervals important: for fewer than ten frames this is the ordinary mean. Different coordinates may keep different input frames. Additional choices such as mean concatenated with standard deviation produce 2 × D features, not D; mean+SD still discards explicit within-interval ordering. Quantiles, weights, and the aggregation method should be selected using validation appropriate to the intended task.

Synthetic stress test: “Perturb one frame” replaces the middle selected frame with the invented vector [2.50, -1.02]. It changes only the toy embedding data, not the waveform or a real model. Labels and other frames remain fixed. The animation shows information being reduced, not embeddings being trained or moved by an optimizer. Frame coordinates are copied from the preceding synthetic fine-tuned illustration and re-expressed as signed toy features.

Scope of sources: the supplied animal2vec paper and supplement describe averaging across Transformer layers, and temporal pooling of label probabilities for event aggregation. They do not report this six-method comparison of temporal embedding summaries. This is an additional mathematical teaching demonstration. See Schäfer-Zimmermann et al. (2025), section 2.2.2 and supplemental “Finetuning animal2vec,” p. 8, for the distinct layer-average operation. See official NumPy mean, median, and max documentation for axis-wise statistics. The medoid and trimmed mean are defined explicitly above; no library defaults are assumed.

For real data, retain the same checkpoint, extraction policy and coordinate system when comparing summaries; filter padding and invalid rows. Reducers shown here are permutation-invariant over the already-computed vectors. This does not imply contextualized representations contain no temporal information. Storing a single summary does not let you recover individual frame trajectories. Preserve the source intervals separately.

animal2vec workshop · MPIABDay 2 · 14:15–15:00

What the built-in a2v routine returns

Attribute	Shape / type	Meaning
`result.embeddings`	`[T, D]`	Layer-averaged frame representations
`result.probabilities`	`[T, C]`	Independent sigmoid label scores
`result.timestamps`	`[T]`	Frame-center times in recording seconds
`result.events`	Tuple of intervals	Pooled, thresholded, fused predictions

Arrays are returned as CPU tensors. Exact T comes from the audio geometry and endpoint trimming.

animal2vec workshop · MPIABDay 2 · 14:15–15:00

Extract all three aligned arrays in one pass

from a2v2 import InferenceRunner
from a2v2.data import load_audio

runner = InferenceRunner("MODEL.native.pt")
waveform, sample_rate = load_audio("AUDIO.wav")
result = runner.run_tensor(waveform, sample_rate)

Z = result.embeddings.numpy()    # [T, D]
t = result.timestamps.numpy()    # [T], seconds
p = result.probabilities.numpy() # [T, C]
e = result.events                # [(S1, E2), ...] Tuple of intervals

Use the provided native frame checkpoint. Replace the two illustrative filenames.