animal2vec workshop · MPIABDay 1 · 09:15–10:35

animal2vec
without math

A conceptual walkthrough.

Julian C. Schäfer-Zimmermann

Max Planck Institute of Animal Behavior
Department for the Ecology of Animal Societies
Communication and Collective Movement (CoCoMo) Group

animal2vec workshop · MPIABDay 1 · 09:15–10:35

The difficult part is not an isolated clean call

Paper Fig. 1a, cropped: alarm calls amid scratching, bumping, and digging.

Find the vocalization.

Ignore the noise.

Learn the concept.

There is a lot of information in this example, not all of it is useful.

animal2vec workshop · MPIABDay 1 · 09:15–10:35

Rare can mean few events—or little time

Paper Fig. 2, cropped. Event counts and durations are different imbalances.

MeerKAT

1,068 h

184 h with complete event labels

Short-note calls are numerous but have a median duration of 37 ms.

Lead calls are long, but almost never happen

animal2vec workshop · MPIABDay 1 · 09:15–10:35

A neural network is a chain of learned transformations

Numbers inSamples from a microphone

→

Many adjustable stepsFilters and transformations

→

Numbers outRepresentations or scores

Before training

The adjustable values do not yet solve the task.

After training

The values have been changed using a training objective.

animal2vec workshop · MPIABanimal2vec without math

From waveform to prediction

1. Input waveform

A raw recording: amplitude changing over time.

→

2. Frontend

Learns local acoustic features from short neighborhoods.

→

3. Transformer

Builds contextual representations: each moment can depend on other moments.

→

4. Classification head

Turns representations into scores or probabilities for the task.

Current stage

Press Run animation to step through the pipeline.

animal2vec workshop · MPIABDay 1 · 09:15–10:35

A model frame is not a waveform sample

80,000 samples10 seconds at 8 kHz

→

Convolutional frontendStride reduces sequence length

→

About 2,000 framesOne learned vector per position

MeerKAT baseline: a nominal 5 ms frame hop. Always use the returned timestamps.

animal2vec workshop · MPIABDay 1 · 09:15–10:35

What does a convolutional layer do?

Core idea

Take a small window of neighboring values.
Ask whether that window looks like a pattern the model has learned to detect.
Slide the same detector across the whole signal.
Write down a new output value at each position.

Why this is useful for audio

Short acoustic motifs can occur anywhere in time.
The same learned detector can respond wherever that motif appears.
Early layers learn local patterns; later layers combine them into richer structure.

You do not need the equations to understand the role: a convolutional layer is a reusable local pattern detector.

animal2vec workshop · MPIABanimal2vec without math

A short animation: one detector sliding across the signal

Input sequence

Detector (kernel)

Output feature map

Current step

Press Run animation.

What to notice

The detector always has the same shape.
Only its position changes.
High output means: "this local region looks like the learned pattern."

animal2vec workshop · MPIABDay 1 · 09:15–10:35

Local evidence becomes contextual evidence

Local question

What pattern is present around this position?

Contextual question

How does that pattern relate to the rest of this segment?

Local feature sequenceOne vector at each position

→

Transformer layersExchange information across positions

→

Contextual embeddingsA representation at each position

animal2vec workshop · MPIABDay 1 · 09:15–10:35

Attention: which other positions matter here?

Move the selected frame. Which parts of the segment could change its interpretation?

animal2vec workshop · MPIABDay 1 · 09:15–10:35

From embedding to prediction

The embedding describes a moment in context. The head asks which labels apply.

READY

Context is already in the embedding. Watch the head turn each frame into label probabilities.

Inspect frame

Suggested narration, 1–2 minutes: “The transformer has already put information about other moments into each of these vectors. Pick one column: it belongs to one model frame. The head reads all its feature values and combines them differently for each label. The result is a raw score for each label. A sigmoid converts each score to a value between zero and one. Then we apply exactly the same head at the other time positions.”

During playback, the selected column moves through four frames. The connections keep the same weights; only the input vector and the resulting scores change. Processing the frames sequentially is a teaching device. The implementation can evaluate them together as a tensor operation; there is no recurrent state or feedback from one prediction to the next.

Use Step to pause on the vector, connections, raw scores or probabilities. After playback, inspect frame 1 (all displayed labels low), frame 2 (Call A high), frame 3 (Call A and Call B high together), and frame 4 (Call B high). These descriptions refer only to the illustrative calculation, not to real audio or a biological ground truth.

Interpretation: The six displayed coordinates are generic learned dimensions, not six known acoustic properties and not six classes. The head uses combinations of dimensions. The three output labels are independent sigmoid outputs: they need not sum to one. Multiple labels may overlap; all can also be low. There is no extra “noise” class unless one was explicitly included in the training vocabulary. These values are model probabilities, not guaranteed calibrated confidence estimates. Thresholding and event fusion are separate downstream steps, not part of the linear head.

Implementation detail, not required on screen: In the supplied workshop code, the frame head receives the mean of the configured final K transformer layer outputs at each time position. This is averaging over layers, NOT over time. At inference, final dropout is inactive, and the classifier computes z_t = W h_t + b. InferenceRunner applies sigmoid independently to each logit. A D-dimensional vector becomes C scores at every valid frame: T × D → T × C. This animation does not represent the optional recording-level CLS head.

Sources: supplied a2v2-workshop_codebase.zip, a2v2/model.py, Animal2VecFineTuningModel (classifier and forward); a2v2/workflows.py, InferenceRunner (sigmoid and layer-averaged embeddings). The paper’s layer averaging is described in section 2.2.2; overlapping MeerKAT labels in section 2.1. Schäfer-Zimmermann et al. (2025), animal2vec and MeerKAT, DOI: 10.1111/2041-210x.70218. The arrays used by this animation are synthetic explanatory values, not copied from a checkpoint.

animal2vec workshop · MPIABDay 1 · 09:15–10:35

Learn, specialize, apply

PretrainUnlabelled audio
Learn useful representations

→

Fine-tuneAnnotated audio
Learn the target labels

→

InferNew audio
Apply fixed weights

Only the first two stages change the learned model.

animal2vec workshop · MPIABanimal2vec without math

One pretraining step: student and teacher

Predict the teacher’s representations of hidden moments—not call labels.

READY

One recording, two views. The student predicts what the teacher represents at hidden time positions.

This slide needs JavaScript for playback. The teacher sees all features; the student predicts hidden-frame embeddings. The loss updates the student pathway, and an EMA of encoder weights then updates the teacher.

Suggested narration, 2–3 minutes. “There are no call labels in this step. First, the audio becomes a sequence of local features. We hide some of those features from the student, but not from the teacher. The student uses the visible moments as context; a decoder restores the missing positions and predicts a vector at each one. The teacher sees the whole sequence and produces the reference vectors. We compare the two at the hidden positions. That mismatch changes the student pathway. Finally, a different operation moves the teacher's weights a little toward the updated student's weights.”

1 — Audio: This is a schematic recording from a mini-batch, not a measured waveform. The supplied code can mix waveforms before computing a single shared set of projected local features. Figure 3 of the paper shows A-weighted stochastic mixing; this slide compresses augmentation into the audio input rather than suggesting two unrelated teacher/student recordings. The shared frontend includes the local convolutional encoder, feature normalization and projection.

2 — Mask: In the example, time positions 3, 4 and 7 are hidden. Masking is applied to the feature sequence, not by erasing the waveform. Dashed empty columns show the missing positions. These are not feature vectors sent to the student Transformer. On the next stage, only positions 1, 2, 5, 6 and 8 remain, visibly packed together while retaining their original time indices. The mask size is illustrative, not the recipe's actual mask probability.

3 — Student: The student contextualizes only the retained tokens. The decoder then restores the full time grid using placeholders at hidden positions and predicts embeddings. In the paper's Figure 3 these placeholders are Gaussian noise; in the workshop code mask_noise_std controls their noise, with zeros when it is zero. The small dashed decoder slots represent these placeholders, not leaked hidden features. This prediction decoder is NOT the call-classification head used in fine-tuning. The displayed vectors are illustrative, not a JavaScript implementation of the actual encoder/decoder.

4 — Teacher: The teacher contextualizes the complete, unmasked feature sequence. The workshop code constructs normalized targets by combining selected teacher layer outputs. The full projected features are detached, and the teacher forward pass runs under torch.no_grad() in evaluation mode. Thus neither teacher parameters nor the shared frontend receive gradients through this target branch. The target embeddings stay fixed for the current loss computation and update.

5 — Compare: Only the masked positions contribute to the frame-prediction regression. The same positions are outlined in the target and prediction rows. Visible positions remain available as context, but they receive no direct frame-regression credit. The paper describes mean-squared-error regression. In the workshop code, RegressionLoss uses scaled MSE when loss_beta=0, or smooth-L1 for a nonzero beta. We therefore label this box “mismatch / loss” rather than imply every possible configuration uses exactly the same loss. It compares embeddings—not raw waveforms, species names or human annotations.

6 — Learn: Gradients from this mismatch propagate through the decoder and student Transformer to the shared frontend. After the optimizer step succeeds, these trainable parameters have changed. The orange route is a learning signal, not another flow of audio or teacher labels. Teacher weights do not change in this stage. The tiny “weights” strips are illustrative parameter fingerprints; their update is not a simulated optimizer. The existing target and prediction graphics are deliberately NOT recomputed after the update. One step does not guarantee a smaller loss on every recording or make predictions equal targets.

7 — EMA: Only now is the teacher updated. Its new encoder weights combine most of the previous teacher weights with a small contribution from the updated student encoder weights: an exponential moving average. This is a WEIGHT average, not an average of predicted embeddings and not a gradient step on the teacher. The decoder is not averaged into the teacher. In the code, EMATeacher stores a float32 copy of the encoder and also copies its buffers; the teacher forward path shown here uses encode_projected on the shared detached features, rather than running a second waveform frontend. The fingerprint illustration uses a 0.90 EMA decay for visibility only; this is not a workshop hyperparameter recommendation. The step's already-computed targets remain fixed; future steps use the updated teacher.

Scope: This illustrates an arbitrary pretraining step, not necessarily the first. At initialization the teacher is a copy of the student. After earlier updates their weights can differ, as represented here. The animation follows the code's student-forward-then-teacher-forward order, although the two-branch diagram itself is not a wall-clock execution trace. “Step” means a successful optimizer update; multiple micro-batches may be accumulated before it. We show only one recording and one mask; clone_batch can create several masked views sharing a teacher target. The optional modern CLS token adds a separate representation-regression term, not a supervised class label; that auxiliary branch is omitted here. Detailed normalization, positional encodings, regularization and optimizer internals are also omitted.

Ask the audience: “Which network changes because of the loss? Which changes because of the weight average? What is the student trying to predict?” Expected answers: the student pathway including its frontend and decoder; the teacher encoder; the teacher's embeddings at hidden positions. The teacher is not a human annotator or an independently trained oracle.

Sources: Schäfer-Zimmermann et al. (2025), animal2vec and MeerKAT, §2.2.1 and Fig. 3, doi:10.1111/2041-210x.70218. Supplied workshop archive: a2v2/model.py, Animal2VecPretrainingModel.forward, ConvDecoder, EMATeacher.update, and Animal2VecPretrainingModel.update_teacher; a2v2/training.py, optimizer-update ordering. The drawing is an original teaching schematic based on these sources.

animal2vec workshop · MPIABanimal2vec without math

Fine-tuning: specialize what the model has learned

Load pretrained weights → train the new head → unfreeze → optimize together.

READY

Reuse the pretrained encoder. Learn a new label mapping, then adapt the contextual representations to the supervised task.

Purpose and timing: Follow the pretraining animation with this one. Pretraining learned a representation without call labels. Fine-tuning now uses human annotations to learn the actual task. Allow about two minutes to explain the complete animation. The five phases play in about twenty seconds; use Step or a numbered phase button to pause the explanation. Time is compressed differently across phases, but the update axis is proportional.

1 — Load pretrained weights: Start a new supervised run with the checkpoint's STUDENT encoder. The workshop loader selects keys prefixed student., removes that prefix, and loads the encoder strictly. This includes the local frontend and contextual encoder, not the teacher, pretraining decoder, regression module or old optimizer. The fine-tuning model adds a new nn.Linear classification head, with Xavier-uniform weights and zero bias. Its number and order of outputs follow task.unique_labels. Architecture, sample rate and convolution geometry must match the pretraining checkpoint. Loading a pretraining checkpoint is not the same as resuming an interrupted fine-tuning run.

2 — Warm-up, first 2,000 updates: Only the classification head receives parameter updates. Frozen components still compute features; freezing is not turning them off. The learning rate increases linearly from 1e-10 toward 3e-5. Call this LEARNING-RATE warm-up, not an automatic end to encoder freezing. The human-label route bypasses the encoder and goes directly to the supervised loss; labels are never part of the waveform input. The purpose of the animation is to distinguish which weights may change from the size of each optimizer step. The small coloured strips represent weights, not embeddings. Their changes are illustrative.

3 — Head-only adaptation, up to 10,000 updates: After learning-rate warm-up ends, the encoder stays frozen for another 8,000 updates. The classifier continues learning, now on the decaying part of the learning-rate schedule. In Animal2VecFineTuningModel.forward, the complete encoder forward is under torch.no_grad() while update < freeze_finetune_updates. This means encoder WEIGHTS remain unchanged. Do not say that every embedding must be numerically identical on every pass: training-time masking, dropout, layerdrop and mixed inputs can still vary the activations.

4 — Unfreeze at update counter 10,000: The code enables encoder gradients when update >= model.freeze_finetune_updates. The lock on the contextual encoder opens. Unfreezing itself does not change weights; a subsequent optimizer update is needed. Notice that the learning rate is already decaying and does not restart here. The new classification head remains trainable.

Exactly what is unfrozen: The supplied recipe has model.feature_grad_mult: 0.0. Consequently, _project keeps encoder.local_encoder under no_grad even after the outer freeze ends. Those are the local Sinc/convolution filters. project_norm, project_features, and the subsequent contextualizing encoder are outside that inner local freeze and can adapt. To keep this distinction visible, the diagram groups projection with the contextual encoder, not with the frozen local frontend. A positive feature-gradient multiplier would require a different diagram and configuration; it is not what this animation depicts.

5 — Joint / final optimization: Continue supervised updates of the contextual encoder AND the classifier. Gradients originate in the supervised loss; there is no EMA teacher. The learning rate decays by a cosine schedule toward its floor of 5e-6 as this recipe approaches optimization.max_update: 30000. The final stage is a continuation of optimization, not a separate optimizer or a second warm-up. The two displayed prediction bars are synthetic values for one illustrative frame, and the labels are fixed annotations. No empirical training-loss or validation curve is claimed. Changing weights does not guarantee monotonic improvement or perfect predictions.

Outputs and loss: The depicted task is the baseline frame classifier, not the optional modern recording-level CLS head. Selected Transformer-layer outputs are averaged across LAYERS at each time position (not across the recording). The new linear head supplies logits; the code uses sigmoid focal loss against the frame labels. Sigmoid probabilities can be shown to users, but the loss is implemented from logits. Multiple labels may overlap; there is no required explicit "noise" class. A/B are placeholders, not the complete 12-class MeerKAT vocabulary. Mixup, time/channel masking, dropout, focal-loss arithmetic, minibatch accumulation and AMP are omitted to keep the phase distinction legible.

Validation and checkpoint choice: Validation uses held-out labelled recordings without training updates. checkpoint_best.pt follows the configured validation metric; the supplied baseline uses metrics/finetune/f1. The last checkpoint is not necessarily the best model. Keep a separate untouched test set for the final performance claim. The animation's finish is not evidence of convergence or validated model quality.

Clock convention: Labels on the timeline denote the successful-update counter used by the training engine, not individual microbatches or epochs. The first 10,000 optimizer calls have frozen encoder weights. Encoder gradients are enabled for the next call made with counter 10,000. The scheduler is stepped after a successful update. The line plots the first-cycle rates used at counters 0 through 29,999, approaching the 30,000 stop boundary; it deliberately does not depict a possible next-cycle bookkeeping rate at counter 30,000 as another training step. The cursor caps at 29,999, while the completed caption reports the 30,000-update budget.

Audience question: "At update 5,000, which weights are learning? And at 15,000?" Expected answer: first only the classifier; later the contextual encoder and classifier, while local frontend weights are still frozen. "Does unfreezing reset the learning rate?" No, not in this recipe.

Source basis: Supplied a2v2-workshop_codebase.zip, configs/MeerKAT/finetune_mixup_100.yaml; a2v2/workflows.py::_load_pretrained_bundle; a2v2/model.py::Animal2VecFineTuningModel, _project and forward; a2v2/training.py::CosineUpdateScheduler. Scientific background: Schäfer-Zimmermann et al. (2025), animal2vec and MeerKAT, §2.2.2, doi:10.1111/2041-210x.70218. Precise timing comes from the supplied configuration; it should not be attributed to the brief main-text paragraph or generalized to all recipes. Framework lifecycle reference: reveal.js events.

animal2vec workshop · MPIABanimal2vec without math

Inference: from a recording to timed events

Each chunk yields embeddings and probabilities. Post-processing turns frame scores into timed events.

Prediction is not training

READYFollow one recording

Play the complete path, or select a stage. The controls below change post-processing—not model weights or embeddings.

Average window Threshold 0.175 Try “No pooling”.

This slide needs JavaScript for the animation. Inference: waveform → encoder → averaged layer embeddings → head → probabilities → average pool → threshold → contiguous events. Embeddings branch before the head.

Inference, event aggregation, and the exported embeddings

Suggested narration: “We now load the fine-tuned detector, not a pretraining-only checkpoint. No student–teacher training loop, labels, or weight updates are needed. For each chunk the frontend and Transformer produce contextual features. Averaging the selected Transformer layers keeps one embedding per time position. We can save those embeddings, or send the very same representation into the trained head. The head produces a probability for each label at each frame. Finally, we average nearby probabilities, apply a threshold, and read the starts and ends of contiguous active stretches.”

1 · Load: InferenceRunner loads a native checkpoint whose stage is finetune and whose classification_head is frame. It restores the frontend, contextual encoder and trained classifier together. eval() and inference_mode() keep weights fixed and disable training-time dropout, mixing and masking. The teacher and pretraining decoder have no role here. Select one channel or average channels, resample to the checkpoint's rate, and normalize each bounded segment. The default segment length is 10 seconds; segments do not overlap in this implementation. Attention context is limited to each segment. The small chart zooms into a constructed 1.2-second example, not the default chunk length.

2 · Embeddings: The frame path averages output.layer_outputs[-K:] with K limited to the number of available layers. The MeerKAT recipe uses all 16 layers; K is configurable. In eval mode this is the representation sent to the linear classifier because final_dropout is inactive. InferenceRunner recomputes that layer average from the same forward pass, removes an optional leading CLS token, adds absolute segment offsets to convolution-center timestamps, trims padded frames, and concatenates aligned embeddings, probabilities and timestamps. result.embeddings has shape [T,D], not [number of events,D]. It includes background frames. T is the number of frames, D the embedding width, and C the label count. The coloured tiles here are schematic; they are not six named acoustic measurements. Temporal averaging within a detected event would be a separate downstream analysis, not something event fusion does.

3 · Probabilities: Each label gets an independent sigmoid probability. The slide visualizes only one label for legibility; separate labels may overlap, and all labels can be low without an explicit background class. result.probabilities contains these RAW, unpooled frame probabilities, shape [T,C]. The code does not return its pooled probabilities as a separate InferenceResult field. It computes pooled probabilities internally for result.events.

4 · Average pooling: Follow the 100 ms sliding window. At the MeerKAT nominal 200 Hz frame rate this is 20 frames. In the supplied implementation an even-width window includes 9 earlier positions, the current position, and 10 later positions. At a recording edge it averages valid values only, not zero padding. The JavaScript follows those rules. The short isolated spike is reduced; short below-threshold interruptions within the two constructed call regions are bridged at the recipe defaults. This is averaging before binarization, NOT a rule that automatically fills every gap shorter than 100 ms. Window size and threshold may remove brief calls, join nearby calls, or shift estimated boundaries. The default 0.175 threshold comes from the workshop YAML, not from the supplement's prose.

5 · Binarize: A frame is active when its pooled score is greater than OR EQUAL TO the threshold. A 0 → 1 transition starts an event; a 1 → 0 transition ends the run. These are threshold-derived boundary estimates, not the location of a waveform peak, and not proof of the exact biological onset. The binary path is constructed from frame runs; it does not interpolate threshold crossings between frames.

6 · Onsets, offsets, scores: For the maximal true run [s,e), the code uses start_seconds=max(0,timestamps[s]) and end_seconds=min(recording_duration,timestamps[e-1]+hop), where hop is the median adjacent frame-timestamp difference. Thus the end index e is exclusive. A run active at the beginning or end is also emitted; it does not need a detected transition inside the recording. The first timestamp follows the frontend's actual receptive-field center, not an assumed zero. In this toy grid the first center is 41.5/8000 seconds and the hop is 40/8000=5 ms. Times in the table are rounded to milliseconds; internal calculations retain full precision. Each exported score is the mean of the POOLED probabilities over that run, not its peak. The event rows are computed for every detected span without annotations. Changing only the fusion parameters leaves the stored frame embeddings and raw probabilities unchanged.

Keep inference and evaluation separate: The supplied supplemental material, “Calculating event-based metrics” (pp. 8–9), next compares predicted spans against ground truth, describes IoU greater than 0.5, assigns average model likelihood within a predicted span, and counts false positives/negatives. Those operations evaluate detections; they do not decide whether the model can emit an event on an unlabelled recording. The runner's score is specifically a mean POOLED probability for every event. Also, the supplied MeerKAT YAML has evaluation iou_threshold=0.0, unlike the supplement's 0.5. These are separate source statements, not silently reconciled here. No metric score, IoU matching policy, or accuracy result is simulated by this slide.

Discussion prompt: “What changes when we switch off pooling? What stays exactly the same?” Expected answer: the detected intervals, count and event scores can change; the fine-tuned weights, contextual frame embeddings and original sigmoid probabilities do not. Then ask whether a 100 ms window is appropriate for the shortest calls in a participant's own study.

Sources: Schäfer-Zimmermann et al. (2025), animal2vec and MeerKAT, SI “Finetuning animal2vec” and “Calculating event-based metrics”, Fig. S5 caption, pp. 8–9. Attached code snapshot 5654c569172e6da0f043c22ff43f39104b8a03c8: a2v2/workflows.py (InferenceRunner, pool_probabilities, fuse_probabilities); a2v2/model.py (Animal2VecFineTuningModel.forward); a2v2/data.py (feature_timestamps); configs/MeerKAT/finetune_mixup_100.yaml. Framework lifecycle follows reveal.js events.

Playback: Approximately 24 seconds at data-speed="1". The animated scan explains operations that the actual implementation computes in parallel. Play/Pause and Step are independent of reveal.js navigation. Reduced-motion users start with a static, complete overview and can step manually. No real model inference or training runs in this browser.

animal2vec workshop · MPIABDay 1 · 09:15–10:35

What did the original study demonstrate?

Paper Fig. 4, cropped: precision–recall curves on MeerKAT.

1% of fine-tuning labels

Micro-average AP: 0.83

100% of fine-tuning labels

Micro-average AP: 0.91

Pretraining

Pretraining used the full 1,068 h.

animal2vec workshop · MPIABDay 1 · 09:15–10:35

Paper, workshop package, and “modern” recipe

Name	Meaning in these slides
animal2vec, original paper	Scientific method and reported experiments
`a2v2` Python package	The supplied native-PyTorch workshop implementation
`configs/MeerKAT/`	Animal2Vec 1.0 reproduction-oriented configurations
`configs/modern/`	A different, opt-in architecture/checkpoint family

animal2vec workshop · MPIABDay 1 · 09:15–10:35

Explain the pipeline to a colleague

A new WAV file enters a fixed checkpoint.
What happens next?

Explain

Samples → local features → contextual embeddings → scores → intervals.

Distinguish

Learning weights, selecting a threshold, and interpreting a prediction.

Two minutes in pairs. Identify one remaining uncertainty.

animal2vec workshop · MPIABDay 1 · 09:15–10:35

Next: make a prediction you can inspect

SelectCompatible frame checkpoint + WAV

→

RunRead the CLI arguments

→

InspectAudio + predicted intervals

Keep the checkpoint, settings, source filename, and output together.

After the break: prepared CoCoMo environment → CLI tour → first inference.