A start time, an end time, and a label.
One or more labels for a recording.
Neighbours in a learned representation.
Candidates for human review.
Choose one of these questions for your own recordings.

Find the vocalization.
Ignore the recorder.
Keep the timing.
Which strong acoustic events in this figure are not the target vocalizations?

184 h with complete event labels
A short-note call has a median duration of 37 ms.
The adjustable values do not yet solve the task.
The values have been changed using a training objective.
When you run the same checkpoint on a new WAV file, which box is absent?
amplitude samples per second
10 seconds → 80,000 samples
Useful for us to inspect frequencies and timing. It is not the input image expected by this model.

Look locally first.
A SincNet-style filterbank learns frequency bands. Convolutions turn nearby samples into a shorter sequence of feature vectors.
A feature vector is a list of numbers describing a local part of the signal.
What pattern is present around this position?
How does that pattern relate to the rest of this segment?
Interactive example requires JavaScript.
Move the selected frame. Which parts of the segment could change its interpretation?
Illustration: rows are time positions; columns are learned dimensions.
A description in numbers.
The classification head reads this description and produces scores for the trained labels.
Embedding ≠ call label.
| Output label | Illustrative score | Interpretation |
|---|---|---|
| Close call | 0.82 | Evidence for this call type |
| Alarm call | 0.07 | Little evidence for this label |
| Focal | 0.76 | The caller may be the focal animal |
Illustrative numbers, not measured checkpoint outputs.
| Representation / output | One row or answer per… | What it supports |
|---|---|---|
| Frame scores | Model position × label | When is label evidence high? |
| Fused events | Contiguous active interval | Where does a predicted event start/end? |
| Recording-level labels | Recording × label | Which labels occur somewhere? |
| Embeddings | Model position × feature | How can sounds be represented or compared? |
Interactive example requires JavaScript.
Use the much larger pool of unlabelled recordings.
The paper uses between-classes learning to expose the model to mixtures.
Focal loss reduces the emphasis on already-easy supervised examples.
These are learning choices—not filters you turn on while running an existing model.
Define meaningful labels and annotate representative data.
Learn how those labels relate to the audio.

Micro-average AP: 0.83
Micro-average AP: 0.91
Pretraining used the full 1,068 h.
MeerKAT experiments and the separately pretrained birdsong transfer experiments.
Performance on recordings representative of your species, site, devices, and target labels.
Name one way your data differ from the data used to train the model you plan to use.
| Name | Meaning in these slides |
|---|---|
| animal2vec, original paper | Scientific method and reported experiments |
a2v2 Python package | The supplied native-PyTorch workshop implementation |
configs/MeerKAT/ | Animal2Vec 1.0 reproduction-oriented configurations |
configs/modern/ | A different, opt-in architecture/checkpoint family |
Interactive example requires JavaScript.
At which stage do we obtain embeddings? At which stage can a threshold change the result?
Inference resamples to the checkpoint’s rate. It cannot restore frequencies lost by that conversion.
The default averages channels. A selected channel uses a zero-based index.
A checkpoint only scores the labels its output head was trained to predict.
Could hide rare calls, interactions, or behavioural contexts.
Could inflate call counts and create unnecessary review work.
Inspect false negatives as well as the events the model already found.
A new WAV file enters a fixed checkpoint.
What happens next?
Samples → local features → contextual embeddings → scores → intervals.
Learning weights, selecting a threshold, and interpreting a prediction.
Two minutes in pairs. Identify one remaining uncertainty.
After the break: prepared CoCoMo environment → CLI tour → first inference.
Interactive example requires JavaScript.
a2v2-infer MODEL.native.pt AUDIO.wav events.tsv \
--threshold 0.50 --method avg --fusion-window-seconds 0 Raise the threshold: which weak events vanish, and which runs fragment?
Interactive example requires JavaScript.
a2v2-infer MODEL.native.pt AUDIO.wav events.tsv \
--threshold 0.50 --method max --fusion-window-seconds 0.10 Average can attenuate short peaks. Maximum can extend or merge nearby events. Inspect both timing and false positives.
project/
wav/clip01.wav
lbl/clip01.h5
manifests/train.tsv
manifests/valid.tsv WAV: the recorded signal
HDF5: sample-index event intervals
TSV: files in each split
train.tsv · first line: audio root · remaining fields separated by a literal TAB
/datasets/project
wav/clip01.wav 80000 The row means 80,000 waveform samples—not 80,000 model frames. Set task.data to the manifest directory.
# Toy vocabulary: task.unique_labels = ["call_A", "call_B"]
# 8 kHz audio; call_A from 1.00 s up to (not including) 1.15 s
start_frame_lbl = [8000]
end_frame_lbl = [9200]
lbl_cat = [0] Same length. Starts inclusive; ends exclusive. Label indices follow the configured vocabulary.
A fully reviewed no-event file can have empty arrays. Missing annotations are not known negatives.
Training requires the configured audio rate. min_label_size filters HDF5 file bytes; 3032 is not a minimum number of calls.
| What to edit | Relevant fields | What to preserve / coordinate |
|---|---|---|
| Data and task | task.data, unique_labelsdataset.train_subset / valid_subset | Label order, complete annotations, held-out groups |
| Starting model | model.w2v_pathtask.sample_rate / conv_feature_layers | Rate and frontend must match the pretrained checkpoint |
| Compute and output | distributed_world_size, max_tokensupdate_freq, checkpoint.save_dir | Match the actual launch; use a new run directory |
| Short-run timing | max_update, freeze_finetune_updateswarmup, validation and saving intervals | A 200-update run must not inherit a 10,000-update freeze schedule |
max_tokens is a waveform-sample budget, not a file count.[T, D]
T time positions; D learned features.
Useful for comparison and downstream analysis.
[T, C]
The same T positions; C trained labels.
Useful for the supervised detection task.
Illustrative matrix; not a measured activation pattern.
Dimension 42 is not automatically “pitch.”
Information may be distributed across many dimensions. A coordinate can encode mixtures of signal and context.
What would you need to show before calling a feature “individual identity”?
| Attribute | Shape / type | Meaning |
|---|---|---|
result.embeddings | [T, D] | Layer-averaged frame representations |
result.probabilities | [T, C] | Independent sigmoid label scores |
result.timestamps | [T] | Frame-center times in recording seconds |
result.events | Tuple of intervals | Pooled, thresholded, fused predictions |
Arrays are returned as CPU tensors. Exact T comes from the audio geometry and endpoint trimming.
| Unit | Question it can support | What may be lost or mixed |
|---|---|---|
| Frame | How does the sound change over time? | Many strongly correlated observations |
| Annotated or predicted event | How do individual calls compare? | Within-call temporal order after pooling |
| Fixed window | What is in a standard duration of audio? | Calls and background in the same window |
| Whole recording | How do broad soundscapes compare? | Rare brief events can be diluted |
Interactive example requires JavaScript.
Can these two frame sequences have the same mean but different biological interpretations?
Learned to match teacher targets from unlabelled audio.
Further shaped by the chosen supervised labels.
A native fine-tuning checkpoint with a frame head.
Returns scores, embeddings, times, and events.
A pure pretraining checkpoint or the checked-in modern CLS classifier.
Do not pass either to the event runner.
The archive has no dedicated embedding-export CLI. Use the Python API in the practical session.
from a2v2 import InferenceRunner
from a2v2.data import load_audio
runner = InferenceRunner("MODEL.native.pt")
waveform, sample_rate = load_audio("AUDIO.wav")
result = runner.run_tensor(waveform, sample_rate)
Z = result.embeddings.numpy() # [T, D]
t = result.timestamps.numpy() # [T], seconds
p = result.probabilities.numpy() # [T, C] Use the provided native frame checkpoint. Replace the two illustrative filenames.
| Record this | Why it matters |
|---|---|
| Checkpoint identity and hash | Different learned coordinate systems |
| Layers and temporal aggregation | Different definitions of the representation |
| Sample rate, channel, segment length | Different input and context |
| Source file and interval | Return to the actual audio |
Find examples near a known call or acoustic pattern.
Inspect structure, candidate groups, and unusual observations.
Train a separate classifier or regression model on labelled outcomes.
Retrieval and exploration can be unlabelled. A labelled downstream classifier is supervised learning.
Choose one unit and one metadata field you will inspect first.
| row_id | source_file | start_s | end_s | unit | context |
|---|---|---|---|---|---|
| event_001 | rec_017.wav | 12.30 | 12.48 | event | 10 s segment |
| event_002 | rec_017.wav | 16.10 | 16.22 | event | 10 s segment |
| event_003 | rec_104.wav | 4.80 | 5.16 | event | 10 s segment |
Illustrative schema—not actual participant recordings.
One interval per call. Compare examples while tracking individual, session, and detector origin.
One fixed window per observation. Keep duration and sampling policy consistent.
Which level will your result generalize to: another frame, recording, individual, or site?
# Z: [T, D]; t: frame-center timestamps in recording seconds
# start_s and end_s come from a declared annotation/detection policy
inside = (t >= start_s) & (t < end_s)
if not inside.any():
raise ValueError("No frame centers fall inside this interval")
z_event = Z[inside].mean(axis=0) # [D] Very short intervals may contain no frame center: flag them, rather than silently producing NaNs.
Will be absent from a prediction-selected repertoire.
May appear as a convincing “new cluster.”
Store both the source interval and the extraction segment/context policy.
| Operation | What it changes | Use it deliberately |
|---|---|---|
| Waveform normalization | The model’s audio input | Part of the extraction pipeline |
| L2-normalize each vector | Each observation’s overall vector length | Makes direction, rather than magnitude, central |
| Scale each feature column | Relative weight of different dimensions | Changes distances and fitted projections |
Interactive example requires JavaScript.
For your question, could vector magnitude carry useful information—or a nuisance factor?
features
row_id → vector
metadata
row_id → file, interval,
individual, site,
device, session Check the join.
Are IDs unique?
Did any rows disappear?
Are there unexpected duplicates?
Can three random rows be traced to audio?
Keep missing metadata explicitly missing; do not invent individuals, sites, or devices.
Checkpoint hash, package snapshot, layer policy, rate, channel, segment length, aggregation.
row_id, source file, interval, annotation or detector origin, threshold/fusion settings.
analysis_bundle/
embeddings.npy # rows identified by rows.csv
rows.csv # row_id, source interval, metadata
extraction.json # checkpoint and processing provenance Proposed export convention for the practical—not a built-in CLI output schema.
Useful temporal detail.
Not many independent calls.
Useful repeated observations.
Not many independent sites.
Interactive example requires JavaScript.
Reveal the colours by call group, site, and device. Which interpretation survives?
A method comparison is uninterpretable when the underlying observations also change unnoticed.
| Check | Required answer |
|---|---|
| What is a row? | One declared frame, event, window, or recording |
| Which space? | One checkpoint and one extraction/preprocessing policy |
| Where is the sound? | Source file + interval + stable row ID |
| What could confound similarity? | At least two biological or technical metadata fields |
Pick one query observation. Explain what would count as a useful neighbour.
| Your question | First tool | Its immediate output |
|---|---|---|
| What resembles this example? | Nearest neighbours | A ranked set of observations |
| What directions contain variation? | PCA | Linear coordinates and explained variance |
| What is locally neighbouring? | UMAP | A neighbourhood-oriented layout |
| Can examples be grouped? | Clustering | Candidate assignments |
| Which examples are unusual? | Outlier scoring | A ranking for inspection |
Interactive example requires JavaScript.
In your real dataset: inspect the audio, remove self-matches, and avoid near-duplicate segments.
Interactive example requires JavaScript.
Which differences appear along each axis?
How much variance do these axes retain?
Large variation is not automatically important biology.
Two dimensions need not preserve all useful distinctions.
Interactive example requires JavaScript.
The graph below is a schematic. No UMAP fit is computed in this slide.
Explore local neighbourhoods.
Inspect examples.
Compare metadata and parameter choices.
Biological category count.
Meaning of empty gaps.
Original-space density or global distance.
Interactive example requires JavaScript.
Cluster assignments are fitted in the 12-D feature space. The PCA coordinates only display them.
Interactive example requires JavaScript.
Grey observations are marked noise. “Noise” here is an algorithmic status—not a biological annotation.
| Inspect | Why |
|---|---|
| Typical examples and boundary cases | A centroid alone can hide variation |
| Individual / session / device / site | Technical or sampling factors may explain the group |
| Different parameters and resamples | A fragile partition may not support a stable interpretation |
| Independent annotated observations | Test the proposed biological interpretation |
What observation would make you reject your current interpretation of a cluster?
Interactive example requires JavaScript.
Rare vocalization? Corrupted file? Unusual background? Annotation mistake? Listen before naming it.
Inspect diverse and underrepresented regions.
Do not review only what the current model considers unusual.
Interactive example requires JavaScript.
State a hypothesis with no labels. Then reveal site, device, and call group in that order.
| Decision to perturb | Question to ask |
|---|---|
| Model / layer choice | Does the proposed structure depend on one representation? |
| Aggregation / normalization / metric | Does the observation definition change the conclusion? |
| Projection / clustering settings / seed | Is the visible or assigned structure stable? |
| Held-out sessions / individuals / sites | Does the interpretation survive a new context? |
Describe the structure.
Inspect examples.
Write a testable hypothesis.
Compare alternative explanations.
Revise the hypothesis.
Identify the next validation step.
A neighbour is a retrieval result.
A cluster is a candidate grouping.
An outlier is a review priority.
Which analysis will you try first—and what would convince you that it is useful?