Status and future plans

animal2vec2

A new project

Information field theory for collective behavior

Max Planck Institute of Animal Behavior

Department for the Ecology of Animal Societies

Communication and Collective Movement (CoCoMo) Group

animal2vec2

Current Status

animal2vec 1: We have predictions for all CCAS species, and are extending to other species

Meerkats: Current predictions are the best we can get with animal2vec
Coati: Current predictions are the best we can get with animal2vec
Hyena: Current predictions are almost the best and the best possible ones are underway (~ two weeks)
Sifaka: In the pipeline. Dataset is ready, waiting for compute resources to become available
Hyraxes, Birds, Cats, ...: Community adoption is happening. I've talked to people at NeurIPS using a2v

animal2vec 2: Code is ready, we are preparing for launching large-scale pretraining

Huge improvements for train and inference speed (2 times larger model in 1/3 of the time)
Dataset aggregation: Almost done

Everything from Xeno-Canto, iNaturalist, Tierstimmenarchiv (1.3M files -> 13k hours)

This covers 96% of all known bird-species

A large subset of Googles Audioset dataset (600 hours)

Animal and Environmental sounds, plus various misc. stuff (noise, soundscapes, motor sounds, ...)

10-15k hours of marine data (BlueFinLibrary, Sanctuary Soundscape Monitoring Project (SanctSound), The William A. Watkins Collection, The Orchive, ...)
Misc. other data (Egyptian fruit bats (Prat et al.), the VibroScape Dataset (Insects), The Hainan gibbon dataset, ...)
Selected datasets from MPIAB

New things in animal2vec 2.0: Nested Tensors

New things in animal2vec 2.0: Class token

This acts as a "summary" of the sequence

The "summary" embedding

The overal results (mAP scores)

Class	animal2vec 1	animal2vec 2
Focal	0.94	0.94
Close Call	0.94	0.93
Short-Note Call	0.92	0.93
Social Call	0.84	0.85
Other Call	0.50	0.53
Alarm Call	0.80	0.82
Aggressive Call	0.71	0.75
Move Call	0.61	0.63
Lead Call	0.50	0.47
Synch signal	0.98	0.98
Eating	0.87	0.86
Beep signal	0.80	0.77
Microaverage	0.91	0.92
Segment	n/a	0.96

Information field theory for collective behavior

Project idea for an ERC and DFG proposal

Information field theory for collective behavior

The Fundamental Challenge

How do complex patterns arise from simple components?

Lagrangian

Tracks individuals (particles)

Vicsek-, Couzin-, Cucker-like models
Behavioral heuristics -> no Hamiltonian
No statistical mechanics

Eulerian

Treats group as fluid

Toner-Tu-, Mogilner-Edelstein-like models
Continuous field equations
No individuals

From data to behavior

The general approach

The cookbook

Train the BNN on all data using all modalities

This is self-supervised learning without any labels

Isolate situations of interest, such as foraging or predator evasion

Those are situations we know beforehand, e.g., from video footage

Derive a social Hamiltonian for each of these situations

These will be learned by a Graph Neural Network fitting the behavioral states of the BNN

Recreate the situation using the information-field theory approach

How is each behavioral state from each individual affecting every other behavioral state from every other individual

Summary & Questions

I want to use audio, GPS, acc. data to infer behavioral states from them and model their temporal evolution using information-field theory

Are the CCAS species good systems for studying these collective behavioral transitions?
What situations can you think of that fit into this idea of collective behavior?
Is there something you would like to get analyzed using this approach?

Status and future plans

animal2vec2

A new project

Information field theory for collective behavior

Max Planck Institute of Animal Behavior

Department for the Ecology of Animal Societies

Communication and Collective Movement (CoCoMo) Group