Coherent Multi-Agent Trajectory Forecasting in Team Sports with CausalTraj

Accepted to the AI4TS workshop @ AAAI 2026
-->

Abstract

Jointly forecasting trajectories of multiple interacting agents is a core challenge in sports analytics and other domains involving complex group dynamics. Accurate prediction enables realistic simulation and strategic understanding of gameplay evolution. Most existing models are evaluated solely on per-agent accuracy metrics (minADE, minFDE), which assess each agent independently on its best-of-k prediction. However these metrics overlook whether the model learns which predicted trajectories can jointly form a plausible multi-agent future. Many state-of-the-art models are designed and optimized primarily based on these metrics. As a result, they may underperform on joint predictions and also fail to generate coherent, interpretable multi-agent scenarios in team sports. We propose CausalTraj, a temporally causal, likelihood-based model that is built to generate jointly probable multi-agent trajectory forecasts. To better assess collective modeling capability, we emphasize joint metrics (minJADE, minJFDE) that measure joint accuracy across agents within the best generated scenario sample. Evaluated on the NBA SportVU, Basketball-U, and Football-U datasets, CausalTraj achieves competitive per-agent accuracy and the best recorded results on joint metrics, while yielding qualitatively coherent and realistic gameplay evolutions.

Quantitative Benchmark

Quantitative benchmark on NBA SportVU dataset

Quantitative benchmark on NBA SportVU dataset.


Quantitative benchmark on Basketball-U and Football-U datasets

Quantitative benchmark on Basketball-U and Football-U datasets.

Best Scenario Sample Comparison

minJADE Scenario Sample Comparison

minJADE Scenario Sample Comparison on NBA SportVU dataset.

Random Scenario Samples

At each subsection, we show 3 samples of joint multi-agent trajectory forecasts generated by each model, based on a single historical context (first 2.0s of the video).

Context 1

MoFlow (joint objective)
CausalTraj (Causal PointNet)
CausalTraj (Mamba2)

Context 2

MoFlow (joint objective)
CausalTraj (Causal PointNet)
CausalTraj (Mamba2)

Context 3

MoFlow (joint objective)
CausalTraj (Causal PointNet)
CausalTraj (Mamba2)

Context 4

MoFlow (joint objective)
CausalTraj (Causal PointNet)
CausalTraj (Mamba2)

BibTeX Citation

@misc{teoh2025coherentmultiagenttrajectoryforecasting,
          title={Coherent Multi-Agent Trajectory Forecasting in Team Sports with CausalTraj},
          author={Wei Zhen Teoh},
          year={2025},
          eprint={2511.18248},
          archivePrefix={arXiv},
          primaryClass={cs.LG},
          url={https://arxiv.org/abs/2511.18248},
          }