DeepLabCut
| DeepLabCut | |
|---|---|
 DeepLabCut logo | |
| Developers | Alexander Mathis and Mackenzie Weygandt Mathis |
| Release | 2018 |
| Written in | Python |
| Type | Pose estimation, computer vision |
| License | LGPL-3.0 |
| Website | Gitbub Profile |
DeepLabCut is an open-source Python toolbox for 2D and 3D markerless pose estimation and behavioral tracking using deep learning and computer vision. It enables researchers to track user-defined body parts in videos without attaching physical markers, supporting quantitative studies of animal behaviour, neuroscience and biomechanics.[1][2]
History
[edit]DeepLabCut was introduced by Mathis and colleagues in Nature Neuroscience in 2018. The study adapted deep-learning methods developed for human pose estimation to biological experiments and demonstrated that user-defined landmarks could be tracked using relatively few manually labelled frames.[1] The approach was discussed in a contemporary commentary on automated behavioural tracking in Nature Neuroscience.[3]
In 2024, Nature published a news feature examining DeepLabCut's development and adoption by researchers.[4]
Methodology
[edit]DeepLabCut uses supervised learning. Researchers manually annotate anatomical landmarks in selected video frames, train a neural network and use the resulting model to predict landmark positions in new images.[1][2] The predicted coordinates can be used to quantify posture, locomotion, joint angles and movement trajectories.
The platform uses convolutional neural networks (CNNs), which learn spatial features from images. One supported network backbone is ResNet-50, a 50-layer residual neural network introduced by He and colleagues in 2016.[5] Its shortcut connections facilitate the training of deep neural networks. In DeepLabCut, the backbone extracts visual features, while prediction layers generate confidence maps representing the likely locations of labelled body parts.[1]
ResNet-50 is not the only architecture supported by DeepLabCut. Other backbone architectures have included MobileNetV2 and EfficientNet, which provide different trade-offs between computational requirements and model performance.[2][6]
Pose-estimation accuracy depends on the quality and diversity of the training annotations, as well as recording conditions. Occlusion, motion blur and unfamiliar postures can produce inaccurate predictions, making model evaluation and validation important parts of the workflow.[2]
Applications and impact
[edit]DeepLabCut has been applied to the analysis of locomotion, reaching, whisker movement and other animal behaviours.[1][2] Its three-dimensional workflow enables researchers to estimate landmark positions across multiple camera views.[2]
In 2022, Lauer and colleagues introduced a multi-animal pose-estimation, identification and tracking framework in Nature Methods. The method addressed challenges including overlapping animals and assigning detected body parts to individual subjects across video frames.[7]
In 2024, Ye and colleagues introduced SuperAnimal pretrained pose-estimation models in Nature Communications, aiming to improve the generalisation of animal pose estimation across species and recording conditions.[8]
DeepLabCut and its developers have received coverage in scientific and mainstream media, including The Atlantic, Bloomberg and Nature.[9] Alexander Mathis and Mackenzie W. Mathis were recognised with the Eric Kandel Young Neuroscientists Prize for their contributions to computational behavioural neuroscience; the award ceremony took place in June 2024.[10]
See also
[edit]References
[edit]- 1 2 3 4 5 Mathis, Alexander; Mamidanna, Pranav; Cury, Kevin M.; Abe, Tomaso; Murthy, Vivek N.; Mathis, Mackenzie W.; Bethge, Matthias (2018). "DeepLabCut: markerless pose estimation of user-defined body parts with deep learning". Nature Neuroscience. 21: 1281–1289. doi:10.1038/s41593-018-0209-y.
- 1 2 3 4 5 6 Nath, Tanmay; Mathis, Alexander; Chen, Abigail C.; Patel, Amar; Bethge, Matthias; Mathis, Mackenzie W. (2019). "Using DeepLabCut for 3D markerless pose estimation across species and behaviors". Nature Protocols. 14: 2152–2176. doi:10.1038/s41596-019-0176-0.
- ↑ Wei, Kevin; Kording, Konrad P. (2018). "Behavioral tracking gets real". Nature Neuroscience. 21: 1146–1147. doi:10.1038/s41593-018-0215-0. PMC 9339263.
- ↑ Madhusoodanan, Jyoti (2024). "DeepLabCut: the motion-tracking tool that went viral". Nature. 629: 960–961. doi:10.1038/d41586-024-01474-x.
- ↑ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2016). "Deep Residual Learning for Image Recognition". Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778. doi:10.1109/CVPR.2016.90.
- ↑ Sandler, Mark; Howard, Andrew; Zhu, Ming; Zhmoginov, Andrey; Chen, Liang-Chieh (2018). "MobileNetV2: Inverted Residuals and Linear Bottlenecks". Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4510–4520. arXiv:1801.04381. doi:10.1109/CVPR.2018.00474.
- ↑ Lauer, Jens; et al. (2022). "Multi-animal pose estimation, identification and tracking with DeepLabCut". Nature Methods. 19: 496–504. doi:10.1038/s41592-022-01443-0. PMC 9007739.
- ↑ Ye, Shunyu; et al. (2024). "SuperAnimal pretrained pose estimation models for behavioral analysis". Nature Communications. doi:10.1038/s41467-024-48792-2.
- ↑ Madhusoodanan, Jyoti (2024). "DeepLabCut: the motion-tracking tool that went viral". Nature. 629: 960–961. doi:10.1038/d41586-024-01474-x.
- ↑ "Eric Kandel Young Neuroscientists Prize". Gemeinnützige Hertie-Stiftung. Retrieved 23 October 2026.
