Skip to content
Networked-System-and-Security-GroupPublic

About

APNet'25 - Mnemosyne: Lightweight and Fast Error Recovery for LLM Training in a Just-In-Time Manner

Topics

Resources

Stars

16 stars

Watchers

1 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Mnemosyne

A lightweight, fast, and transparent error recovery framework for LLM training in a just-in-time manner.

Overview.

Figure: The workflow of Mnemosyne. The left part shows the steady-state work, and the right part shows the recovery process.

Advantages

  • ⚡️ Transparent and periodic-checkpointing-free recovery inherited from Microsoft's just-in-time checkpointing.
  • ✂️ Tailed mechanisms for error recovery to minimize daily overhead.
  • 🛠️ Support for runtime adjustment of collective communication groups.

Note

The prototype of device proxy can be found in the device-proxy directory, and the prototype of flexible CCL can be found in the flexible-ccl directory.

Cite

Your citations are greatly appreciated. 🥰

  • ACM Reference

    Jinyi Xia, Menghao Zhang, Jiaxun Huang, Yuezheng Liu, Xiaohe Hu, Xudong Liu, and Chunming Hu. 2025. Mnemosyne: Lightweight and Fast Error Recovery for LLM Training in a Just-In-Time Manner. In Proceedings of the 9th Asia-Pacific Workshop on Networking (APNET '25). Association for Computing Machinery, New York, NY, USA, 157–163. https://doi.org/10.1145/3735358.3735372

  • Bibtex
    @inproceedings{10.1145/3735358.3735372,
        author    = {Xia, Jinyi and Zhang, Menghao and Huang, Jiaxun and Liu, Yuezheng and Hu, Xiaohe and Liu, Xudong and Hu, Chunming},
        title     = {Mnemosyne: Lightweight and Fast Error Recovery for LLM Training in a Just-In-Time Manner},
        year      = {2025},
        isbn      = {9798400714016},
        publisher = {Association for Computing Machinery},
        address   = {New York, NY, USA},
        doi       = {10.1145/3735358.3735372},
        booktitle = {Proceedings of the 9th Asia-Pacific Workshop on Networking},
        pages     = {157-163},
        numpages  = {7},
        series    = {APNET '25}
    }

About

APNet'25 - Mnemosyne: Lightweight and Fast Error Recovery for LLM Training in a Just-In-Time Manner

Topics

Resources

Stars

16 stars

Watchers

1 watching

Forks

Releases

Contributors

Languages