Issues

• Multimodal MoE

  help wantedHigh Priority
  xrsrke/pipegoose#61 · Filter by author xrsrke opened on Dec 11, 2023
  

• Distributed CLIP

  help wanted
  xrsrke/pipegoose#60 · Filter by author xrsrke opened on Dec 10, 2023

• DiLoCo replication (DiLoCo: Distributed Low-Communication Training of Language Models)

  help wantedHigh Priority
  xrsrke/pipegoose#59 · Filter by author xrsrke opened on Dec 10, 2023
  

• Making pipeline parallelism compatible with transformers

  xrsrke/pipegoose#52 · Filter by author xrsrke opened on Dec 1, 2023
  

• End-to-end FP8 training

  help wantedHigh Priority
  xrsrke/pipegoose#45 · Filter by author xrsrke opened on Nov 26, 2023
  1

  

• Automatic module mapping using torch.fx

  good first issue
  xrsrke/pipegoose#40 · Filter by author xrsrke opened on Nov 22, 2023 · v1
  3

  

• Tensor Parallelism

  xrsrke/pipegoose#37 · Filter by author 3outeille opened on Nov 20, 2023

• Distributed Logger

  good first issue
  xrsrke/pipegoose#33 · Filter by author xrsrke opened on Nov 14, 2023
  1

  

• Save and load checkpoints

  good first issuehelp wanted
  xrsrke/pipegoose#29 · Filter by author xrsrke opened on Nov 8, 2023
  

• Lazy initialization of massive models

  help wanted
  xrsrke/pipegoose#25 · Filter by author xrsrke opened on Oct 27, 2023
  1

• Sequence Parallelism

  xrsrke/pipegoose#22 · Filter by author xrsrke opened on Oct 25, 2023
  1

  

• Mixture of Experts

  xrsrke/pipegoose#19 · Filter by author xrsrke opened on Oct 25, 2023
  

• Trainer

  good first issue
  xrsrke/pipegoose#18 · Filter by author xrsrke opened on Oct 25, 2023
  4

  

• Implement new tensor parallelism technique

  help wanted
  xrsrke/pipegoose#17 · Filter by author xrsrke opened on Oct 25, 2023

• Setup documentation

  good first issuehelp wanted
  xrsrke/pipegoose#16 · Filter by author xrsrke opened on Oct 25, 2023

• Mixed precision training in FP16

  help wanted
  xrsrke/pipegoose#14 · Filter by author xrsrke opened on Oct 25, 2023

• Fused Optimizer

  xrsrke/pipegoose#13 · Filter by author xrsrke opened on Oct 25, 2023
  8

  

• Kernel Fusion using torch.jit

  good first issue
  xrsrke/pipegoose#10 · Filter by author xrsrke opened on Oct 25, 2023
  3

  

• Port CUDA Kernels

  good first issuehelp wanted
  xrsrke/pipegoose#8 · Filter by author xrsrke opened on Oct 25, 2023
  6

  

• Implement new pipeline parallelism technique

  High Priority
  xrsrke/pipegoose#7 · Filter by author xrsrke opened on Oct 25, 2023
  

• Model partitioning for pipeline parallelism

  xrsrke/pipegoose#6 · Filter by author xrsrke opened on Oct 25, 2023 · v1
  1

  

• Bucket small tensors and collective operations into larger ones

  xrsrke/pipegoose#5 · Filter by author xrsrke opened on Oct 25, 2023 · v1
  

• Gradient Checkpointing

  xrsrke/pipegoose#4 · Filter by author xrsrke opened on Oct 25, 2023
  1