add 3d

README.md

@@ -1,6 +1,6 @@
 # 🚧 PipeGoose: Train 🤗 `transformers` in 3D parallelism - WIP
 
-[<img src="https://img.shields.io/badge/license-MIT-blue">](https://github.com/xrsrke/pipegoose) [![tests](https://github.com/vwxyzjn/cleanrl/actions/workflows/tests.yaml/badge.svg)](https://github.com/xrsrke/pipegoose/actions/workflows/tests.yaml) [<img src="https://img.shields.io/discord/s9ZS9VXZ3p?label=discord">](https://discord.gg/s9ZS9VXZ3p) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[<img src="https://img.shields.io/badge/license-MIT-blue">](https://github.com/xrsrke/pipegoose) [![tests](https://github.com/vwxyzjn/cleanrl/actions/workflows/tests.yaml/badge.svg)](https://github.com/xrsrke/pipegoose/actions/workflows/tests.yaml) [<img src="https://img.shields.io/discord/767863440248143916?label=discord">](https://discord.gg/s9ZS9VXZ3p) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
 
 
 ![pipeline](3d-parallelism.png)
@@ -33,8 +33,7 @@ pipeline.fit(dataloader)
 
 **Implementation Details**
 
-- Supports training `transformers` model.
-- Supports ZeRO-1.
+- Supports training `transformers` model in Megatron 3D parallelism and ZeRO-1 (write from scratch).
 - Implements parallel compute and data transfer using separate CUDA streams.
 - Gradient checkpointing will be implemented by enforcing virtual dependency in the backpropagation graph, ensuring that the activation for gradient checkpoint will be recomputed just in time for each (micro-batch, partition).
 - Custom algorithms for model partitioning with two default partitioning models based on elapsed time and GPU memory consumption per layer.