Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support deepspeed sequence parallel #31525

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Support deepspeed sequence parallel #31525

wants to merge 15 commits into from

Commits on Jun 20, 2024

  1. Add original deepspeed

    @zeyugao
    zeyugao committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    6d6fb4b View commit details
    Browse the repository at this point in the history

  2. Support override the seed_worker in Trainer

    @zeyugao
    zeyugao committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    b5f054c View commit details
    Browse the repository at this point in the history

  3. Add some necessary check on sequence parallel argument

    @zeyugao
    zeyugao committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    6f72b8a View commit details
    Browse the repository at this point in the history

  4. Add DistributedAttention

    @zeyugao
    zeyugao committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    e311c0b View commit details
    Browse the repository at this point in the history

  5. Add starcoder2 as sequence parallel supported

    @zeyugao
    zeyugao committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    f2a6cc9 View commit details
    Browse the repository at this point in the history

  6. Add llama, mistral 

    Raise exception when sdpa
    @zeyugao
    zeyugao committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    42fd905 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Move DistributedSampler initialization into trainer

    @zeyugao
    zeyugao committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    1c1eed2 View commit details
    Browse the repository at this point in the history

  2. Fix llama query shape when _upad_input

    @zeyugao
    zeyugao committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    d3b0ce0 View commit details
    Browse the repository at this point in the history

  3. Use all_to_all for flexiablity

    @zeyugao
    zeyugao committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    8d565f8 View commit details
    Browse the repository at this point in the history

  4. Support sdpa for llama and mistral

    @zeyugao
    zeyugao committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    2727691 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Fix miss understood train_batch_size calcuation

    @zeyugao
    zeyugao committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    fbb7e0b View commit details
    Browse the repository at this point in the history

  2. Fix args.world_size calcuation in model parallel

    @zeyugao
    zeyugao committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    cf29d6d View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/main' into support-deepspeed-seq… 

    …uence-parallel
    @zeyugao
    zeyugao committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    01a4cdd View commit details
    Browse the repository at this point in the history

  4. Run ruff check

    @zeyugao
    zeyugao committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    57488b8 View commit details
    Browse the repository at this point in the history

  5. Run ruff format

    @zeyugao
    zeyugao committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    278873c View commit details
    Browse the repository at this point in the history