Skip to content

Releases: modelscope/ms-swift

v2.6.0

13 Nov 08:06
@tastelikefeet tastelikefeet
v2.6.0
f934283
Compare
Choose a tag to compare
v2.6.0 Latest
Latest

English Version

Models

  1. Support Qwen2.5 coder models

Feature

  1. Correct and support the new loss and gradient accumulation algorithm from transformers.trainer

中文版本

模型

  1. 支持千问coder系列模型

功能

  1. 支持新的transformers loss和GA计算算法，并修正了其中的bug

What's Changed

Full Changelog: v2.5.2...v2.6.0

Contributors

yingdachen, Jintao-Huang, and 2 other contributors
Assets 2
Loading

v2.5.2

02 Nov 07:50
@Jintao-Huang Jintao-Huang
v2.5.2
284daa1
Compare
Choose a tag to compare
v2.5.2

New Models:

  1. emu3-chat
  2. aya-expanse
  3. ministral-8b-inst-2410

New Datasets:

  1. llava-video-178k
  2. moviechat-1k-test

What's Changed

New Contributors

Full Changelog: v2.5.1...v2.5.2

Contributors

mi804, Yunnglin, and 4 other contributors
Assets 2
Loading

v2.5.1

21 Oct 12:05
@Jintao-Huang Jintao-Huang
v2.5.1
64d8080
Compare
Choose a tag to compare
v2.5.1

English Version

New Features:

  1. Support for RM for LLM and MLLM, as well as PPO for LLM.

New Models:

  1. molmo series
  2. mplug-owl3 1b/2b
  3. llama3.1-nemotron-70b-instruct
  4. deepseek-janus

中文版

新特性：

  1. 支持LLM和MLLM的RM, 以及LLM的PPO.

新模型：

  1. molmo系列
  2. mplug-owl3 1b/2b
  3. llama3.1-nemotron-70b-instruct
  4. deepseek-janus

What's Changed

New Contributors

Full Changelog: v2.5.0...v2.5.1

Contributors

mi804, Jintao-Huang, and 4 other contributors
Assets 2
Loading

v2.5.0

10 Oct 02:21
@Jintao-Huang Jintao-Huang
v2.5.0
7af4682
Compare
Choose a tag to compare
v2.5.0

English Version

New Features:

  1. Support for GPTQ & AWQ quantization of multimodal LLMs.
  2. Support for dynamic addition of gradient checkpointing in the ViT section to reduce memory consumption.
  3. Support for multimodal model pre-training.

New Models:

  1. llama3.2, llama3.2-vision series
  2. got-ocr2
  3. llama3.1-omni
  4. ovis1.6-gemma2
  5. pixtral-12b
  6. telechat2-115b
  7. mistral-small-inst-2409

New Datasets:

  1. egoschema

中文版

新特性：

  1. 支持多模态LLM的gptq&awq量化.
  2. 支持动态在vit部分增加gradient_checkpointing, 减少显存消耗.
  3. 支持多模态模型预训练.

新模型：

  1. llama3.2, llama3.2-vision系列
  2. got-ocr2
  3. llama3.1-omni
  4. ovis1.6-gemma2
  5. pixtral-12b
  6. telechat2-115b
  7. mistral-small-inst-2409

新数据集：

  1. egoschema

What's Changed

New Contributors

Full Changelog: v2.4.2...v2.5.0

Contributors

Digital2Slave, LukeForeverYoung, and 5 other contributors
Assets 2
Loading

v2.4.2

18 Sep 16:56
@Jintao-Huang Jintao-Huang
v2.4.2
264b59f
Compare
Choose a tag to compare
v2.4.2

English Version

New Features:

  1. RLHF reconstruction, supporting all integrated multimodal models, compatible with DeepSpeed Zero2/Zero3, and supports lazy_tokenize.
  2. Using infer_backend vllm, inference deployment of multimodal large models supports multiple images.

New Models:

  1. Qwen2.5 series, Qwen2-vl-72b series (base/instruct/gptq-int4/gptq-int8/awq)
  2. Qwen2.5-math, Qwen2.5-coder series (base/instruct)
  3. Deepseek-v2.5

New Datasets:

  1. longwriter-6k-filtered

中文版

新特性：

  1. RLHF重构，支持所有已接入的多模态模型，兼容deepspeed zero2/zero3，支持lazy_tokenize
  2. 使用infer_backend vllm，推理部署多模态大模型支持多图.

新模型：

  1. qwen2.5系列、qwen2-vl-72b系列（base/instruct/gptq-int4/gptq-int8/awq）
  2. qwen2.5-math, qwen2.5-coder系列（base/instruct）
  3. deepseek-v2.5

新数据集：

  1. longwriter-6k-filtered

What's Changed

New Contributors

Full Changelog: v2.4.1...v2.4.2

Contributors

wangxingjun778, irexyc, and 4 other contributors
Assets 2
Loading

v2.4.1

13 Sep 05:03
@Jintao-Huang Jintao-Huang
v2.4.1
7545374
Compare
Choose a tag to compare
v2.4.1

English Version

New Features:

  1. Inference and deployment support for logprobs.
  2. RLHF support for lazy_tokenize.
  3. Multimodal model support for neftune.
  4. dynamic_eos compatibility with glm4 series and other models.

New Models:

  1. mplug-owl3, best practices can be found here.
  2. yi-coder 1.5b, base/chat model of 9b.
  3. minicpm3-4b.
  4. reflection-llama3.1-70b.

中文版

新功能：

  1. 推理和部署支持 logprobs。
  2. RLHF支持lazy_tokenize。
  3. 多模态模型支持neftune。
  4. dynamic_eos兼容glm4系列等模型。

新模型：

  1. mplug-owl3，最佳实践可以查看这里。
  2. yi-coder 1.5b、9b 的base/chat模型。
  3. minicpm3-4b。
  4. reflection-llama3.1-70b。

What's Changed

Full Changelog: v2.4.0...v2.4.1

Contributors

Jintao-Huang, Zhikaiiii, and 3 other contributors
Assets 2
Loading

v2.4.0

13 Sep 04:50
@Jintao-Huang Jintao-Huang
v2.4.0
0e4e9c3
Compare
Choose a tag to compare
v2.4.0

English Version

New Features:

  1. Support for Liger, which accommodates models like LLaMA, Qwen, Mistral, etc., and reduces memory usage by 10% to 60%.
  2. Support for custom loss function training using a registration mechanism.
  3. Training now supports pushing models to ModelScope and HuggingFace.
  4. Support for the freeze_vit parameter to control the behavior of full parameter training for multimodal models.

New Models:

  1. Qwen2-VL series includes GPTQ/AWQ quantized models. For best practices, see here.
  2. InternVL2 AWQ quantized models.

New Datasets:

  1. qwen2-pro series

中文版

新特性：

  1. 支持 Liger训练LLaMA、Qwen、Mistral 等模型，内存使用降低 10% 至 60%。
  2. 支持使用注册机制进行自定义损失函数的训练。
  3. 训练支持将模型推送至 ModelScope 和 HuggingFace。
  4. 支持 freeze_vit 参数，以控制多模态模型全参数训练的行为。

新模型：

  1. Qwen2-VL 系列包括 GPTQ/AWQ 量化模型，最佳实践可以查看这里。
  2. InternVL2 AWQ 量化模型。

新数据集：

  1. qwen2-pro 系列

What's Changed

Full Changelog: v2.3.2...v2.4.0

Contributors

Jintao-Huang, baoleai, and 2 other contributors
Assets 2
Loading

v2.3.2

24 Aug 04:42
@Jintao-Huang Jintao-Huang
v2.3.2
e2bba6e
Compare
Choose a tag to compare
v2.3.2

English Version

New Features:

  1. ReFT support: achieves parameter efficiency that is 15× to 65× greater than LoRA.
  2. Multimodal model supports zero3.
  3. Supports using environment variables to control parameters such as hd_num, max_num, and video_segments.

New Models:

  1. longwriter-glm4-9b, longwriter-llama3_1-8b
  2. phi3_5-mini-instruct, phi3_5-moe-instruct, phi3_5-vision-instruct
  3. llava-onevision-qwen2-0_5b-ov, llava-onevision-qwen2-7b-ov, llava-onevision-qwen2-72b-ov

New Datasets:

  1. longwriter-6k
  2. rlaif-v
  3. latex-ocr-print, latex-ocr-handwrite

中文版

新功能：

  1. 支持ReFT，实现了比 LoRA 高 15 倍到 65 倍的参数效率。
  2. 多模态模型支持 zero3。
  3. 支持使用环境变量控制模型特有的参数，如 hd_num、max_num 和 video_segments。

新模型：

  1. longwriter-glm4-9b, longwriter-llama3_1-8b
  2. phi3_5-mini-instruct, phi3_5-moe-instruct, phi3_5-vision-instruct
  3. llava-onevision-qwen2-0_5b-ov, llava-onevision-qwen2-7b-ov, llava-onevision-qwen2-72b-ov

新数据集：

  1. longwriter-6k
  2. rlaif-v
  3. latex-ocr-print, latex-ocr-handwrite

What's Changed

New Contributors

Full Changelog: v2.3.1...v2.3.2

Contributors

wning13, Jintao-Huang, and 4 other contributors
Assets 2
Loading

v2.3.1

19 Aug 03:11
@Jintao-Huang Jintao-Huang
v2.3.1
742d16b
Compare
Choose a tag to compare
v2.3.1

English Version

New Features:

  1. ms-swift paper published: https://arxiv.org/abs/2408.05517
  2. Web-UI supports audio and video.
  3. Support for deploying audio and video models using the OpenAI API.
  4. Utilizes a new multimodal training framework.
  5. supports inference acceleration for video models (lmdeploy & internvl2 series).

New Models:

  1. idefics3-8b-llama3
  2. llava-hf 72b, 110b, llama3-llava
  3. deepseek-coder-v2, deepseek-coder-lite-v2, deepseek-v2

中文版

新功能：

  1. 发布了 ms-swift 论文：https://arxiv.org/abs/2408.05517
  2. Web-UI 支持音频和视频。
  3. 支持使用 OpenAI API 部署音频和视频模型。
  4. 采用新的多模态训练框架。
  5. 支持视频模型的推理加速（lmdeploy 和 internvl2 系列）。

新模型：

  1. idefics3-8b-llama3
  2. llava-hf 72b、110b、llama3-llava
  3. deepseek-coder-v2、deepseek-coder-lite-v2、deepseek-v2

What's Changed

New Contributors

Full Changelog: v2.3.0...v2.3.1

Contributors

Yunnglin, Jintao-Huang, and 3 other contributors
Assets 2
Loading

v2.3.0

09 Aug 15:43
@tastelikefeet tastelikefeet
v2.3.0
451ef3d
Compare
Choose a tag to compare
v2.3.0

English Version

New Features

  1. Support for readthedocs documentation site at: https://swift.readthedocs.io/en/latest
  2. Support Megatron architecture training for QianWen series models, and added new pt command for pretraining. See docs: https://swift.readthedocs.io/en/latest/LLM/Megatron-training.html
  3. Support LMDeploy for inference and deployment, improving inference acceleration for multi-modal models. See: https://swift.readthedocs.io/en/latest/Multi-Modal/LmDeploy-inference-acceleration.html
  4. Support passing lora target modules via regular expressions
  5. Support configuring max_memory usage for each GPU in device_map
  6. export command supports BitsAndBytes quantization
  7. export command supports Ollama export: https://swift.readthedocs.io/en/latest/LLM/OLLaMA-Export.html
  8. Support Q-GaLore algorithm
  9. Support RLHF training for multi-modal models: https://swift.readthedocs.io/en/latest/Multi-Modal/human-preference-alignment-training-documentation.html
  10. Support evaluation on 100+ datasets for multi-modal models: https://swift.readthedocs.io/en/latest/LLM/LLM-eval.html
  11. Support resizing input images when memory usage is too high for multi-modal models
  12. Modified default lora injection for multi-modal model training. Now takes effect on LLM and projector, results are better without significantly increasing training memory.
  13. Support PEFT 0.12, and added new tuner: fourierft
  14. Support rope-scaling for multi-modal models
  15. Support streaming processing of datasets to reduce memory usage, enable with --streaming
  16. Support vLLM multi-modal inference and deployment
  17. Support grounding task for popular multi-modal models.

New Models

  1. qwen2-audio series
  2. qwen2-math
  3. codegeex4
  4. internvl2 series
  5. llava video
  6. xcomposer2.5
  7. cogvlm2-video
  8. numina-math
  9. mistral-nemo
  10. llama3.1 series
  11. mistral-large
  12. gemma-2-2b
  13. internlm2.5 1.8b 20b
  14. minicpm-v-v2_6-chat

Check: https://swift.readthedocs.io/en/latest/LLM/Supported-models-datasets.html

New Datasets

  1. zhihu-kol and zhihu-kol-filtered
  2. SA1B series multi-modal zh datasets

Check: https://swift.readthedocs.io/en/latest/LLM/Supported-models-datasets.html

中文版本

新功能

  1. 支持readthedocs文档库， 地址：https://swift.readthedocs.io/zh-cn/latest
  2. 支持千问系列模型的Megatron结构训练，并支持了新的pt命令用于预训练，详见文档：https://swift.readthedocs.io/zh-cn/latest/LLM/Megatron%E8%AE%AD%E7%BB%83%E6%96%87%E6%A1%A3.html
  3. 支持LMDeploy的推理和部署，更好地支持了多模态模型的推理加速，详见：https://swift.readthedocs.io/zh-cn/latest/Multi-Modal/LmDeploy%E6%8E%A8%E7%90%86%E5%8A%A0%E9%80%9F%E6%96%87%E6%A1%A3.html
  4. 支持以正则表达式方式传入lora target模块
  5. 支持配置device_map各GPU用量的max_memory
  6. export命令支持BitsAndBytes量化
  7. export命令支持Ollama导出：https://swift.readthedocs.io/zh-cn/latest/LLM/OLLAMA%E5%AF%BC%E5%87%BA%E6%96%87%E6%A1%A3.html
  8. 支持Q-GaLore算法
  9. 支持多模态模型的RLHF训练：https://swift.readthedocs.io/zh-cn/latest/Multi-Modal/%E4%BA%BA%E7%B1%BB%E5%81%8F%E5%A5%BD%E5%AF%B9%E9%BD%90%E8%AE%AD%E7%BB%83%E6%96%87%E6%A1%A3.html
  10. 支持多模态模型100+数据集的评测能力：https://swift.readthedocs.io/zh-cn/latest/LLM/LLM%E8%AF%84%E6%B5%8B%E6%96%87%E6%A1%A3.html
  11. 支持多模态模型显存占用过高时对输入图片进行缩放
  12. 修改了多模态模型训练的默认lora注入，目前对LLM和projector生效，不显著提高训练显存情况下效果更好
  13. 支持PEFT0.12，并支持了新的tuner：fourierft
  14. 支持多模态模型的rope-scaling
  15. 支持数据集的流式处理，降低显存消耗，使用--streaming开启
  16. 支持了vLLM的多模态推理部署能力
  17. 对部分多模态模型支持了grounding任务

新模型

  1. qwen2-audio系列模型
  2. qwen2-math
  3. codegeex4
  4. internvl2系列模型
  5. llava video
  6. xcomposer2.5
  7. cogvlm2-video
  8. numina-math
  9. mistral-nemo
  10. llama3.1系列
  11. mistral-large
  12. gemma-2-2b
  13. internlm2.5 1.8b 20b
  14. minicpm-v-v2_6-chat

参考：https://swift.readthedocs.io/zh-cn/latest/LLM/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.html

新数据集

  1. zhihu-kol和zhihu-kol-filtered数据集
  2. SA1B系列中文多模态数据集

参考：https://swift.readthedocs.io/zh-cn/latest/LLM/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.html

What's Changed

Read more

Contributors

Wondersui, starxhong, and 8 other contributors
Assets 2
Loading