Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trainer.tokenizer 的警告信息需要处理吗?另外输出的日志级别可以调整吗? #6101

Closed
1 task done
joyyyhuang opened this issue Nov 21, 2024 · 2 comments
Labels
solved This problem has been already solved

Comments

@joyyyhuang
Copy link

Reminder

  • I have read the README and searched the existing issues.

System Info

  • llamafactory version: 0.9.1.dev0
  • Platform: Linux-5.4.241-1-tlinux4-0017.7-x86_64-with-glibc2.17
  • Python version: 3.12.7
  • PyTorch version: 2.5.0+cu124 (GPU)
  • Transformers version: 4.46.1
  • Datasets version: 3.1.0
  • Accelerate version: 1.0.1
  • PEFT version: 0.12.0
  • TRL version: 0.9.6
  • GPU type: NVIDIA H20
  • vLLM version: 0.6.3.post2.dev172+gd087bf86

Reproduction

    --stage sft \
    --model_name_or_path Qwen2.5-0.5B-Instruct \
    --preprocessing_num_workers 16 \
    --finetuning_type lora \
    --quantization_method bitsandbytes \
    --template qwen \
    --flash_attn auto \
    --dataset_dir data \
    --eval_dataset alpaca_zh_demo \
    --cutoff_len 1024 \
    --max_samples 100000 \
    --per_device_eval_batch_size 128 \
    --predict_with_generate True \
    --max_new_tokens 512 \
    --top_p 0.7 \
    --temperature 0.95 \
    --output_dir saves/Qwen2.5-0.5B-Instruct/lora/eval_2024-11-21-15-46-cd..49 \
    --do_predict True ```

### Expected behavior

_No response_

### Others

[WARNING|2024-11-21 15:54:20] trainer.py:760 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
反复出现上面的警告信息。
@github-actions github-actions bot added the pending This problem is yet to be addressed label Nov 21, 2024
@hiyouga
Copy link
Owner

hiyouga commented Nov 21, 2024

@hiyouga hiyouga closed this as completed Nov 21, 2024
@hiyouga hiyouga added solved This problem has been already solved and removed pending This problem is yet to be addressed labels Nov 21, 2024
@rzhao-zhsq
Copy link

@joyyyhuang 您好,请问有什么方便的途径解决这个问题吗?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
solved This problem has been already solved
Projects
None yet
Development

No branches or pull requests

3 participants