New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Big fluctuations in training-loss #347

Open

vir-95 opened this issue Nov 22, 2024 · 0 comments

vir-95 commented Nov 22, 2024

Hi, I found the trainer_state.json for your stage-1.5 training here (using qwen2-0.5b of one-vision approach): https://huggingface.co/lmms-lab/llava-onevision-qwen2-0.5b-mid-stage-a4/raw/main/trainer_state.json

When graphing the training-loss I noticed that there are big fluctuations (that you can see it in the image).

And I'm also experiencing big fluctuations in my training-loss for this stage.

Is there any reason why this is happening? Can you explain it, please?

Thanks in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment