[Bug] Why does max_new_tokens affect token usage？

[CSEEduanyu]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

so it is not paged_attn ? max new tokens should be allocated on demand，memory waste when max_new_tokens are large?

Reproduction

“The case of serving being too conservative can happen when users send many requests with a large max_new_tokens but the requests stop very early due to EOS or stop strings.”

Environment

A100 llama2-13B

[zhyncs]

It's not a bug. If there are similar issues next time, please ask in the discussion.

[CSEEduanyu]

It's not a bug. If there are similar issues next time, please ask in the discussion.

Ok, I see。I just think that if paged_attention is implemented, max_new_tokens should not be a waste of memory?This confused me

[Ying1123]

if you accept too many requests in, they will run out of memory later and you need to pause them or evict some. This can be inefficient.

[CSEEduanyu]

Can you share your benchmark scripts? We can take a look. @hnyls2002

sorry i can not , but i thinks it is easy to reproduce。Like average output=50tokens ，max_new_token=4096 and max_new_token=128 will make a big difference in throughput performance

when max_new_token=4096 token_usage is very small，This case should only exist for system species without pagedattn。

[merrymercy]

Thanks for reporting this. We correctly implement the page attention, but there may be different scheduling policies. We would greatly appreciate it if you could share a minimal reproducible script. Otherwise, we can take a look later.

[CSEEduanyu]

Thanks for reporting this. We correctly implement the page attention, but there may be different scheduling policies. We would greatly appreciate it if you could share a minimal reproducible script. Otherwise, we can take a look later.

Sorry I can't provide any code or script, but I can describe this scenario as best I can:
For example, a test set with 100,000 samples has an average input=200 tokens and an average output=50 tokens.
Test A:max_new_token=2048
Test B:max_new_token=128
The throughput of test A is much worse than that of test B. This problem exists for both tgi and sglang, but not for vllm, so I always thought that is only vllm implementing paged_attn correctly? I think paged_attn is a solution to the kv cahce waste caused by not knowing how long a request will end

[CSEEduanyu]

Thanks for reporting this. We correctly implement the page attention, but there may be different scheduling policies. We would greatly appreciate it if you could share a minimal reproducible script. Otherwise, we can take a look later.

You guys have done a great job and I have read all the documentation carefully. In this document: https://github.com/sgl-project/sglang/blob/main/docs/en/hyperparameter_tuning.md, It was mentioned that "The case of serving being too conservative can happen when users send many requests with a large max_new_tokens but the requests stop very early due to EOS or stop strings." I think this scenario is identical to the one I encountered, where the kv_cache is allocated token by token dimension as paged_attn is implemented, Instead of pre-splitting according to max_new_token, I don't know if I understand correctly?So my question is, if we have implemented all the logic of paged_atnn correctly, why is the throughput of the service still limited by max_new_tokens? That's what I'm confused about.

[CSEEduanyu]

Thanks for reporting this. We correctly implement the page attention, but there may be different scheduling policies. We would greatly appreciate it if you could share a minimal reproducible script. Otherwise, we can take a look later.

tgi claims to support paged_attn, but I'm pretty sure they just use a kernel named "paged_attn". kv_cache is pre-allocated according to max_new_token, so there is a lot of wasted kv_cache memory. This is also the main reason for the poor throughput of tgi. I'm not sure about sglang because I haven't been able to look at the source code yet, just take a guess from the test results.

[merrymercy]

see also #1278