Question about tokenizer_manager.py line 200 : Regarding the preprocessing of multimodal models

[CSEEduanyu]

sglang/python/sglang/srt/managers/tokenizer_manager.py

Line 200 in 6fcd6d7

input_ids = image_inputs["input_ids"]

[CSEEduanyu]

Question 1: According to the code here, the multi-modal inputs_ids is generated in the preprocessing stage, and the process is asynchronous. So the infer of the visual model and the infer of the LLM may happen at the same time. Will there be any conflict?

[CSEEduanyu]

Question 2: In the few multimodal models that have been implemented so far, process_images_async() does not return the input_ids field, so I am confused by the code on line 200 here.

[CSEEduanyu]

My current dilemma is that I want to add support for a new multimodal model, but I don't know whether process_images_async() should return the input_ids of the image mappings or the raw pixel information?

[merrymercy]

You can learn from existing multimodal examples (llama 3.2 and llava).

sglang/python/sglang/srt/managers/tokenizer_manager.py

Lines 206 to 210 in f407fcf

	image_inputs = await self.image_processor.process_images_async(
	obj.image_data, input_text or input_ids, obj
	)
	if image_inputs and "input_ids" in image_inputs:
	input_ids = image_inputs["input_ids"]