[REQUEST] Apply tokenizer chat template for HumanEvalPack

[timrbula]

Add chat prompt type for HumanEvalPack tasks for more realistic and fair model evaluation and comparison.

Motivation

As implied in the docs and current implementation, instruction-tuned models perform better on the tasks when prompted in a way that aligns with how they were trained. There are custom prompts already implemented for several models, but it is not exhaustive and requires an update for each new prompt format or change per provider. e.g. new tokens/different format used in the latest Meta Llama or IBM Granite models.

Related to a problem/frustration with the current features?

Yes, scores are lower when not applying the chat template.

Benchmark results

Model meta-llama/Llama-3.2-3B-Instruct

instruct prompt

"humanevalfixdocs-python": {
  "pass@1": 0.5182926829268293
}

codellama prompt

"humanevalfixdocs-python": {
  "pass@1": 0.45121951219512196
}

chat prompt

"humanevalfixdocs-python": {
  "pass@1": 0.551829268292683
}

Model: ibm-granite/granite-3.0-8b-instruct

instruct prompt

"humanevalfixdocs-python": {
  "pass@1": 0.5853658536585366
}

chat prompt

"humanevalfixdocs-python": {
  "pass@1": 0.628048780487804
}

Is it related to something you would need for a project?

Yes, it's needed for anyone that wants to easily and fairly evaluate instruction-tuned models perform on these tasks.

Is it something you worked on and think could benefit the community?

Yes! Makes it much easier to evaluate instruction-tuned models out-of-the-box without needing to update the code. Users can more easily and fairly compare instruct models on these tasks.

It could also simplify the if/else block in the get_prompt function where prompt is formatted to make the codebase simpler.

Description

HuggingFace AutoTokenizer has a chat template for instruct models. Let's use that.

• Add chat as a value for --prompt
• Pass the tokenizer to the HumanEvalPack tasks
• Apply model's chat template via tokenizer apply_chat_template function

# get_prompt function
...
elif self.prompt == "chat":
  assert self.tokenizer.chat_template is not None, "The `chat` prompt should only be used for models that have a chat template."
  prompt = self.tokenizer.apply_chat_template(
      [{"role": "user", "content": inp}],
      tokenize=False,
      add_generation_prompt=True,
  )
  prompt += prompt_base
...

accelerate launch main.py tasks --humanevalfixtests-python --prompt chat --model ibm-granite/granite-3.0-8b-instruct ...

[timrbula]

Fork with proposed changes

Diff view