google

/

gemma-4-31B-it

like 3.3k

Follow

Google 62.1k

Hi @ggerganov - removing this extra \n was one of the things I experimented while working on the updated version of the chat_template, but it ended up causing severe regressions to the models (ie. 7%+ regeressions on tool calling + thinking (the once scenario where the template stores thinking thoughts) on benchmarks like BFCL, Tau2 and TerminalBench2.

could you share more details about the scenario you are testing?

Thanks!
Luciano Martins.

Hi Luciano,

Here is a simple way to observe the issue:

• Use the following test.sh script:

#!/bin/bash

curl -s http://localhost:8014/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "What is 2 + 3?"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "sum_numbers",
          "description": "Sums two numbers together",
          "parameters": {
            "type": "object",
            "properties": {
              "a": {
                "type": "number",
                "description": "The first number"
              },
              "b": {
                "type": "number",
                "description": "The second number"
              }
            },
            "required": ["a", "b"]
          }
        }
      }
    ]
  }'

Run this against an OpenAI compatible server. For example, let's compare the results between Gemma4 and Qwen3.6:

# run Gemma4
llama serve -hf ggml-org/gemma-4-31B-it-GGUF:Q4_0 --host 0.0.0.0 --port 8014 --reasoning-preserve -lv 4

# run Qwen3.6
llama serve -hf ggml-org/Qwen3.6-27B-GGUF:Q8_0 --host 0.0.0.0 --port 8014 --reasoning-preserve -lv 4

The generated reasoning_content in the response from Gemma4 does not have a trailing new line:

bash test.sh | jq .choices[0]
{
  "finish_reason": "tool_calls",
  "index": 0,
  "message": {
    "role": "assistant",
    "content": "",
    "reasoning_content": "The user is asking for the sum of 2 and 3. I should use the `sum_numbers` tool to calculate this.",
    "tool_calls": [
      {
        "type": "function",
        "function": {
          "name": "sum_numbers",
          "arguments": "{\"a\":2,\"b\":3}"
        },
        "id": "ZQvxTem0OM3XTUTOMWFcbCw4Yst5m7fu"
      }
    ]
  }
}

While the generated reasoning_content in the response from the Qwen3.6 has a trailing new line:

bash test.sh | jq .choices[0]
{
  "finish_reason": "tool_calls",
  "index": 0,
  "message": {
    "role": "assistant",
    "content": "",
    "reasoning_content": "Thinking Process:\n1.  Identify the user's request: The user wants to know the sum of 2 and 3.\n2.  Check available tools: `sum_numbers(a, b)` is available.\n3.  Determine tool usage: Call `sum_numbers` with `a=2` and `b=3`.\n4.  Execute tool call.\n5.  Formulate response based on the tool's output.✅\n",
    "tool_calls": [
      {
        "type": "function",
        "function": {
          "name": "sum_numbers",
          "arguments": "{\"a\":2,\"b\":3}"
        },
        "id": "MbzVAncEqti3PVAJZ6GCQeWsscslKxvZ"
      }
    ]
  }
}

The missing trailing new line has implications for the follow-up turns because when the chat template logic renders the result from the tool call, there will be a new line added (due to the template logic) which is not actually present in the context of the model (i.e. it is not present in its KV cache). This causes trashing of the prefix and leads to unnecessary reprocessing that, depending on the tool calls, can be quite large (f.ex. when reading large files).

If changing the chat template causes a regression, then I would investigate why the trailing new line is not being emitted by the model during the generation phase. There is some chance that this is a problem in llama.cpp - we will take futher look into it. But there is also a possibility that the issue is somewhere else. It would be worth running the above test with other engines and comparing the results.