new dataset: which local LLM best *drives an agent*? benchmarked 4 models for pairing with Hermes Agent (@NousResearch ) - a CodeAct agent that writes python to call its tools. RTX 5090, llama.cpp. two phases, hybrid:

>>> phase A (synthetic): scored 4 axes — code-as-action, long-context, instruction-following under Hermes' real ~3.5K-token prompt, multi-step loops. top was a near-tie (within noise): an 18B frankenmerge (Qwopus) edged Qwen3.6-27B, and Hermes' own 36B came LAST.

>>> phase B (real harness): installed Hermes, ran the top 3 through 14 multi-step tasks x3 repeats. the tie broke — and an efficiency gap appeared:

Qwen3.6-27B    100%   | 3.0 turns | 364 tok
Qwopus-18B     85.7%  | 3.6 turns | 870 tok
Nemotron-30B   85.7%  | 4.4 turns | 1334 tok

Qwen is perfect AND 2.4-3.7x more token-efficient — something a synthetic test can't see (only the real agent loop can). verdict: Qwen3.6-27B for local Hermes.

dataset: witcheer/hermes-pairing-bench
collection: witcheer/rtx-5090-benchmark-rig-6a17e365b534abb474250e11

updated a collection 2 months ago

RTX 5090 Benchmark Rig

Collection

Speed and quality benchmarks for quantized LLMs on NVIDIA RTX 5090 32GB. Data, tooling, and results. • 2 items • Updated May 30 • 1

updated a dataset 2 months ago

witcheer/hermes-pairing-bench

Viewer • Updated Jun 2 • 7 • 29

witcheer

AI & ML interests

Recent Activity

Organizations

witcheer's activity

gemma-challenge/gemma-witcheer

gemma-challenge/gemma-witcheer

Copy Repo Contents to Buckets Instantly

Filter Models page by Base Models only

Publish models from CI without HF_TOKEN