RigRoute LabsDAENRigRoute ↗
Methodology

standard-ai-v0.1

RigRoute Labs' first local-LLM-inference methodology. Measures prompt-processing and generation throughput at a fixed context depth, across an ~8B and an ~14B Q4-family model, using llama-bench's own repetition/statistics support.

Why methodology is versioned

Methodology versions are immutable. If anything that affects comparability changes — model, quantization, context, prompt/generation size, repetitions, or other benchmark settings — that becomes a new methodology id (e.g. standard-ai-v0.2), never a silent edit of this one. Every published result records which methodology version produced it, and two results are only ever compared when that version matches.

Engine: llama.cpp / llama-bench

Labs does not implement its own inference engine. It orchestrates llama-bench, part of llama.cpp, which already measures throughput reliably and reports the statistics this methodology needs natively. Every accepted result in this version of the methodology used the same llama.cpp source revision and build number, compiled for each platform (Metal on macOS, Vulkan on Windows) — so the binaries are not byte-identical across platforms:

Version0.5.0
Build number11146
Commit7fe450e19

Fixed test parameters

Context depth8192
Prompt tokens per test512
Generation tokens per test128
Repetitions5
Timeout1800s
Accelerator requiredtrue

"Context depth" is llama-bench's -d flag: the number of tokens already in context before the timed prompt-processing/generation test begins. llama-bench has no separate context-size flag — this is the correct, native way to test throughput at a given context length.

Models and quantization

Tested models are ~8B and ~14B parameter classes at a Q4-family quantization. Every accepted result records the model file's real SHA-256 (computed from the file's own bytes) and the declared identity (architecture, parameter count, quantization) as reported by llama-bench's own GGUF parsing — never inferred from a filename. See Data for exact hashes.

Prompt-processing and generation measurement

Each test measures one of two things at the fixed context depth above: prompt-processing throughput (feeding 512 new tokens) or generation throughput (producing 128 new tokens). llama-bench repeats each 5 times and reports the individual repetitions, mean, and standard deviation natively — Labs relays these directly rather than recomputing them. The repetitions run back to back in one session on one host. They show spread within that run; they are not independent replications.

GPU-acceleration verification

A benchmark that silently fell back to CPU would be worthless — or actively misleading — if reported as GPU performance. Every accepted result is checked against llama.cpp's own load_tensors: offloaded X/Y layers to GPU log line, parsed from the real captured output of that run, not assumed from the hardware being present. On a machine with more than one GPU-class device, the specific device benchmarked is also explicitly targeted and recorded — never left to an ambiguous default.

PASS semantics and failure classifications

A benchmark run always ends in exactly one status. Failures are first-class results, not discarded errors: PASS, FAIL_MODEL_LOAD, OUT_OF_MEMORY, BACKEND_UNSUPPORTED, DRIVER_UNSUPPORTED, RUNTIME_UNAVAILABLE, CPU_FALLBACK, TIMEOUT, CRASH, NOT_TESTED, UNKNOWN_FAILURE. Only PASS results are published on this site; a real failure would still be preserved as evidence in the source repository, with its own honest classification, not silently hidden.

PASS means the run completed without any of the failure classifications above — the model loaded, every layer was offloaded to the targeted accelerator, and there was no CPU fallback, timeout or crash — and it was then reviewed into the accepted registry. It is an execution and acceptance status, not a verdict on answer quality or usability.

Evidence level

Every result on this site is evidence level E1: a local observation — a performance run on one host per system, with incomplete host control (no power or thermal control, and different host systems per card). None has been independently reproduced. RigRoute Labs' scale runs from E0 (an external, sourced claim) through E2 (a controlled local measurement with linked host profiles, model hash, runtime build, exact controls and at least three performance repetitions) and E3 (independent reproduction of the same claim and protocol) to E4 (independently reproduced cross-platform evidence with disclosed controls). Results are not relabelled to a higher level after the fact.

These dimensions are not measured by standard-ai-v0.1:

UNKNOWN and UNAVAILABLE

A fact that could not be determined is recorded as the literal string UNKNOWN. Telemetry (power, temperature, accelerator memory usage) that could not be measured — because the tooling to read it wasn't safely available — is recorded as UNAVAILABLE. Neither is ever silently converted to zero, and missing telemetry never blocks a valid throughput result.

Raw-evidence retention

Every accepted run retains its exact command line, full raw llama-bench JSON output, and captured stdout/stderr — the acceleration-verification line above is parsed live from that same raw evidence, not summarized separately. See Data for what is published.

What this methodology does measure

Prompt-processing and generation throughput, in tokens/second, for a specific model file, at a specific context depth, on a specific accelerator, using llama.cpp. That is the whole scope.

What this methodology does NOT establish

Errata

Accepted results and their acceptance records are immutable, so corrections are published here instead of editing them. Measurements are unaffected by every item below.