AI Model Rankings — Raw Tests

🔬 Test Methodology

What is "RAW" Testing?

RAW tests evaluate models with direct prompts — no augmentation, no prompt engineering, no external tools. This measures each model's baseline capability across coding, reasoning and planning tasks when given a pure task description.

Test Pipeline

1️⃣ Task Prompt

Send a coding, reasoning or planning challenge directly to the model with clear requirements

2️⃣ Model Response

Model returns implementation code + unit tests

3️⃣ Vitest Run

Execute unit tests to verify correctness

4️⃣ TSC Check

TypeScript Compiler checks type safety

5️⃣ Wilson Score

Calculate adjusted ranking score

📐 How Rankings Are Calculated

⚠️ The 100% Problem

Simple pass rate (passes / total) is misleading:

1 pass, 0 fail = 100% (but only 1 test!)
10 pass, 1 fail = 90.9% (but 11 tests!)

A model that passes 1/1 looks better than one that passes 10/11. That's wrong.

🎯 Solution: Adjusted Wilson Score

Wilson Score Lower Bound accounts for both pass rate AND sample size:

Wilson = (p + z²/2n - z√[p(1-p)/n + z²/4n²] / (1+z²/n)) × n

Where:

p = pass rate (passes / total)
n = total tests attempted
z = 1.96 (95% confidence)

Result: A model with 10/11 passes gets a higher Wilson score than 1/1 passes. More tests = more confidence = fair ranking.

Score Components

Metric	Description
Pass	All unit tests passed
Fail	Tests ran but failed (logic/implementation errors)
Wilson	Adjusted Wilson Score (primary ranking)
Pass Rate	For reference only (not used for ranking)

🏆 Overall Rankings (All Categories Combined)

Ranked by Adjusted Wilson Score — higher is better

Rank	Model	Pass	Fail	Wilson	Rate
1	deepseek-v3.2	47	6	41.03	88.7%
2	qwen3-coder-next	43	9	36.54	82.7%
3	grok-4	40	3	35.00	93.0%
4	minimax-m2.5	41	8	34.77	83.7%
5	glm-4.7	39	10	32.52	79.6%
6	glm-5	34	1	29.91	97.1%
7	qwen3-coder	31	3	26.19	91.2%
8	haiku-4.5	27	1	23.04	96.4%
9	gemini-2.5-pro	26	0	22.65	100%
10	grok-4.1-fast-reason	23	5	18.03	82.1%

Rank	Model	Pass	Fail	Wilson	Rate
11	deepseek-v3	22	7	16.79	75.9%
12	kimi-k2.5	18	0	14.83	100%
13	gpt-5-mini	15	0	11.94	100%
14	gpt-oss-120b	15	1	11.47	93.8%
15	qwen3.5-397b	14	0	10.99	100%
16	gpt-5.2	14	5	10.52	73.7%
17	codestral	14	5	10.52	73.7%
18	o3	13	2	9.82	86.7%
19	gemini-3.1-pro	13	0	9.67	100%
20	grok-code-fast-1	12	6	8.00	66.7%
21	o4-mini	12	3	8.00	80.0%
22	devstral-2	10	8	5.56	55.6%
23	devstral-medium	7	7	4.00	50.0%
24	gemini-3-pro	6	0	3.00	100%
25	gemini-3-flash	5	0	2.50	100%
26	grok-4.1-fast-noreas	5	8	2.00	38.5%
27	gpt-5-nano	1	2	0.33	33.3%

🦀 Rust Rankings

Rank	Model	Pass	Fail	Wilson
1	gpt-5.3	272	1	267.42
2	gemini-3.1-pro	254	0	250.22
3	gpt-5.4	245	0	241.22
4	gemini-3-pro	226	0	222.22
5	haiku-4.5	225	5	218.53
6	gpt-5.1	215	103	198.06
7	gemini-3-flash	195	0	191.23
8	gpt-5.2	185	99	168.78
9	o3	155	0	151.25
10	o4-mini	158	103	142.23

📝 TypeScript Rankings

Rank	Model	Pass	Fail	Wilson
1	minimax-m2.5	52	9	45.31
2	sonnet-4.6	43	1	38.80
3	opus-4.5	34	1	29.91
4	grok-4.1-fast	35	4	29.80
5	opus-4.6	33	1	28.93
6	qwen3-coder	33	3	28.14
7	haiku-4.5	32	3	27.17
8	grok-4	31	1	26.96
9	qwen3-coder-next	28	4	23.02
10	glm-4.7	25	4	20.14

🧠 Reasoning Rankings

Rank	Model	Pass	Fail	Wilson
1	sonnet-4.6	15	0	11.94
2	grok-code-fast-1	15	0	11.94
3	gpt-5-mini	15	0	11.94
4	seed-2.0-mini	15	0	11.94
5	gpt-5.4-pro	15	0	11.94
6	minimax-m2.5	14	0	10.99
7	llama-4-maverick	14	1	10.53
8	glm-4.7	14	1	10.53
9	glm-5	14	1	10.53
10	haiku-4.5	13	2	9.32

✍️ Non-Coding Rankings

Rank	Model	Pass	Fail	Wilson
1	deepseek-v3.2	40	0	36.50
2	gemini-3-flash	40	0	36.50
3	qwen3-coder	36	4	30.78
4	glm-5	32	4	26.89
5	kimi-k2.5	26	6	20.70
6	deepseek-v3.2-exp	20	0	16.78
7	gpt-5-mini	20	0	16.78
8	llama-4-maverick	20	0	16.78
9	sonnet-4.6	19	1	15.28
10	minimax-m2.5	19	1	15.28

📋 Test Tasks by Category

📝 TypeScript

async-retry: Exponential backoff with jitter, configurable retry predicate

typed-emitter: Generic type-safe event emitter

rate-limiter: Token bucket algorithm

🦀 Rust

builder-pattern: Builder creational pattern

channel-mpmc: Multi-producer multi-consumer

functional-pipeline: Iterator combinators

generic-cache: Generic cache implementation

state-machine: State machine pattern

🧠 Reasoning

Logic puzzles, cryptarithmetic, spatial reasoning, tournament brackets

✍️ Non-Coding

Writing tasks, analysis, structured output, JSON extraction

📝 TypeScript Tasks — Detailed

mcp-server

Write a TypeScript MCP (Model Context Protocol) server implementation with tool registration, JSON-RPC request handling, and input validation against JSON Schema.

Features: Tool registration, tools/list, tools/call methods, type validation
Types: ToolDefinition, ToolResult, MCPRequest, MCPResponse

skill-router

Write a TypeScript skill/command router for an AI assistant with prefix, regex, and keyword matching.

Features: Multiple trigger types (prefix, regex, keyword), parameter extraction, priority handling
Matching: prefix > regex > keyword priority

tool-planner

Write a TypeScript tool execution planner that builds dependency graphs and executes tools in parallel.

Features: DAG-based planning, parallel execution, dependency resolution
Validation: Checks all inputs are satisfied from available data or previous steps

🦀 Rust Tasks — Detailed

trait-dispatch

Write a Rust plugin system using trait objects with chain and parallel execution modes.

Features: Plugin trait, PluginRegistry, chain() and parallel() execution
Error types: NotFound, ExecutionError, ChainError with step index

generic-cache

Write a generic LRU cache with TTL expiration using only std collections.

Features: LRU eviction, time-to-live, get/insert/remove/cleanup
Constraints: No external crates, use HashMap + Vec/VecDeque

state-machine

Write a type-safe state machine for order processing using typestate pattern.

States: Draft → Submitted → Processing → Shipped → Delivered / Cancelled
Feature: Compile-time enforcement of valid transitions

iterator-combinators

Write custom iterator combinators (windowed, interleave, unique_by, batched) without itertools.

Combinators: Windowed, Interleave, UniqueBy, Batched
Extension trait: IteratorExt with chainable methods

channel-mpmc

Write a multi-producer multi-consumer bounded channel using std::sync primitives.

Features: Bounded capacity, try_send/try_recv, proper disconnect handling
Primitives: Mutex, Condvar, Arc, VecDeque

builder-pattern

Write a type-safe HTTP request builder using typestate pattern.

States: NoUrl → HasUrl → HasMethod
Feature: Compile-time enforcement of required fields (url, method)

result-combinators

Write Result extension trait with error handling combinators.

Methods: map_err_with_context, or_retry, and_validate, tap, tap_err
Utilities: compose_results, fallback_chain

functional-pipeline

Write a functional data processing pipeline with closures and higher-order functions.

Pipeline: map, flat_map, filter, inspect, unwrap
Utilities: compose, pipe, apply_n, fold_while, scan_collect, group_by_key

tokio-task-pool

Write an async task pool using tokio with work stealing and graceful shutdown.

Features: spawn, spawn_blocking, shutdown, map_concurrent, race, timeout_race
Concurrency: Bounded parallelism, work stealing

🧠 Reasoning Tasks — Detailed

Logic Deduction

Solve logic puzzles involving suspects, truth-telling, and deductions.

Example: "Three suspects: Alex, Blake, Casey. Exactly one is guilty. The guilty person always lies; innocent people always tell the truth..."

Math Word Problems

Solve multi-step word problems involving rates, percentages, and work calculations.

Topics: Train meeting problems, discounts with tax, work collaboration

Sequence Prediction

Find patterns in number sequences and predict next values.

Example: "2, 3, 5, 9, 17, 33, __, __, __" (hint: look at differences)

Spatial Reasoning

Solve puzzles involving spatial arrangements, grids, and visual patterns.

Cryptarithmetic

Solve alphametic puzzles where letters represent digits.

Tournament Brackets

Determine tournament outcomes given constraints.

✍️ Non-Coding Tasks — Detailed

Writing Tasks

Generate coherent, well-structured text based on prompts.

Evaluated on: Grammar, coherence, relevance, style

Analysis Tasks

Analyze data or text and provide insights.

Structured Output

Generate output in specific formats (JSON, CSV, markdown tables).

Verification: Output must match exact schema

JSON Extraction

Extract structured data from unstructured text.

Classification

Categorize items based on provided criteria.

Summarization

Condense longer text into key points.

📖 Abbreviations

Abbreviation	Meaning
`RAW`	Direct model calls without any augmentation
`VITEST`	JavaScript/TypeScript test runner
`TSC`	TypeScript Compiler (type checking)
`Wilson`	Adjusted Wilson Score (confidence-weighted ranking)
`TS`	TypeScript
`Pass Rate`	passes / (passes + failures) — for reference only