Claude Opus 4.8 vs GLM-5.2 for coding

Q: Which is better for coding, Claude Opus 4.8 or GLM-5.2?

Claude Opus 4.8 ranks higher overall across 140 tasks from 4 public repositories. On the pairwise Bradley-Terry rating (Elo-scaled, anchored on the human commit at 0), Claude Opus 4.8 sits at -1 and GLM-5.2 at -84. Claude Opus 4.8 costs about $103.088 per run, GLM-5.2 about $45.754 per run.

Q: How is the ranking calculated?

Each task is a real commit from a public repository. The agent gets the code at the commit before the change and a written description of the work, then edits the files itself. Every agent's diff is then compared head-to-head against every other one — and against the human commit — by a blind judge that never sees which model produced which diff. Those pairwise wins and losses become a Bradley-Terry rating (Elo-scaled), so the ranking reflects which changes actually win comparisons, not whether they match the original wording.

Q: Could the agents have seen the answers already?

Two things limit that. The task description is rewritten from the change rather than copied from the commit message, and the grader is given the rubric instead of the original answer. Building tasks from recent commits also reduces how much a model could have trained on the result.

Q: Can I compare Claude Opus 4.8 and GLM-5.2 on my own code?

Yes. Point CommitBench at your repository and it builds the same kind of tasks from your own commit history, so the ranking reflects the code you actually work on.

Claude Opus 4.8 ranks higher on real GitHub commits. Each agent gets a repository at a past commit and a description of the next change, edits the code on its own, and is graded from 0 to 10. Pick any two models below to switch the comparison.

BT rating · Claude Opus 4.8

-1elo

BT rating · GLM-5.2

-84elo

Per-task wins

70–28–16

Tasks · repos

140 · 4

Edge

Claude

What the numbers say

Over 140 benchmarked tasks from 4 public repositories, Claude Opus 4.8 has a Bradley-Terry rating of -1 and GLM-5.2 -84 (Elo-scaled, with the human commit anchored at 0). On the 114 tasks where both ran, Claude Opus 4.8 won more often (70 to 16, with 28 tied). On cost, Claude Opus 4.8 runs at about $103.088 and GLM-5.2 at about $45.754 per run.

Head-to-head, per task (rubric scores, 0–10)

Repo	Task	Claude Opus 4.8	GLM-5.2
ggml-org/llama.cpp	model : support granite multilingual embeddings R2 (ibm-granite/granite-embedding-{97,311}m-multilingual-r2) (#22716) · conversion	10.0	0.0
pooled: unsloth-zoo + vllm + llama.cpp	[Model]Fix MiniMaxM2ForCausalLM perf regression (#45935) · tests	10.0	0.0
pooled: unsloth-zoo + vllm + llama.cpp	model : support granite multilingual embeddings R2 (ibm-granite/granite-embedding-{97,311}m-multilingual-r2) (#22716) · conversion	10.0	0.0
vllm-project/vllm	[Model]Fix MiniMaxM2ForCausalLM perf regression (#45935) · tests	10.0	0.0
pooled: unsloth-zoo + vllm + llama.cpp	[DeepEP V2] Fill invalid recv_topk_idx with -1 (#46432) · vllm	9.5	0.0
vllm-project/vllm	[DeepEP V2] Fill invalid recv_topk_idx with -1 (#46432) · vllm	9.5	0.0
ggml-org/llama.cpp	convert : minor fixes for numpy 2.x (#23571) · examples	8.6	0.0
pooled: unsloth-zoo + vllm + llama.cpp	convert : minor fixes for numpy 2.x (#23571) · examples	8.6	0.0
ggml-org/llama.cpp	cuda : fix KQ mask offset integer overflow in fattn MMA kernel (#23610) · ggml	10.0	1.5
pooled: unsloth-zoo + vllm + llama.cpp	cuda : fix KQ mask offset integer overflow in fattn MMA kernel (#23610) · ggml	10.0	1.5
pooled: unsloth-zoo + vllm + llama.cpp	[Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in sampler kernels (#46560) · tests	9.0	1.0
vllm-project/vllm	[Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in sampler kernels (#46560) · tests	9.0	1.0
pooled: unsloth-zoo + vllm + llama.cpp	[Bugfix] Default tie_weights to sharing the weight (fix tied quantized embeddings, e.g. ModelOpt Gemma4) (#45544) · vllm	9.2	1.5
vllm-project/vllm	[Bugfix] Default tie_weights to sharing the weight (fix tied quantized embeddings, e.g. ModelOpt Gemma4) (#45544) · vllm	9.2	1.5
pooled: unsloth-zoo + vllm + llama.cpp	Fix gptoss 4bit (#524) · unsloth_zoo	9.5	2.0
unslothai/unsloth-zoo	Fix gptoss 4bit (#524) · unsloth_zoo	9.5	2.0
pooled: unsloth-zoo + vllm + llama.cpp	fix(moe_wna16): access tp_size via moe_config for RoutedExperts compatibility (#45404) · vllm	8.1	2.0
vllm-project/vllm	fix(moe_wna16): access tp_size via moe_config for RoutedExperts compatibility (#45404) · vllm	8.1	2.0
pooled: unsloth-zoo + vllm + llama.cpp	Auto-install fused lm_head + cross_entropy forward across transformers · tests	10.0	4.0
unslothai/unsloth-zoo	Auto-install fused lm_head + cross_entropy forward across transformers · tests	10.0	4.0
pooled: unsloth-zoo + vllm + llama.cpp	add fixes for moe · unsloth_zoo	10.0	5.2
unslothai/unsloth-zoo	add fixes for moe · unsloth_zoo	10.0	5.2
pooled: unsloth-zoo + vllm + llama.cpp	fix(moe-fp8): inline weight/quant-state lookup; narrow ImportError scope · unsloth_zoo	7.5	3.0
pooled: unsloth-zoo + vllm + llama.cpp	Humming support for 2/3/5/6/7-bit pack-quantized weight-only inference (#46389) · vllm	9.0	4.5
unslothai/unsloth-zoo	fix(moe-fp8): inline weight/quant-state lookup; narrow ImportError scope · unsloth_zoo	7.5	3.0
vllm-project/vllm	Humming support for 2/3/5/6/7-bit pack-quantized weight-only inference (#46389) · vllm	9.0	4.5
pooled: unsloth-zoo + vllm + llama.cpp	Capture outputs fixes for transformers v5 (#713) · tests	9.3	5.0
unslothai/unsloth-zoo	Capture outputs fixes for transformers v5 (#713) · tests	9.3	5.0
pooled: unsloth-zoo + vllm + llama.cpp	fix(moe-bnb): dequant Params4bit experts in transformers v5 grouped/batched MoE forward (B6) · unsloth_zoo	5.5	9.5
unslothai/unsloth-zoo	fix(moe-bnb): dequant Params4bit experts in transformers v5 grouped/batched MoE forward (B6) · unsloth_zoo	5.5	9.5
pooled: unsloth-zoo + vllm + llama.cpp	perf(moe-fp8): batched 3D dequant + FP8Experts dispatcher + Trainer guard · unsloth_zoo	9.7	5.9
unslothai/unsloth-zoo	perf(moe-fp8): batched 3D dequant + FP8Experts dispatcher + Trainer guard · unsloth_zoo	9.7	5.9
pooled: unsloth-zoo + vllm + llama.cpp	[Perf][1/N] Expand Triton kernel warmup coverage, DSv4 (#46634) · vllm	9.0	5.3
vllm-project/vllm	[Perf][1/N] Expand Triton kernel warmup coverage, DSv4 (#46634) · vllm	9.0	5.3
pooled: unsloth-zoo + vllm + llama.cpp	Fix review findings for PR #3: Gemma4 LoRA/BnB patches, GDN extraction, finalize_huggingface_model · unsloth_zoo	9.6	6.0
unslothai/unsloth-zoo	Fix review findings for PR #3: Gemma4 LoRA/BnB patches, GDN extraction, finalize_huggingface_model · unsloth_zoo	9.6	6.0
ggml-org/llama.cpp	sycl : support MUL_MAT and OUT_PROD with Q1_0 (#24721) · ggml	9.0	5.5
ggml-org/llama.cpp	ggml-webgpu: support non-square subgroup matrix configs for Intel GPUs (#21669) · ggml	8.5	5.0
pooled: unsloth-zoo + vllm + llama.cpp	sycl : support MUL_MAT and OUT_PROD with Q1_0 (#24721) · ggml	9.0	5.5
pooled: unsloth-zoo + vllm + llama.cpp	ggml-webgpu: support non-square subgroup matrix configs for Intel GPUs (#21669) · ggml	8.5	5.0
pooled: unsloth-zoo + vllm + llama.cpp	[XPU][MoE] Add WNA16 oracle backend for GPTQ sym-int4 (xpu_fused_moe) (#41426) · vllm	2.5	5.9
vllm-project/vllm	[XPU][MoE] Add WNA16 oracle backend for GPTQ sym-int4 (xpu_fused_moe) (#41426) · vllm	2.5	5.9
pooled: unsloth-zoo + vllm + llama.cpp	nixl_ep: Skip post-receive quantization for NVFP4 (#45606) · vllm	9.5	6.5
vllm-project/vllm	nixl_ep: Skip post-receive quantization for NVFP4 (#45606) · vllm	9.5	6.5
ggml-org/llama.cpp	[SYCL] Add BF16 support to GET_ROWS operation (#21391) · ggml	10.0	7.3
pooled: unsloth-zoo + vllm + llama.cpp	[SYCL] Add BF16 support to GET_ROWS operation (#21391) · ggml	10.0	7.3
ggml-org/llama.cpp	ggml: `gguf_init_from_callback` and `gguf_init_from_buffer` (#22341) · ggml	10.0	7.4
pooled: unsloth-zoo + vllm + llama.cpp	ggml: `gguf_init_from_callback` and `gguf_init_from_buffer` (#22341) · ggml	10.0	7.4
ggml-org/llama.cpp	ggml: support concat for scalar types at cuda backend (#24011) · ggml	10.0	7.5
pooled: unsloth-zoo + vllm + llama.cpp	ggml: support concat for scalar types at cuda backend (#24011) · ggml	10.0	7.5
pooled: unsloth-zoo + vllm + llama.cpp	Fix dense vLLM state dict parity · tests	0.0	2.3
unslothai/unsloth-zoo	Fix dense vLLM state dict parity · tests	0.0	2.3
pooled: unsloth-zoo + vllm + llama.cpp	[Kernel] Add swap AB optimization to fused_moe_kernel (#36559) · vllm	4.0	2.0
vllm-project/vllm	[Kernel] Add swap AB optimization to fused_moe_kernel (#36559) · vllm	4.0	2.0
ggml-org/llama.cpp	cuda: Q1_0 initial backend (#21629) · ggml	8.0	6.3
pooled: unsloth-zoo + vllm + llama.cpp	cuda: Q1_0 initial backend (#21629) · ggml	8.0	6.3
ggml-org/llama.cpp	metal: Q1_0 backend (#21528) · ggml	8.5	10.0
ggml-org/llama.cpp	[SYCL] support bf16 on bin_bcast OP and unary OPs (#24838) · ggml	9.0	7.5
ggml-org/llama.cpp	sycl : fix failed ut cases of norm (#25044) · ggml	10.0	8.5
pooled: unsloth-zoo + vllm + llama.cpp	fix(moe): loud-fail on silent fallbacks in MoE merge + FP8 forward paths · unsloth_zoo	8.0	6.5
pooled: unsloth-zoo + vllm + llama.cpp	fix(peft-param-wrapper): handle merge_and_unload for 4-bit MoE experts (B4) · unsloth_zoo	10.0	8.5
pooled: unsloth-zoo + vllm + llama.cpp	Fix ModuleNotFoundError when loading gpt-oss models without triton_kernels (#4088) (#539) · unsloth_zoo	8.0	9.5
pooled: unsloth-zoo + vllm + llama.cpp	[DSv4 Perf] DSv4 flashinfer sparse index cache for metadata, 2%~4% TTFT improvement (#45863) · tests	9.0	7.5
pooled: unsloth-zoo + vllm + llama.cpp	metal: Q1_0 backend (#21528) · ggml	8.5	10.0
pooled: unsloth-zoo + vllm + llama.cpp	[SYCL] support bf16 on bin_bcast OP and unary OPs (#24838) · ggml	9.0	7.5
pooled: unsloth-zoo + vllm + llama.cpp	sycl : fix failed ut cases of norm (#25044) · ggml	10.0	8.5
unslothai/unsloth-zoo	fix(moe): loud-fail on silent fallbacks in MoE merge + FP8 forward paths · unsloth_zoo	8.0	6.5
unslothai/unsloth-zoo	fix(peft-param-wrapper): handle merge_and_unload for 4-bit MoE experts (B4) · unsloth_zoo	10.0	8.5
unslothai/unsloth-zoo	Fix ModuleNotFoundError when loading gpt-oss models without triton_kernels (#4088) (#539) · unsloth_zoo	8.0	9.5
vllm-project/vllm	[DSv4 Perf] DSv4 flashinfer sparse index cache for metadata, 2%~4% TTFT improvement (#45863) · tests	9.0	7.5
ggml-org/llama.cpp	vulkan: Switch MUL_MAT_VEC to 4 K per iteration for F16/32 (#22887) · ggml	9.3	8.0
pooled: unsloth-zoo + vllm + llama.cpp	vulkan: Switch MUL_MAT_VEC to 4 K per iteration for F16/32 (#22887) · ggml	9.3	8.0
pooled: unsloth-zoo + vllm + llama.cpp	fix(moe-quant): gate v5-only patches · tests	7.5	8.7
unslothai/unsloth-zoo	fix(moe-quant): gate v5-only patches · tests	7.5	8.7
pooled: unsloth-zoo + vllm + llama.cpp	feat(mlx): add save_method to save_pretrained_merged · unsloth_zoo	9.7	8.6
unslothai/unsloth-zoo	feat(mlx): add save_method to save_pretrained_merged · unsloth_zoo	9.7	8.6
ggml-org/llama.cpp	dflash: refactor draft model conversion (#25110) · conversion	10.0	9.0
pooled: unsloth-zoo + vllm + llama.cpp	fix(mlx): repair stub injection on Apple Silicon (3 sub-bugs) · unsloth_zoo	10.0	9.0
pooled: unsloth-zoo + vllm + llama.cpp	[XPU] Fix Triton attn fp8/bf16 check failing (#45758) · vllm	9.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	[Bugfix] Fix minimax_qk_norm_fusion (#44983) · vllm	5.5	4.5
pooled: unsloth-zoo + vllm + llama.cpp	dflash: refactor draft model conversion (#25110) · conversion	10.0	9.0
unslothai/unsloth-zoo	fix(mlx): repair stub injection on Apple Silicon (3 sub-bugs) · unsloth_zoo	10.0	9.0
vllm-project/vllm	[XPU] Fix Triton attn fp8/bf16 check failing (#45758) · vllm	9.0	10.0
vllm-project/vllm	[Bugfix] Fix minimax_qk_norm_fusion (#44983) · vllm	5.5	4.5
ggml-org/llama.cpp	metal : fix FA support logic (#21898) · ggml	8.0	8.5
pooled: unsloth-zoo + vllm + llama.cpp	metal : fix FA support logic (#21898) · ggml	8.0	8.5
ggml-org/llama.cpp	[SYCL] Add Q8_0 reorder optimization (~3x tg speedup on Intel Arc) (#21527) · ggml	10.0	10.0
ggml-org/llama.cpp	CUDA: Various fixes to `cpy.cu` (#25000) · ggml	10.0	10.0
ggml-org/llama.cpp	CUDA & CPU: support F32 kernel type for `CONV_TRANSPOSE_2D` (#17094) · ggml	10.0	10.0
ggml-org/llama.cpp	ggml: vectorize ggml_vec_dot_q4_1_q8_1 with WASM SIMD128 (#22209) · ggml	10.0	10.0
ggml-org/llama.cpp	fix: GLM-DSA crash in llama-tokenize when using vocab_only (#22102) · src	10.0	10.0
ggml-org/llama.cpp	ggml : fix ARM NEON nvfp4 dot product on non-dotprod targets (#21559) · ggml	8.5	8.5
pooled: unsloth-zoo + vllm + llama.cpp	Use torch.Tensor.reshape for non-contiguous tensor in ce loss function (#591) · unsloth_zoo	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	Harden fused-forward AST rewriter and adapter · unsloth_zoo	9.2	9.2
pooled: unsloth-zoo + vllm + llama.cpp	Fp8 compressed (#358) · unsloth_zoo	9.0	9.0
pooled: unsloth-zoo + vllm + llama.cpp	[Bugfix] Restrict FlashInfer cuDNN FP8 ViT attention gate to Blackwell (SM 100) (#45251) · vllm	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	[Kernel][Bugfix] Fix INT8 per-token-head KV cache rounding in Triton reshape-and-cache (#45361) · tests	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	[Bugfix][Quantization] Fix W8A8 int-quantized scheme selection regression (#46860) · tests	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	[Kernel] Enable TritonW4A16LinearKernel as CUDA fallback for non-Marlin-aligned W4A16 shapes (#43731) · vllm	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	[Perf] Fix dsv3_router_gemm heuristic (#44217) · vllm	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	[SYCL] Add Q8_0 reorder optimization (~3x tg speedup on Intel Arc) (#21527) · ggml	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	CUDA: Various fixes to `cpy.cu` (#25000) · ggml	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	CUDA & CPU: support F32 kernel type for `CONV_TRANSPOSE_2D` (#17094) · ggml	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	ggml: vectorize ggml_vec_dot_q4_1_q8_1 with WASM SIMD128 (#22209) · ggml	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	fix: GLM-DSA crash in llama-tokenize when using vocab_only (#22102) · src	10.0	10.0
pooled: unsloth-zoo + vllm + llama.cpp	ggml : fix ARM NEON nvfp4 dot product on non-dotprod targets (#21559) · ggml	8.5	8.5
unslothai/unsloth-zoo	Use torch.Tensor.reshape for non-contiguous tensor in ce loss function (#591) · unsloth_zoo	10.0	10.0
unslothai/unsloth-zoo	Harden fused-forward AST rewriter and adapter · unsloth_zoo	9.2	9.2
unslothai/unsloth-zoo	Fp8 compressed (#358) · unsloth_zoo	9.0	9.0
vllm-project/vllm	[Bugfix] Restrict FlashInfer cuDNN FP8 ViT attention gate to Blackwell (SM 100) (#45251) · vllm	10.0	10.0
vllm-project/vllm	[Kernel][Bugfix] Fix INT8 per-token-head KV cache rounding in Triton reshape-and-cache (#45361) · tests	10.0	10.0
vllm-project/vllm	[Bugfix][Quantization] Fix W8A8 int-quantized scheme selection regression (#46860) · tests	10.0	10.0
vllm-project/vllm	[Kernel] Enable TritonW4A16LinearKernel as CUDA fallback for non-Marlin-aligned W4A16 shapes (#43731) · vllm	10.0	10.0
vllm-project/vllm	[Perf] Fix dsv3_router_gemm heuristic (#44217) · vllm	10.0	10.0
ggml-org/llama.cpp	model: mistral small 4 support (#20649) · convert_hf_to_gguf.py	-	9.7
ggml-org/llama.cpp	sycl : support nvfp4 type in mul_mat (#21227) · ggml	9.7	-
ggml-org/llama.cpp	ggml : extend im2col f16 (ggml/1434) · ggml	10.0	-
ggml-org/llama.cpp	hexagon: dma optimizations (mostly fixing regressions) (#21137) · ggml	7.0	-
ggml-org/llama.cpp	sycl : fix the failed UT cases of conv_3d (#24900) · ggml	1.5	-
ggml-org/llama.cpp	CUDA : Fix CUB's argsort when nrows % block_size == 0 CCCL < 3.1 (#21181) · ggml	8.0	-
ggml-org/llama.cpp	llama-fit: fix regex pattern for gate_up tensors (#20910) · src	7.0	-
pooled: unsloth-zoo + vllm + llama.cpp	Fix LoRA scaling count mismatch on merge for Qwen2.5-VL exports (#2966) (#806) · tests	9.0	-
pooled: unsloth-zoo + vllm + llama.cpp	[Bugfix] Fix corrupt outputs in MoE FP8 LoRA responses and MoE base model responses when LoRAs are loaded (#42120) · tests	-	0.8
pooled: unsloth-zoo + vllm + llama.cpp	[AMD][OCP MX][CI] Fix tests to not dispatch on `UNFUSED_TRITON` backend on MI300, improve w_mxfp4_a_fp8 emulation support (#46142) · tests	-	4.5
pooled: unsloth-zoo + vllm + llama.cpp	[Bugfix] Support non-power-of-2 top_k in legacy triton_kernels routing (#46406) · vllm	10.0	-
pooled: unsloth-zoo + vllm + llama.cpp	[Bugfix][Kernel] Fix mHC fused-RMSNorm big-fuse miscompile for hidden_size != 4096 (#44692) · vllm	10.0	-
pooled: unsloth-zoo + vllm + llama.cpp	Add weights padding for fp8 per-block online quantization (#44763) · vllm	-	10.0
pooled: unsloth-zoo + vllm + llama.cpp	model: mistral small 4 support (#20649) · convert_hf_to_gguf.py	-	9.7
pooled: unsloth-zoo + vllm + llama.cpp	sycl : support nvfp4 type in mul_mat (#21227) · ggml	9.7	-
pooled: unsloth-zoo + vllm + llama.cpp	ggml : extend im2col f16 (ggml/1434) · ggml	10.0	-
pooled: unsloth-zoo + vllm + llama.cpp	hexagon: dma optimizations (mostly fixing regressions) (#21137) · ggml	7.0	-
pooled: unsloth-zoo + vllm + llama.cpp	sycl : fix the failed UT cases of conv_3d (#24900) · ggml	1.5	-
pooled: unsloth-zoo + vllm + llama.cpp	CUDA : Fix CUB's argsort when nrows % block_size == 0 CCCL < 3.1 (#21181) · ggml	8.0	-
pooled: unsloth-zoo + vllm + llama.cpp	llama-fit: fix regex pattern for gate_up tensors (#20910) · src	7.0	-
unslothai/unsloth-zoo	Fix LoRA scaling count mismatch on merge for Qwen2.5-VL exports (#2966) (#806) · tests	9.0	-
vllm-project/vllm	[Bugfix] Fix corrupt outputs in MoE FP8 LoRA responses and MoE base model responses when LoRAs are loaded (#42120) · tests	-	0.8
vllm-project/vllm	[AMD][OCP MX][CI] Fix tests to not dispatch on `UNFUSED_TRITON` backend on MI300, improve w_mxfp4_a_fp8 emulation support (#46142) · tests	-	4.5
vllm-project/vllm	[Bugfix] Support non-power-of-2 top_k in legacy triton_kernels routing (#46406) · vllm	10.0	-
vllm-project/vllm	[Bugfix][Kernel] Fix mHC fused-RMSNorm big-fuse miscompile for hidden_size != 4096 (#44692) · vllm	10.0	-
vllm-project/vllm	Add weights padding for fp8 per-block online quantization (#44763) · vllm	-	10.0

Scores run from 0 to 10. Head-to-head counts only the 114 tasks where both models ran. Bold marks the winner on that task.

By area of the codebase

Area	Tasks	Claude Opus 4.8	GLM-5.2
ggml	44	8.9	7.7
vllm	30	8.4	5.9
unsloth_zoo	28	9.0	7.3
tests	22	8.4	4.9
conversion	4	10.0	4.5
src	4	8.5	10.0
examples	2	8.6	0.0
convert_hf_to_gguf.py	2	-	9.7

How the benchmark works

A task here is a single commit. CommitBench checks out the repository one commit before the change, writes a short description of the work from the diff, and gives the agent nothing but that description and the codebase. The agent navigates and edits the files itself, the same way it would on a real ticket.

Every agent's change is then compared head-to-head against every other one — and against the human commit — by a blind judge that never sees which model wrote which diff. Those pairwise wins become a Bradley-Terry rating (Elo-scaled, with the human commit anchored at 0), which is the ranking shown here. A fixed rubric still scores each change for the per-task detail. Cost and speed come from each run's own token and timing reports, priced at one consistent rate so the numbers line up across agents.

Common questions

Which is better for coding, Claude Opus 4.8 or GLM-5.2?

Claude Opus 4.8 ranks higher overall across 140 tasks from 4 public repositories. On the pairwise Bradley-Terry rating (Elo-scaled, anchored on the human commit at 0), Claude Opus 4.8 sits at -1 and GLM-5.2 at -84. Claude Opus 4.8 costs about $103.088 per run, GLM-5.2 about $45.754 per run.

How is the ranking calculated?

Each task is a real commit from a public repository. The agent gets the code at the commit before the change and a written description of the work, then edits the files itself. Every agent's diff is then compared head-to-head against every other one — and against the human commit — by a blind judge that never sees which model produced which diff. Those pairwise wins and losses become a Bradley-Terry rating (Elo-scaled), so the ranking reflects which changes actually win comparisons, not whether they match the original wording.

Could the agents have seen the answers already?

Two things limit that. The task description is rewritten from the change rather than copied from the commit message, and the grader is given the rubric instead of the original answer. Building tasks from recent commits also reduces how much a model could have trained on the result.

Can I compare Claude Opus 4.8 and GLM-5.2 on my own code?

Yes. Point CommitBench at your repository and it builds the same kind of tasks from your own commit history, so the ranking reflects the code you actually work on.

Other comparisons

Claude Opus vs Claude Sonnet Claude Opus vs GPT-5.5 Claude Sonnet vs GPT-5.5 Claude Sonnet vs GLM-5.2 GPT-5.5 vs GLM-5.2

Run this on your codebase

Public repos give the general picture. Benchmark Claude Opus 4.8 against GLM-5.2 on your own commit history to see which one does better on the code you actually ship.

Benchmark your repo →