M Survey Clean Experiment Progress

Updated 2026-11-01T21:39:27.893Z · outdir /home/arbel/sites/m_survey/clean_experiment/outputs

Topline Results

Pooled Unmasking Accuracy

88.3%

Scenario-balanced clean · Original manuscript: 91.9% · Delta: -3.5 pp

Raw clean: 201/240 · Raw original parse: 203/230

Pooled Strict Reconstruction

53.9%

Scenario-balanced clean · Original manuscript: 61.4% · Delta: -7.5 pp

Raw clean: 97/240 · Raw original parse: 106/230

Attempts

656

Canonical Runs

120

Canonical Items

1257

Call Errors

Parse Errors

By Model

model	canonical_runs	canonical_items	attempts	call_errors	parse_errors	cache_hits
gemini-3.1-pro	20	210	134	0	18	40
glm-5.1	20	210	80	0	0	3
gpt-5.4	20	209	160	0	0	84
grok-4.2	20	208	80	0	0	6
opus-4.6	20	210	119	32	0	13
qwen-3.6	20	210	83	0	0	2

model

canonical_runs

canonical_items

attempts

call_errors

parse_errors

cache_hits

gemini-3.1-pro

210

134

glm-5.1

210

gpt-5.4

209

160

grok-4.2

208

opus-4.6

210

119

qwen-3.6

210

Scorer: Clean vs Original

Q1 and strict columns use the manuscript's scenario-balanced estimator. Original manuscript source: /home/arbel/sites/m_survey/analysis_build/llm_summary.json. Raw answer-change matching still uses /home/arbel/sites/m_survey/llm_runs/run_records.jsonl.

model	original_runs	clean_runs	original_q1	clean_q1	q1_delta	original_objective	clean_objective	objective_delta	original_strict	clean_strict	strict_delta
gemini-3.1-pro	20	20	70.4%	96.7%	+26.3 pp	70.8%	75.6%	+4.7 pp	60.0%	66.7%	+6.7 pp
glm-5.1	20	20	95.8%	70.0%	-25.8 pp	76.1%	70.8%	-5.2 pp	66.7%	66.7%	+0.0 pp
gpt-5.4	20	20	96.7%	88.3%	-8.3 pp	77.3%	70.2%	-7.0 pp	66.7%	50.0%	-16.7 pp
grok-4.2	20	20	100.0%	93.3%	-6.7 pp	72.2%	68.2%	-4.0 pp	56.7%	40.0%	-16.7 pp
opus-4.6	20	20	100.0%	100.0%	+0.0 pp	77.8%	77.8%	+0.0 pp	66.7%	66.7%	+0.0 pp
qwen-3.6	20	20	88.3%	81.7%	-6.7 pp	71.1%	62.0%	-9.1 pp	53.3%	33.3%	-20.0 pp

model

original_runs

clean_runs

original_q1

clean_q1

q1_delta

original_objective

clean_objective

objective_delta

original_strict

clean_strict

strict_delta

gemini-3.1-pro

70.4%

96.7%

+26.3 pp

70.8%

75.6%

+4.7 pp

60.0%

66.7%

+6.7 pp

glm-5.1

95.8%

70.0%

-25.8 pp

76.1%

70.8%

-5.2 pp

66.7%

+0.0 pp

gpt-5.4

96.7%

88.3%

-8.3 pp

77.3%

70.2%

-7.0 pp

66.7%

50.0%

-16.7 pp

grok-4.2

100.0%

93.3%

-6.7 pp

72.2%

68.2%

-4.0 pp

56.7%

40.0%

-16.7 pp

opus-4.6

100.0%

+0.0 pp

77.8%

+0.0 pp

66.7%

+0.0 pp

qwen-3.6

88.3%

81.7%

-6.7 pp

71.1%

62.0%

-9.1 pp

53.3%

33.3%

-20.0 pp

Answer Changes

Matched on model, run index, scenario, and objective question. Historical failed attempts are ignored.

model	matched_objective_items	answer_changed	answer_change_rate	correctness_changed	correctness_change_rate	improved	worsened
gemini-3.1-pro	41	15	36.6%	13	31.7%	13	0
glm-5.1	40	16	40.0%	15	37.5%	1	14
gpt-5.4	52	9	17.3%	9	17.3%	2	7
grok-4.2	59	8	13.6%	5	8.5%	2	3
opus-4.6	70	0	0.0%	0	0.0%	0	0
qwen-3.6	58	12	20.7%	10	17.2%	1	9

model

matched_objective_items

answer_changed

answer_change_rate

correctness_changed

correctness_change_rate

improved

worsened

gemini-3.1-pro

36.6%

31.7%

glm-5.1

40.0%

37.5%

gpt-5.4

17.3%

grok-4.2

13.6%

8.5%

opus-4.6

0.0%

qwen-3.6

20.7%

17.2%

Clean Scores By Scenario

model	scenario_id	exposures	q1_accuracy	objective_accuracy	strict_accuracy	attn_pass_rate
gemini-3.1-pro	artist	10	100.0%	100.0%	100.0%	100.0%
gemini-3.1-pro	contingency	10	100.0%	100.0%	100.0%	100.0%
gemini-3.1-pro	milos	20	90.0%	45.0%	0.0%	100.0%
glm-5.1	artist	10	100.0%	100.0%	100.0%	100.0%
glm-5.1	contingency	10	100.0%	96.7%	100.0%	100.0%
glm-5.1	milos	20	10.0%	9.1%	0.0%	100.0%
gpt-5.4	artist	10	100.0%	80.0%	60.0%	100.0%
gpt-5.4	contingency	10	90.0%	96.6%	90.0%	100.0%
gpt-5.4	milos	20	75.0%	42.9%	0.0%	100.0%
grok-4.2	artist	10	100.0%	75.0%	50.0%	100.0%
grok-4.2	contingency	10	80.0%	89.3%	70.0%	100.0%
grok-4.2	milos	20	100.0%	50.0%	0.0%	100.0%
opus-4.6	artist	10	100.0%	100.0%	100.0%	100.0%
opus-4.6	contingency	10	100.0%	100.0%	100.0%	100.0%
opus-4.6	milos	20	100.0%	50.0%	0.0%	100.0%
qwen-3.6	artist	10	100.0%	50.0%	0.0%	100.0%
qwen-3.6	contingency	10	100.0%	100.0%	100.0%	100.0%
qwen-3.6	milos	20	45.0%	31.0%	0.0%	100.0%

model

scenario_id

exposures

q1_accuracy

objective_accuracy

strict_accuracy

attn_pass_rate

gemini-3.1-pro

artist

100.0%

gemini-3.1-pro

contingency

100.0%

gemini-3.1-pro

milos

90.0%

45.0%

0.0%

100.0%

glm-5.1

artist

100.0%

glm-5.1

contingency

100.0%

96.7%

100.0%

glm-5.1

milos

10.0%

9.1%

0.0%

100.0%

gpt-5.4

artist

100.0%

80.0%

60.0%

100.0%

gpt-5.4

contingency

90.0%

96.6%

90.0%

100.0%

gpt-5.4

milos

75.0%

42.9%

0.0%

100.0%

grok-4.2

artist

100.0%

75.0%

50.0%

100.0%

grok-4.2

contingency

80.0%

89.3%

70.0%

100.0%

grok-4.2

milos

100.0%

50.0%

0.0%

100.0%

opus-4.6

artist

100.0%

opus-4.6

contingency

100.0%

opus-4.6

milos

100.0%

50.0%

0.0%

100.0%

qwen-3.6

artist

100.0%

50.0%

0.0%

100.0%

qwen-3.6

contingency

100.0%

qwen-3.6

milos

45.0%

31.0%

0.0%

100.0%

Latest Attempts

model_key	run_id	scenario_id	step	cache_hit	timestamp
glm-5.1	glm-5.1_0_016	contingency	1	false	2026-06-22T12:19:24.867445+00:00
glm-5.1	glm-5.1_0_016	contingency	2	false	2026-06-22T12:20:05.789240+00:00
glm-5.1	glm-5.1_0_016	milos	1	false	2026-06-22T12:20:35.488446+00:00
glm-5.1	glm-5.1_0_016	milos	2	false	2026-06-22T12:20:51.266256+00:00
glm-5.1	glm-5.1_0_017	contingency	1	false	2026-06-22T12:21:08.023066+00:00
glm-5.1	glm-5.1_0_017	contingency	2	false	2026-06-22T12:21:33.093673+00:00
glm-5.1	glm-5.1_0_017	milos	1	false	2026-06-22T12:22:13.734436+00:00
glm-5.1	glm-5.1_0_017	milos	2	false	2026-06-22T12:22:33.382429+00:00
glm-5.1	glm-5.1_0_018	milos	1	false	2026-06-22T12:22:47.706896+00:00
glm-5.1	glm-5.1_0_018	milos	2	false	2026-06-22T12:23:14.242682+00:00
glm-5.1	glm-5.1_0_018	artist	1	false	2026-06-22T12:23:37.521249+00:00
glm-5.1	glm-5.1_0_018	artist	2	false	2026-06-22T12:23:59.727172+00:00
glm-5.1	glm-5.1_0_019	milos	1	false	2026-06-22T12:24:14.797215+00:00
glm-5.1	glm-5.1_0_019	milos	2	true	2026-06-22T12:24:14.798094+00:00
glm-5.1	glm-5.1_0_019	artist	1	false	2026-06-22T12:24:35.828478+00:00
glm-5.1	glm-5.1_0_019	artist	2	false	2026-06-22T12:25:12.917002+00:00
glm-5.1	glm-5.1_0_020	milos	1	false	2026-06-22T12:25:38.665433+00:00
glm-5.1	glm-5.1_0_020	milos	2	false	2026-06-22T12:26:04.992843+00:00
glm-5.1	glm-5.1_0_020	artist	1	false	2026-06-22T12:26:17.853963+00:00
glm-5.1	glm-5.1_0_020	artist	2	false	2026-06-22T12:26:52.904895+00:00

model_key

run_id

scenario_id

step

cache_hit

call_error

parse_error

timestamp

glm-5.1

glm-5.1_0_016

contingency

false

2026-06-22T12:19:24.867445+00:00

glm-5.1

glm-5.1_0_016

contingency

false

2026-06-22T12:20:05.789240+00:00

glm-5.1

glm-5.1_0_016

milos

false

2026-06-22T12:20:35.488446+00:00

glm-5.1

glm-5.1_0_016

milos

false

2026-06-22T12:20:51.266256+00:00

glm-5.1

glm-5.1_0_017

contingency

false

2026-06-22T12:21:08.023066+00:00

glm-5.1

glm-5.1_0_017

contingency

false

2026-06-22T12:21:33.093673+00:00

glm-5.1

glm-5.1_0_017

milos

false

2026-06-22T12:22:13.734436+00:00

glm-5.1

glm-5.1_0_017

milos

false

2026-06-22T12:22:33.382429+00:00

glm-5.1

glm-5.1_0_018

milos

false

2026-06-22T12:22:47.706896+00:00

glm-5.1

glm-5.1_0_018

milos

false

2026-06-22T12:23:14.242682+00:00

glm-5.1

glm-5.1_0_018

artist

false

2026-06-22T12:23:37.521249+00:00

glm-5.1

glm-5.1_0_018

artist

false

2026-06-22T12:23:59.727172+00:00

glm-5.1

glm-5.1_0_019

milos

false

2026-06-22T12:24:14.797215+00:00

glm-5.1

glm-5.1_0_019

milos

true

2026-06-22T12:24:14.798094+00:00

glm-5.1

glm-5.1_0_019

artist

false

2026-06-22T12:24:35.828478+00:00

glm-5.1

glm-5.1_0_019

artist

false

2026-06-22T12:25:12.917002+00:00

glm-5.1

glm-5.1_0_020

milos

false

2026-06-22T12:25:38.665433+00:00

glm-5.1

glm-5.1_0_020

milos

false

2026-06-22T12:26:04.992843+00:00

glm-5.1

glm-5.1_0_020

artist

false

2026-06-22T12:26:17.853963+00:00

glm-5.1

glm-5.1_0_020

artist

false

2026-06-22T12:26:52.904895+00:00

Recent Errors

model_key	run_id	scenario_id	step	call_error	parse_error
opus-4.6	opus-4.6_0_017	milos	1	HTTP 403: {"error":{"message":"Key limit exceeded (total limit). Manage it using [openrouter key settings URL redacted]","code":403}}
opus-4.6	opus-4.6_0_018	milos	1	HTTP 403: {"error":{"message":"Key limit exceeded (total limit). Manage it using [openrouter key settings URL redacted]","code":403}}
gemini-3.1-pro	gemini-3.1-pro_0_002	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_003	milos	1		Unterminated string starting at: line 6 column 11 (char 83)
gemini-3.1-pro	gemini-3.1-pro_0_010	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_013	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_015	milos	1		Expecting value: line 6 column 10 (char 82)
gemini-3.1-pro	gemini-3.1-pro_0_019	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_002	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_003	milos	1		Unterminated string starting at: line 6 column 11 (char 83)
gemini-3.1-pro	gemini-3.1-pro_0_010	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_013	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_015	milos	1		Expecting value: line 6 column 10 (char 82)
gemini-3.1-pro	gemini-3.1-pro_0_019	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_002	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_003	milos	1		Unterminated string starting at: line 6 column 11 (char 83)
gemini-3.1-pro	gemini-3.1-pro_0_010	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_013	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)
gemini-3.1-pro	gemini-3.1-pro_0_015	milos	1		Expecting value: line 6 column 10 (char 82)
gemini-3.1-pro	gemini-3.1-pro_0_019	milos	1		Expecting ',' delimiter: line 6 column 14 (char 86)

model_key

run_id

scenario_id

step

call_error

parse_error

opus-4.6

opus-4.6_0_017

milos

HTTP 403: {"error":{"message":"Key limit exceeded (total limit). Manage it using [openrouter key settings URL redacted]","code":403}}

opus-4.6

opus-4.6_0_018

milos

HTTP 403: {"error":{"message":"Key limit exceeded (total limit). Manage it using [openrouter key settings URL redacted]","code":403}}

gemini-3.1-pro

gemini-3.1-pro_0_002

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_003

milos

Unterminated string starting at: line 6 column 11 (char 83)

gemini-3.1-pro

gemini-3.1-pro_0_010

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_013

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_015

milos

Expecting value: line 6 column 10 (char 82)

gemini-3.1-pro

gemini-3.1-pro_0_019

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_002

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_003

milos

Unterminated string starting at: line 6 column 11 (char 83)

gemini-3.1-pro

gemini-3.1-pro_0_010

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_013

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_015

milos

Expecting value: line 6 column 10 (char 82)

gemini-3.1-pro

gemini-3.1-pro_0_019

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_002

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_003

milos

Unterminated string starting at: line 6 column 11 (char 83)

gemini-3.1-pro

gemini-3.1-pro_0_010

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_013

milos

Expecting ',' delimiter: line 6 column 14 (char 86)

gemini-3.1-pro

gemini-3.1-pro_0_015

milos

Expecting value: line 6 column 10 (char 82)

gemini-3.1-pro

gemini-3.1-pro_0_019

milos

Expecting ',' delimiter: line 6 column 14 (char 86)