arc-task-gen Hit 11,000 Stars on 12 Commits and One Column We Cannot Check

Our own star tracker caught this one mid-climb. On 31 August at 05:49 UTC, pathwaycom/arc-task-gen sat at 9,064 stars. By 2 September at 16:00 UTC it was at 10,198, which is 1,134 stars in 58 hours, or roughly 469 a day. When we checked the GitHub API on 7 September at 13:14 UTC it read 11,052. That is another 854 stars, but spread over five days, so the rate has fallen to about 175 a day.

Behind that curve is a repository with 12 commits, all of them landed between 4 and 11 August, all of them by one person. It has 69 forks, 1 open issue, and 19 watchers. Eleven thousand people starred a project that fewer than twenty are subscribed to. That is worth a look, so we cloned it and read all 1,890 lines.

What it actually is

ARC-AGI-1's public evaluation set is 400 grid puzzles, and it has been sitting on GitHub since 2019, which means it is in every web-scraped training corpus on earth. A model's score on it is therefore an upper bound on genuine few-shot rule induction, not a measurement of it. arc-task-gen's job is to build you a fresh private set with the same measurable shape, so the two scores can be compared.

It works by sampling target properties from the real evaluation set, asking a model to invent a task hitting those numbers, then filtering. The workflow is two commands:

python describe_eval_tasks.py     # once, ~5 min, caches rule descriptions
python generate_tasks.py --n 400  # generate a matched task set

The prompt is the interesting artefact. It hands the model five simultaneous numeric constraints (approximate rows, approximate columns, approximate distinct colours, exact training pair count, exact test pair count) and demands strict JSON back, while explicitly forbidding derivation from any existing ARC task, "public or private".

We reproduced the reference column exactly

The README publishes a two-column table: the real evaluation set's distribution next to a generated set's. Half of that table is checkable without spending a cent, so we checked it.

We pulled the ARC-AGI-1 evaluation set from the same tarball URL the script uses, got 400 tasks, then loaded generate_tasks.py with its OpenAI and httpx imports stubbed out and called the repository's own compute_stats() against the real data. Every reference number came back identical:

Property	README says	We measured
input area, mean	226.92	226.92
input area, median	144.00	144.00
input rows, mean	13.23	13.23
input cols, mean	13.70	13.70
colours, mean	5.35	5.35
training pairs 4 or more	34.2%	34.2% (137/400)
test pairs exactly 2	4.8%	4.8% (19/400)

No rounding drift, no cherry-picked subset. That is a good sign about the code, and it is the part of the claim we can stand behind.

While we were in there we computed something the README leaves out. Input area has a standard deviation of 226.07 against a mean of 226.92, with a minimum of 4 and a maximum of 900. The distribution is wildly skewed, which is why matching a mean on its own would be weak evidence. To Pathway's credit, they match the median and the pair-count shape too, which is the right answer to that objection.

The half we could not run

Generation needs an OpenAI-compatible endpoint and defaults to gpt-5.6. We did not run it, so every number in the generated column of the README stays attributed to Pathway. Their own note on cost is useful context: a 400-task run on gpt-5.6 logged 27 parse failures across 758 calls, and a weaker generation model costs extra calls because it drops constraints and fails the JSON contract.

What we wanted to verify	Could we?
Reference distribution numbers	Yes, all 7 reproduced exactly
Source read end to end	Yes, 1,890 lines across 5 files
Generated distribution numbers	No, needs a paid endpoint
Anyone else's generated set	No, they are never published
Whether generated tasks are solvable	No, nothing in the repo checks this

That last row is not a gotcha, it is just the design. validate_tasks() checks structure and only structure: at least two training pairs, rectangular non-ragged grids, cell values within 0 to 9. Solvability is delegated to the generating model, which the prompt asks to verify its own work before returning. The docstring next to it is refreshingly candid about why even the structural check exists: "Two ragged grids reached a scoring run before this was enforced."

What it does not do

• Gate on colour count. The README table reports colours mean 5.35 against 5.24 generated, but run_sanity_check() never checks it. It runs six checks: input area mean and median, input rows mean, input cols mean, the modal training pair count, and the single-test-pair fraction. Colour distribution is measured, printed, and then not enforced.
• Hold a tight tolerance. TOLERANCE = 0.40. An input area mean anywhere from 136.15 to 317.69 passes against a reference of 226.92. Their reported run came in at 224.70, a 1.0% deviation, so the actual output is forty times tighter than the gate requires. The gate is nearly decorative.
• Match its own stated threshold. The single-test-pair check labels the expected value as ">=95%" and then passes anything at 80% or above. We measured the real set at 95.25%.
• Filter for eval similarity unless you pay first. This is the one that matters. The check that removes generated tasks too close to real evaluation tasks needs embeddings of the evaluation set, which means running describe_eval_tasks.py and label_eval_tasks.py first, which means more API calls. Skip that step and generate_tasks.py prints a warning and carries on without the filter. The property the whole tool exists to guarantee is opt-in.
• Get maintained. Last commit 11 August. The stars kept arriving for another four weeks.

We want to be fair about the similarity threshold itself, because the reasoning behind it is the best comment in the codebase. It sits at 0.92 cosine similarity, and the code explains that nearest-neighbour similarity between genuinely distinct evaluation tasks has a median of 0.760, a 95th percentile of 0.879 and a maximum of 0.912, with 38 of 400 pairs above 0.85. So a threshold of 0.85 would delete legitimate novel work. Setting it just above the observed maximum for unrelated real tasks is the defensible choice, and they showed the calibration instead of asserting it. Most repositories at this star count do not.

Who made it, and does that matter

Twelve commits, one human: Ludovic Arnould, committing as ludovicPathway. Six of those twelve commits are README edits from a single day. The bus factor is 1, and the repository is supplementary code for a paper (Pathway's BDH-CQ, which the README says reaches 29.5% pass@2 on public ARC-AGI-1 at a computed $0.0007 per task, and which we have not independently evaluated).

The bus factor barely matters here, though, and that is the honest read. This is 1,890 lines of dependency-light Python: numpy, httpx, openai, matplotlib. If Pathway abandons it tomorrow you can vendor the whole thing into your evaluation harness in an afternoon and maintain it yourself. Small enough to own is worth more than actively maintained.

Verdict

Adopt today if you are evaluating your own model and want an internal check on how much of your public ARC-AGI-1 score is memorisation. That is the use case the tool is built for, and the distribution matching holds up under the parts we could test.

Wait if you were hoping for a shared benchmark, a leaderboard, or a way to compare against a published result. It cannot do that, not because the code is weak but because the privacy that makes the tasks valuable is the same privacy that makes them unauditable.

A benchmark whose tasks can never be published is a private instrument, not a public one. Both are useful. Only one is a leaderboard.

What would change our mind: run describe_eval_tasks.py, generate a set, and publish the sanity_check.json plus the similarity histogram without the tasks themselves. That would make the generated column checkable without burning the set. It is a small amount of work and it is the difference between a claim and evidence.