Check AI self-improvement claims with fixed-seed reruns

You are reading a vendor post: its agent improved a model, and the number looks good. AutoTrust's self-reported NanoGPT Speedrun time was 24.90 seconds, compared with Recursive's 75.36-second June record.

AutoTrust said its NanoGPT Speedrun and Time-to-GPT-2 results had not yet been reviewed by benchmark maintainers. Treat the claim as unverified until a fixed-seed rerun on a public benchmark with a published target says otherwise.

A startup of about 10 researchers spent September generating results on public evaluations of AI that improves AI. AutoTrust AI reported that its ScienceGuru agentic research system produced results on four public benchmarks for testing self-improving AI.

• Choose a public benchmark with a published target, a fixed validation set, and a hardware spec.
• Lock the seed and record the run log before the agent starts.
• Run N reruns on comparable hardware, recording final loss, elapsed time, and crashes.
• Compare the result to the public record, not the vendor narrative.
• Label the claim verified, unverified, or not comparable.

Choose a benchmark with a published target

The target must be a number you can check, not a leaderboard rank that can move. AutoTrust reported that ScienceGuru finished first on Autoresearch@Home and was the leading validated submission on MedARC's NanoPath when posted. A rank is useful, but it is not a fixed target. For a speedrun-style test, the target should be a loss value, where lower is better, or a time-to-loss threshold. The modded-nanogpt repository describes the NanoGPT speedrun as the fastest route to hit a 3.28 cross-entropy loss on FineWeb validation data using eight NVIDIA H100 GPUs. The 3.28 FineWeb loss target was chosen to match Andrej Karpathy's llm.c GPT-2 reproduction, which the repository says reaches that loss in 45 minutes.

Prefer a benchmark where you can download the validation set and inspect the scoring script. The page should show the target, dataset, GPU count, and stopping rule. A rank alone is weak; ask what it is based on before spending compute.

Lock the seed and the hardware

Record the random seed before the agent starts. The run log should name the seed, model architecture, optimizer settings, batch size, and stopping condition. An agent that chooses its own seed after a bad result is not doing a rerun. A missing log means stop and request it before spending compute.

Use comparable hardware: same GPU class, GPU count, memory, and software stack as the original claim. Unmatched hardware means mark the comparison not comparable. A faster machine can make a weak agent look clever; a slower machine can make a strong agent look lazy.

Treating a vendor screenshot as a run is the common failure. It can hide the seed, stopping rule, or hardware. Ask for the run log, not the highlight reel.

Run the rerun against the public record

Run the same task with the locked seed on comparable hardware, more than once when compute allows. Record the final loss, elapsed time, and number of reruns. A result that misses the target has not survived. A crashed rerun is a result, not a fluke.

Compare the result to the public record, not to the vendor's narrative. A public leaderboard is useful when it states the validation rule and submission status. Unreviewed status matters: a result that has not yet been reviewed is a starting point, not a finished proof. Until review is complete, the claim needs the same fixed-seed scrutiny you would apply to any other result.

Label the claim, not the vendor

Write the result as one of three labels: verified, unverified, or not comparable. Verified requires the fixed-seed rerun to meet the published target on comparable hardware. Unverified covers a missed target or a missing run log. Not comparable covers hardware, seed, or stopping-rule differences from the original claim.

Keep the label attached to the claim, not the company. A system can be good at one benchmark and weak at another. A vendor can be honest about one result and sloppy about another. The fixed-seed rerun gives you a clean way to say which is which.

Keep the evidence in one place: claim, benchmark, seed, hardware, result, label. That makes the next evaluation faster and less emotional. When someone asks why you rejected the claim, show the record.