Lessons From Running 71 AI Configurations On My News Pipeline

🔍 Read the full analysis: Lessons From Running 71 AI Configurations On My News Pipeline on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a test of 71 AI configurations on his news-writing pipeline found that GPT-6 Astra produced the highest-scoring articles, while lower-cost GPT-6 Luna was the model he chose to run in production. The results, published October 13, also led him to remove two fallback models after his evaluation found fabricated quotes in one and no usable outputs in another.

Meyer tested the configurations from September 8 to October 13, 2026, using the production prompt and two real news sources: a 6,340-character product filing and a 619-character political news item. Each configuration produced articles in English and German. Two AI judges assessed source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with German-language quality also scored.

GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test and 3.93 across 12 articles in a follow-up robustness run. Meyer reported perfect source-fidelity scores in the standard test. But Astra cost about $0.12 per article and took roughly 50 seconds. GPT-6 Luna scored 3.67 in the robustness run, with source fidelity of 4.71, and all its articles were valid under the pipeline’s output checks. Its reported production cost was about $0.0013 per article, with a median generation time of 17 seconds.

The author also tested Luna at six reasoning-effort settings. He reported no reliable quality advantage from raising effort: differences were within the test’s uncertainty, while the maximum setting cost about 12 times more and took about 12 times longer than the no-effort production setting. Meyer said he removed GPT-4.1 nano and locally run gpt-oss-120b from the route on October 7. The former received source-fidelity scores of 1.00, with judges flagging invented quotes; the latter produced zero valid articles in four attempts, according to his account.

Cost and Quality in Production

The test illustrates a practical distinction between a model that tops a quality ranking and one that makes economic sense for a high-volume publishing workflow. In Meyer’s measurements, Astra led on article quality, but its reported per-article cost was about 100 times Luna’s. Luna’s lower score did not make it the best writer in the test; Meyer describes it as the production choice because it cleared his quality and validity requirements at far lower cost.

The reported fallback failures also point to a separate risk from the quality of the primary model. A backup can be invoked precisely when the usual route is unavailable, but if it fabricates quotations or fails formatting checks, it can undermine source fidelity or halt publication. Meyer’s account says his pipeline’s validity rule counted an article only if it produced usable output after the same repair step used in production. That makes the results relevant to publishers deciding how to test not only their main model, but every model in their failover chain.

A Test Built Around News Sources

Meyer framed the experiment as a task-specific evaluation, rather than an attempt to identify the strongest general-purpose model. He writes about public AI leaderboards but says they do not answer which model should produce his site’s daily articles. The evaluation instead used the same prompt as his production pipeline and sources of different lengths, including a short item because he says much of his news work begins with limited source material.

The study had two stages. Initial articles were scored by GPT-6.1 Sol and Claude Opus 5.5, both given the original source. Meyer then ran the strongest configurations on ten additional articles, mostly based on short sources, and reported 95% bootstrap intervals for that robustness run. He cautioned that with only two initial test articles, score gaps below about 0.3 points should be treated as noise; he considered the larger run more useful for ranking. These are results from one publisher’s pipeline and prompts, not a broad independent benchmark.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer

Limits of the Model Comparison

The findings are Meyer’s own evaluation, not a peer-reviewed study or an independent replication. The reported scores depend on his sources, prompt, pipeline repair rules and the two AI judges. The initial test used only two source items, and Meyer himself says small score gaps in that stage may be noise. The ten-article robustness run offers more observations but remains limited in size and scope.

The account does not provide enough information to establish how the models would perform across other publishers, topics, prompts or editorial standards. It also does not report human editors’ scores or a direct comparison with articles written by people. The source material describes Luna as the current first-choice route after the fallback removals, but its final line about the backup configuration is incomplete, so the exact replacement or remaining failover arrangement is unclear.

Monitoring the Production Route

Meyer says GPT-6 Luna is now used for production, while GPT-4.1 nano and gpt-oss-120b were removed from the news route on October 7. The next relevant evidence will be whether Luna continues to meet the pipeline’s quality and validity requirements across a larger stream of real articles, particularly short-source stories like those emphasized in the test.

The report does not give a date for another benchmark or identify the complete current fallback chain. Any future comparison would need to account for changes in model versions, prices and prompts; the published cost and timing figures describe Meyer’s test and production period, not a guarantee of future performance.

Key Questions

Which model scored highest in the test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test, and 3.93 across 12 articles in the robustness run, according to Meyer.

Why did Meyer choose GPT-6 Luna for production?

Meyer reported that Luna scored 3.67 in the robustness run and produced valid articles consistently, while costing about $0.0013 per article in production. He said Astra’s roughly $0.12 cost per article was too high for his news workflow.

Did higher reasoning effort improve Luna’s articles?

Meyer reported that score differences across the six effort settings were within the test’s uncertainty. The maximum setting cost and took about 12 times more than the no-effort setting, without a reliable quality gain.

Why were two fallback models removed?

According to Meyer’s evaluation, GPT-4.1 nano received a source-fidelity score of 1.00, and judges flagged invented quotations. Locally run gpt-oss-120b produced zero valid articles in four attempts, with output-format failures and fabricated named quotes reported.

Do these results establish which AI model is best for all newsrooms?

No. The results reflect one publisher’s prompt, sources and pipeline, assessed by two AI judges. Performance may differ with other material, editorial standards, model versions or workflows.

Source: ThorstenMeyerAI.com