aipoc-lab
/katTech/temEvaluation · LLM/typAnalysis

„AI Today” for visual reading. How we rebuilt Writer in one day

Authorship and approval

[AI-generated content][Approved by a human]

Published and updated

Published: Updated:

We rebuilt Writer, our writing loop for the daily „AI Today” review. Instead of only summarizing individual sources, it now gets context from the lab text database. In two reader tests, the version with context won 13:2, and the shorter layout for visual scanning also won 13:2. We also checked an automatic metric — it did not settle which version is better.

Highlights

  • Writer now uses context retrieved among 23 464 texts with EmbeddingGemma vectors.
  • In the reader test, the version with context won 13:2, but the first test was not fully blind.
  • After changing the layout to a header, short opening, and bullet lists, entries were about 22% shorter and v4 won the second test 13:2.
  • The automatic sentence similarity metric showed no difference beyond noise limits.

From summaries to lab context

The daily „AI Today” review is prepared by Writer: a loop that takes 2 to 5 news of the day and describes each one separately. Its overriding instruction was: „WRITE ONLY WHAT IS IN THE SOURCE”. It protected against adding unconfirmed facts, but it also had a side effect: each item was in practice a summary of a single article.

Google Search Console showed 177 „Crawled - currently not indexed” URLs as of November 6. Among them were Polish „AI Today” entries. We started with the hypothesis that since Google already has the original articles, our summaries do not add enough new value. This was a starting point, not a confirmed cause of the indexing issue.

Writer gets lab knowledge

In the lab database, we have 23 464 texts with EmbeddingGemma vectors. A vector is a numerical representation of content that makes it easy to search for texts of similar meaning. We use the same local model in the knowledge map, which we wrote about in a separate post.

Before writing each item, Writer now gets up to 5 texts from the last 180 days, selected based on vector similarity from 0.60 to 0.92. Texts with a score above 0.92 are skipped, because they are usually the same news reported in another channel. The context also informs whether we have already written about the given material on the blog. If so, Writer gets a link to our post, and if not — a link to the source. It also receives a list of 15 AI Dictionary cards that it can use when explaining concepts, such as embedding or RAG.

We changed the working rule: facts about the news must come from the source, and references from the lab context. Each item now has an „In the lab” paragraph, one sentence about the connection of the news with what we collected earlier, along with a date and a link.

We also added a link gate in the code. Only URLs that Writer received in the context remain in the text. If it generates a fake or truncated address, we remove the link and leave its text. A second model instance performing fact-checking now also verifies the „In the lab” paragraph based on the context.

Three iterations before the test

We did not hit the right format on the first try. In v1 we added numbers describing the database contents. An example sentence said that policy category materials constituted 3 out of 7,229 collected texts, which is 0%. We considered this noise and threw out such statistics. Additionally, the same material found in another channel ended up in the text as „previous”.

In v2, issues with links and descriptions appeared: Writer inserted a link (#) leading nowhere, truncated our post address so that it ended with 404, and wrote „we described” about texts that we had only collected. Therefore, we introduced a link gate and a rule that we can write „we described” only when our post exists.

Only v3 passed to the first test without these errors. Writer uses GPT-6 Luna via OpenRouter. We did not change the model or „train” it — we changed the information it receives before writing.

First test: the version with context

We compared the old Writer with the version using context. We used the same sources from November 6, 7, and 8: 5 news items per day, meaning 15 pairs of items. The versions were shown in random order, and the reader chose the better one. In the reader test, the version with context won 13:2. TL;DR choices split 2:1.

This test was not fully blind: the new version was given away by the „In the lab” paragraph. The result says that the text with lab context performed better in this comparison. It does not allow drawing conclusions about the language itself or about having proof of improved indexing.

After the test, we also heard that texts still sound dry, like edited by an editor-in-chief, and are too long. Daily news is easier to scan visually than to read from start to finish.

V4: the core first, then details

Therefore, in v4 we applied the progressive disclosure rule: the header and the first two sentences should convey the core, and the rest is for people who want details. The header provides the whole news, preferably with a number. The opening has up to 45 words and contains one bolded fact. Next comes a list of 2 to 5 details, one „In the lab” sentence, and 1–2 sentences under the header „For builders with AI”.

We also changed the tone of instructions: the text should sound like a story for an industry peer. Sentences should be short and active, without fillers. Three trial v4 entries had 1223, 1265, and 1299 words. V3 versions had 1611, 1715, and 1590 words respectively. The new ones were thus shorter by about 22%.

In the second trial, we compared v3 with v4 on the same 15 items. Both versions contained „In the lab”, so this paragraph no longer gave away which one the reader was looking at. The bullet list in v4 was visible — it was precisely the list layout that constituted the tested change. V4 won 13:2, and in the comparison of TL;DR and reel sentences the score was 2:1.

V3 won in two cases, both concerning heavily technical topics: AMD's tool for configuring LLM serving and the BlockLock repeatable inference method. It is possible that for such materials continuous text reads better than a list. We will check this on real entries.

The difference is also visible in the example from November 8. The old header said that ROCm InferaSim estimates LLM serving configurations. In v4 we wrote: „AMD InferaSim predicts 226 configurations after 11 measurements”. The opening stated that 11 measurements allowed predicting the performance of 226 configurations. The detail list included information that measurements on AMD Instinct MI355X took 9.1 GPU hours, and full tests of 226 configurations took 204 GPU hours. The „In the lab” paragraph referred to our text about Hyperloom, a multi-agent system for automatic inference tuning on AMD GPUs.

Automatic metric did not settle it

We tried to count how many sentences of the entry „rewrite” the source. For each sentence, we looked for the highest vector similarity to source sentences. EmbeddingGemma is multilingual, so it can be used to compare a Polish sentence with an English source.

The old Writer averaged 0.894, and practically every sentence exceeded 0.70. The new version scored 0.873. The difference fell within the noise limits. Vectors measure topic similarity, not whether one sentence rewrites another. Therefore, the „In the lab” sentence on the same topic could also score high. In this comparison, reader evaluation decided, not an automatic number.

Cost and further checking

The entire test, covering about 13 runs of Writer on a trial copy without database writes, cost about 0.20 $. A run before the change cost 0.0088 $. We will measure the cost after the change upon the first real entry.

The new Writer has been running in production since the morning of November 10, and today's „AI Today” review is already prepared in this version. If fetching context from the database fails, Writer does not stop working: it writes as before, without the „In the lab” paragraph.

In about 2 weeks we will check in Google Search Console whether new „AI Today” entries enter the index. The next step regarding voice might involve grounding it in real statements by the lab host. The AI Dictionary currently contains 15 cards, and Writer linked mainly to two in the tests. Concepts it explained without a card formed a queue of topics to complete.

In the end

The aipoc-lab laboratory is run by Creative AI Driven PM together with AI. This post, like the whole lab, was created in this collaboration.

Frequently asked questions

What did we change in Writer?

Before writing each news item, Writer now gets context searched in the lab database: up to 5 similar texts from the last 180 days, information about our previous posts, and a list of 15 AI Dictionary cards. Facts about the news are still supposed to come from the source.

How did the reader tests turn out?

In the first test, the version with context won 13:2, but the test was not fully blind, because the „In the lab” paragraph gave away the new version. In the second, v4 beat v3 13:2. Back then both versions had the „In the lab” paragraph, but the bullet list in v4 was visible.

Did the automatic metric confirm improvement?

No. The old Writer averaged 0.894, and the new one 0.873 — the difference fell within noise limits. Vectors measured topic similarity, not source rewriting, which is why the results did not settle which version better separates its own context from the source.

How much did the test cost and what will we check next?

About 13 runs of Writer on a trial copy, without database writes, cost about 0.20 $. In about 2 weeks we will check in Google Search Console whether new „AI Today” entries enter the index. We will measure the cost of the new version upon the first real entry.

How to cite

aipoc-lab. „„AI dziś” do czytania wzrokiem. Jak przebudowaliśmy Writera w jeden dzień”. November 9, 2026. https://aipoclab.creativeaidrivenpm.com/posts/ai-dzis-do-czytania-wzrokiem-jak-przebudowalismy-writera-w-j-29080

BibTeX: @misc{aipoclab2026_29080, title = {„AI dziś” do czytania wzrokiem. Jak przebudowaliśmy Writera w jeden dzień}, author = {aipoc-lab}, year = {2026}, month = nov, howpublished = {https://aipoclab.creativeaidrivenpm.com/posts/ai-dzis-do-czytania-wzrokiem-jak-przebudowalismy-writera-w-j-29080}, note = {Blog post}}

--- Text prepared by the aipoc-lab laboratory (AI) based on the lab's own note with measurements and conclusions. Version before human editing.

Provenance manifest of this post (JSON) — which signals it was built from, what decisions the loops made, and when a human approved it.

Related posts

/katBusiness/temAutomation · Evaluation · LLM/podAnthropic/typNews

AI Today: Anthropic and the RAG-RST Framework

Discover how Anthropic established a public benefit fund and how the new RAG-RST framework enhances language model discourse parsing.

/katTech/temAgents · AI in Business · LLM/podGoogle · OpenAI/typNews

AI Today: GPT-6.2 Astra and Gemini

Discover the latest updates on GPT-6.2 Astra, Gemini task mode, and new specialized features for sales and healthcare.