Fiszka 0.3 is public. The model, model card, and a week in production alongside Gemini
Authorship and approval
Published and updated
We are publishing the Fiszka 0.3 weights, model card, and code. For a week, Fiszka assigned topics to new blog posts while Gemini tagged the exact same materials in a shadow comparison mode. Out of 1298 texts tagged by both sides, at least one topic matched in 75% of the cases. We also evaluated the model on manually labeled texts and describe the limitations of its performance.
Key Takeaways
- We published the model on Hugging Face and the code on GitHub under the Apache 2.0 license.
- Fiszka has 252,194 parameters, and the published weights produced the exact same outputs as the production model.
- In a one-week test against Gemini, at least one common topic appeared in 75% of the 1298 texts.
- On two manually labeled datasets, Fiszka achieved an F1 score of 0.51 and 0.55.
- Version 0.4 showed no clear advantage, which is why we decided to stick with 0.3.
Fiszka 0.3 Is Public
We are publishing Fiszka's weights and model card on Hugging Face and its code on GitHub. The model card is in English and covers results as well as limitations. Everything is released under the Apache 2.0 license, which permits commercial use and modifications. The training set (9,968 texts with labels) is published separately as fiszka-dataset, under CC BY 4.0; the training code is in the model repo.
The weights are saved in the safetensors format rather than pickle. Pickle can execute arbitrary code when loading a file, which is why Hugging Face flags that format as unsafe. To run computations, you only need the fiszka.py file and the numpy library—PyTorch is not required. Text vectors are computed locally by Ollama using EmbeddingGemma 2.
Before the release, we compared the published file against the running production model. Across 2000 random vectors, the probability difference was 0.00, and for 22 real texts, the assigned topics matched 22 out of 22 times.
We are not publishing the training texts, as they belong to third-party articles, nor are we releasing the training code yet. The code is tightly coupled with the lab's database and will be published in a second step after a cleanup.
This is the third post about Fiszka: we previously described version 0.1 on 4 November and version 0.3 on 6 November.
How Fiszka Works
Every day we collect AI texts: arXiv papers, company blog posts, and news articles. Each text can receive between 0 and 3 topics from a list of 17 categories, such as agents, safety, robotics, or business. Fiszka is our neural network that assigns these topics.
We built and trained it from scratch on the lab server. The network has 252,194 parameters with layer sizes of 256, 128, and 64, along with a skip connection leading directly to the topics. The external, pre-built component is EmbeddingGemma 2, which turns text into a vector of 768 numbers. Fiszka processes this vector rather than raw text.
Since 7 November, Fiszka has been tagging every new text in the lab. On the /fiszka page, you can select any text and inspect the network's calculations in the browser, layer by layer.
A Week in Production, Gemini in Shadow Mode
From 7 to 13 November, Fiszka tagged blog texts while Gemini received the exact same materials solely for a comparison table. We call this shadow mode: the model operates on real data, but its outputs do not replace production results. On days without new texts, there was nothing to compare.
The table includes 1300 texts for the current version of Fiszka, running on EmbeddingGemma vectors, and 286 texts from the first day for an earlier 0.3 variant that worked on words and TF-IDF. The results below cover 1298 texts tagged by both sides.
Comparison on 1298 texts | Result |
|---|---|
At least one common topic | 75% of texts |
Exactly the same set of topics | 22% of texts |
Average number of topics: Gemini | 1.84 per text |
Average number of topics: Fiszka | 1.90 per text |
Texts without any topic from Fiszka | 13 |
Part of the discrepancies stemmed from topics that Gemini simply lacked in our configuration: machine learning, business, and politics. When we restricted the comparison to 861 texts where Fiszka did not use any of those three topics, at least one common topic appeared in 93% of cases.
The usage of categories also differed. Gemini assigned the "evaluation" topic 371 times, whereas Fiszka did nside it 157 times. In our definition, evaluation refers to work whose main output is a tool to assess other models, such as a benchmark or a metric. When manually labeling arXiv papers, we found about 17% of such texts; in the weekly test, Gemini assigned this category to 29% of texts, while Fiszka gave it to 12%.
The "automation" topic appeared 142 times with Gemini and 29 times with Fiszka. Our prompt for Gemini included a rule to pick automation when nothing else fit. In practice, this category became a catch-all bin for remaining texts with Gemini.
Agreement with Gemini does not prove which side is right. Manual labels serve that purpose.
Results Against Manual Labels
We used the F1 score for evaluation. It combines two factors: whether the model found the correct topics and whether it avoided adding incorrect ones. We manually labeled test texts, and Fiszka never saw them during training.
Test dataset | Fiszka F1 |
|---|---|
First dataset, 75 texts | 0.51 |
Second dataset, 83 newer production texts | 0.55 |
The second, independent dataset maintained a similar performance level. Across 2522 arXiv papers, Fiszka found 80% of the topics derived from the categories assigned by the authors themselves.
Fiszka learned from the labels of two open language models: DeepSeek and Nemotron. A topic entered the training data only when both models agreed. On the second dataset, those labels alone had an F1 of 0.51, and Fiszka scored 0.55. In this comparison, Fiszka matched the quality of the labels it learned from. The limiting factor is the quality of those labels, not the network size.
In our test, Gemini 3.5 Flash Lite tags on the first dataset achieved an F1 of 0.34, compared to Fiszka's 0.51. This is the outcome of our internal test of our configuration, not a general evaluation of Gemini: the tags were generated using our prompt and an older list of 13 topics, excluding business and politics. Part of the difference thus stems from topics we never even asked Gemini about. In our use case, Fiszka performed better, and that is all this number implies. Fiszka did not learn from Gemini tags; we used them solely for comparison.
Why We Publish 0.3 Instead of 0.4
Before the release, we tested version 0.4. It featured the same network structure but additionally read 5000 words directly from the text and had 337,211 parameters. On the first dataset, it achieved an F1 of 0.486 compared to 0.510 for 0.3. On the second dataset, the score was 0.580 compared to 0.554. Overall, it was a tie.
We also attempted to fine-tune Fiszka on 160 manually labeled texts. The F1 score increased by 0.010, which falls within the margin of random chance. With datasets of around 80 texts, differences of 0.03 are just noise. We saw no clear advantage that would justify changing the model, so we are releasing 0.3. For now, we have wrapped up work on improving Fiszka. We will return to it if an external reason arises, such as more manually labeled texts or a stronger teacher model.
Limitations
The test datasets are small, consisting of 75 and 83 texts. Each text was labeled by a single person who showed some inconsistency across certain topics, including language models, evaluation, agents, and open source. Fiszka assigns the "language models" topic too infrequently because one of the teacher models rarely selected it, leaving few such labels in the training set.
The model was trained on English AI texts. We have not tested it on other languages or domains. It works exclusively with vectors from EmbeddingGemma 2 in the exact same input format.
Final Remarks
The lab is run by a single person, Creative AI Driven PM, without a formal technical background. It is built with the help of AI, documenting what worked and what didn't.
Frequently Asked Questions
Does agreement with Gemini mean Fiszka was right?
No. In the weekly test, we measured output agreement, which in itself does not indicate which side assigned the correct topics. To evaluate Fiszka, we compared its outputs against manually labeled texts that the model never saw during training.
Do we publish the training texts and training code?
We do not publish the training texts because they are articles belonging to third parties. We are also not yet releasing the training code, which is intertwined with the lab database. We plan to publish it in a second step after cleanup.
What are Fiszka's limitations?
The test datasets contain 75 and 83 texts, each labeled by a single person. Fiszka was trained on English AI texts and has not been tested in other languages or domains. It only works with EmbeddingGemma 2 vectors in the exact same input format.
Why did we publish version 0.3 instead of 0.4?
The results for 0.4 and 0.3 varied across datasets, resulting in an overall tie. Additional fine-tuning on 160 manually labeled texts raised F1 by 0.010, which is within the margin of chance. With such small datasets, we had no clear reason to change the model.
How to Cite
aipoc-lab. "Fiszka 0.3 jest publiczna. Model, karta i tydzień na produkcji obok Gemini". 14 November 2026. https://aipoclab.creativeaidrivenpm.com/posts/fiszka-0-3-jest-publiczna-model-karta-i-tydzien-na-produkcji-30442
BibTeX: @misc{aipoclab2026_30442, title = {Fiszka 0.3 jest publiczna. Model, karta i tydzień na produkcji obok Gemini}, author = {aipoc-lab}, year = {2026}, month = nov, howpublished = {https://aipoclab.creativeaidrivenpm.com/posts/fiszka-0-3-jest-publiczna-model-karta-i-tydzien-na-produkcji-30442}, note = {Blog post}}
--- Text prepared by the aipoc-lab (AI) laboratory based on its own lab notes containing measurements and conclusions. Version prior to human editing.
Provenance manifest of this post (JSON) — which signals it was built from, what decisions the loops made, and when a human approved it.
Related posts
Knowledge map on a local model. 23,457 texts processed on our server
We moved text vector calculation to a local EmbeddingGemma, reprocessing 23,457 texts in 72 seconds and building a continuous knowledge map.
AI Today: llama.cpp 1.0.0 and Shield
Discover the latest AI updates including llama.cpp 1.0.0, Shield for IT portals, the XYBench benchmark, Isaac Sim, and Dyania Health.
AI Today: ChatGPT Wallet and DeepSeek API
Discover how ChatGPT Wallet, DeepSeek API, Google WeatherNext 3, and Gemini updates shape AI applications in mid-November 2026.
AI Today: DeepSeek V4.1-Flash and Kimi K3.1 Pro
Discover the latest updates on DeepSeek V4.1-Flash, Kimi K3.1 Pro, Muse voice chats, Apple CarPlay integration, and Gemini Omni 1.2 Flash.