A Small Model Matched Gemini in Topic Tagging. Fiszka 0.3
Authorship and approval
Published and updated
In Fiszka 0.3, we tested the impact of a larger dataset, refined topic definitions, and class weights. On 75 non-arXiv texts, we achieved an F1 of 0.496, and on topics shared with Gemini, 0.417 compared to 0.409. This is a tie, not an advantage. Fiszka's higher overall score comes mainly from topics that Gemini does not have at all.
Key Takeaways
- On an exam of 2522 arXiv papers, Fiszka 0.2.1 with class weights achieved a mean sensitivity of 0.776 across seven topics.
- On 75 evaluated non-arXiv texts, Fiszka 0.3 achieved an F1 of 0.496, and on topics shared with Gemini, the score was 0.417 versus 0.409.
- The result was improved primarily by more data, with a contribution of +0.195, while class weights added +0.055.
- The entire study, including teacher candidate exams and definition trials, cost a total of $0.82, and Fiszka itself runs with no API fees.
From Fiszka 0.1 to 0.3
This is a continuation of our post "Small Model vs. Gemini", in which we described Fiszka 0.1 from November 4. Back then, we achieved a macro-F1 of 0.34 compared to Gemini's 0.44. We also saw that imprecise topic definitions can matter more than model choice.
Fiszka is our small classifier that assigns topics to texts, such as agents, safety, or robotics. The lab collects papers from arXiv, company blog posts, and news daily. Previously, Gemini in the cloud assigned topics to them. Fiszka runs locally, without an API, and is meant to be public.
What We Changed
In the 0.2 definitions, we increased the number of topics from 14 to 17. We added machine learning (ml), policy and regulation (policy), and business (business). We also clarified descriptions for language models, evaluation, agents, open source, and tools. Each topic has guidelines on when to assign it and when not to. The main rule: what the text is about matters, not what tool was used.
Separately, we improved evaluation, which is the topic related to assessing models. Teachers assigned it to 39% of texts, while a human labeled 17% of those same arXiv papers with it. The revised definition alone lowered the score to 34%. Only a separate yes/no question—does the text concern a tool for evaluating other models—brought it down to 18%. A second teacher answered "yes" to this question in 98% of cases, so ultimately only one model asks it.
We also changed the teachers. Instead of two models running on our server, we used DeepSeek V4.1 Flash under the MIT license via OpenRouter and Nemotron 3.5 Lightning from NVIDIA under the OpenMDW 1.1 license. A topic was assigned to the label if both models indicated it. On the Fiszka 0.1 exam, this pair achieved 0.452, and the previous pair 0.436. The difference was within the noise floor, so we chose the new solution for the sake of time: about 0.7–1.2 seconds per text instead of about 70 seconds.
We also increased the training set. Fiszka 0.2 used 2000 texts, version 0.2.1 used 8342 texts from before October 18, 2026, and 0.3 used 9968. The last 1626 texts came from outside arXiv: from company blogs and news from August and September. We added class weights, meaning we increased the influence of rare topics such as robotics, hardware, and safety during training.
We kept the model that performed best in Fiszka 0.1: TF-IDF with logistic regression. Simplified, the model counts words in the text and assigns topics based on that. It runs in about 0.5 ms per text on a standard CPU. In this round, we skipped vector-based models because the vectors for new texts were still being computed.
How We Ran the Exam
The first part covered 2522 arXiv papers published from October 18 to 31, 2026, which are newer than all training texts. We used categories assigned by the paper authors, such as cs.RO for robotics, as our benchmark. arXiv metadata is under the CC0 license. This part measures sensitivity: the proportion of papers with a given category that the model recognized. We also checked the average number of topics assigned to a text to evaluate whether the model was boosting its score by assigning many topics.
The second part consists of 76 non-arXiv texts from the same period, from company blogs and news originating from 12 sources. We labeled them manually and then re-reviewed some of the labels. One text was deemed impossible to evaluate, so we calculate results for 75. Here we measured precision, sensitivity, and F1—a metric combining precision and sensitivity.
Results on arXiv
Version | Mean sensitivity (7 topics) | Topics per text | Robotics | Hardware | Safety |
|---|---|---|---|---|---|
Fiszka 0.2 (2000 texts) | 0.526 | 1.93 | 0.30 | 0.05 | 0.56 |
Fiszka 0.2.1 without class weights (8342) | 0.721 | 1.82 | 0.67 | 0.43 | 0.87 |
Fiszka 0.2.1 (8342, class weights) | 0.776 | 1.97 | 0.83 | 0.62 | 0.90 |
Fiszka 0.3 (9968, class weights) | 0.756 | 1.84 | 0.83 | 0.57 | 0.89 |
The transition from 0.526 to 0.721 after increasing the dataset yielded +0.195. Class weights added another +0.055, raising the score to 0.776. Rare topics benefited the most: sensitivity for hardware rose from 0.43 to 0.62, and for robotics from 0.67 to 0.83.
On the six topics shared with version 0.1, the score first dropped from 0.537 to 0.497 in version 0.2, and then rose to 0.780 in 0.2.1. Fiszka 0.3 maintained the level of 0.2.1. The difference is related to a higher threshold for assigning topics and fewer topics per text. Hardware is harder to compare: only 21 papers with this category were in the exam, so a single paper changes the score by about 0.05.
Results Outside arXiv
System | Precision | Sensitivity | F1 | F1 only on topics that Gemini also has |
|---|---|---|---|---|
Gemini (previous tags) | 0.295 | 0.407 | 0.342 | 0.409 |
Fiszka 0.2.1 | 0.414 | 0.575 | 0.481 | 0.368 |
Fiszka 0.3 | 0.441 | 0.566 | 0.496 | 0.417 |
Fiszka 0.2.1 without class weights | 0.337 | 0.301 | 0.318 | – |
On topics shared with Gemini, we have a tie: F1 is 0.417 for Fiszka 0.3 and 0.409 for Gemini. Fiszka's higher score calculated across all topics—0.496 versus 0.342—stems mainly from topics that Gemini does not have at all, namely business and policy. Business applied to 34 out of 75 texts. We do not describe this as an advantage on the shared set of topics.
The additional 1626 non-arXiv texts helped in the areas they came from: F1 on topics shared with Gemini rose from 0.368 for Fiszka 0.2.1 to 0.417 for 0.3. The variant without class weights performed poorly, with an F1 of 0.318, and assigned no topic to 17 texts.
We Also Checked Manual Labels
During manual labeling, the "language models" topic ended up on 29 out of 75 texts. Often the reason was simply the appearance of ChatGPT or Gemini, even in a text about something else, such as "4 tips for styling outfits with Gemini". Meanwhile, the definition says not to assign this topic if the language model is merely a tool for another task.
We re-reviewed those 29 texts according to the definition. The topic remained on 8 and was dropped on 21. This is an important caveat to the results: re-evaluation according to the same definition used by teachers brings manual labels closer to training labels. After the review, Fiszka still performed poorly on the language models topic: F1 was 0.18.
We found no glaring errors, such as assigning the European Commission to medicine. The closest to such a mistake was a text about dementia marked as business instead of medicine and a text about XR glasses marked as tools instead of hardware. Remaining discrepancies concerned adjacent topics.
Gemini Only for Comparison
Fiszka does not learn from Gemini. Training labels were prepared by DeepSeek V4.1 Flash and Nemotron 3.5 Lightning. We use Gemini's previous tags in our database solely as a ready-made comparison point.
How Much Did It Cost
Labeling 8342 texts by teachers in the cloud took about two hours, and an additional 1626 took about 25 minutes, running three jobs in parallel. By comparison, two models on our server needed about 46 hours for 2000 texts. The quality of the new pair was similar to the previous one.
The entire API key used in the study consumed a total of $0.82 according to the OpenRouter dashboard as of November 6. The amount includes teacher candidate exams, definition trials, labeling all 9968 texts, and evaluation queries. This is not a cost calculated per text.
Fiszka itself costs nothing besides electricity. Training takes a moment on a standard CPU, and assigning topics takes about 0.5 ms per text.
What We Learned and What's Next
In this experiment, more data yielded more than the class weight correction: +0.195 and +0.055, respectively. We also saw that a poorly defined topic can crowd out others. With a limit of three topics, evaluation appeared in 39% of texts and took up space from rarer categories.
We also learned that manual labels do not always stick to definitions. We assigned "language models" out of habit to texts mentioning ChatGPT or Gemini. Without re-review, we might have mistaken this for a model weakness. Comparison with an existing system must be conducted on shared topics, otherwise the score might look better than it is.
Next, we will test vector-based models, including the local EmbeddingGemma 2, and compare them with TF-IDF—especially on non-arXiv texts and for the language models topic. We also plan to publish the model, code, report, and decision log on Hugging Face and GitHub.
Closing Thoughts
There is no team behind the lab: experiments are prepared by a single person together with AI, hence the "we" in the text. The project is a hobby, done for science and fun.
Frequently Asked Questions
Is Fiszka already better than Gemini?
On shared topics, the results are close: Fiszka 0.3 achieved an F1 of 0.417, and Gemini 0.409. Fiszka's higher score across all topics, 0.496 versus 0.342, is mainly due to business and policy, which Gemini does not have.
Does Fiszka learn from Gemini?
No. Labels used for training were prepared by DeepSeek V4.1 Flash and Nemotron 3.5 Lightning. Gemini tags from our database serve solely as a ready-made comparison point.
Why teachers in the cloud instead of your own server?
Two models on our server needed about 46 hours for 2000 texts. Teachers used via OpenRouter labeled a pool of 8342 texts in about two hours, and another 1626 in about 25 minutes. The quality of the new pair was similar to the previous one.
What had the biggest impact?
Increasing the dataset had the largest contribution: +0.195 in mean sensitivity on the arXiv exam. Class weights added +0.055 and particularly helped rare topics like robotics and hardware.
How much does running Fiszka cost?
Fiszka itself has no costs beyond electricity: training takes a moment on a standard CPU, and assigning topics takes about 0.5 ms per text. The one-time data labeling by teachers was paid; the entire API key in this study consumed a total of $0.82.
Sources
- DeepSeek V4.1 Flash in OpenRouter
- Nemotron 3.5 Lightning in OpenRouter
- arXiv API, terms of use
- Logistic regression and class weights in scikit-learn
How to Cite
aipoc-lab. "A Small Model Matched Gemini in Topic Tagging. Fiszka 0.3". November 6, 2026. https://aipoclab.creativeaidrivenpm.com/posts/maly-model-dorownal-gemini-w-tagowaniu-tematow-fiszka-0-3-28751
BibTeX: @misc{aipoclab2026_28751, title = {Mały model dorównał Gemini w tagowaniu tematów. Fiszka 0.3}, author = {aipoc-lab}, year = {2026}, month = nov, howpublished = {https://aipoclab.creativeaidrivenpm.com/posts/maly-model-dorownal-gemini-w-tagowaniu-tematow-fiszka-0-3-28751}, note = {Blog post}}
--- Text prepared by the aipoc-lab (AI) laboratory based on an internal lab note with measurements and conclusions. Pre-human-edited version.
Provenance manifest of this post (JSON) — which signals it was built from, what decisions the loops made, and when a human approved it.
Related posts
Small model versus Gemini: first trial of an open AI topic classifier
We built a first version of our AI text topic classifier, tested it against manual labels, and compared small models to Gemini.
AI Today: llama.cpp 1.0.0 and Shield
Discover the latest AI updates including llama.cpp 1.0.0, Shield for IT portals, the XYBench benchmark, Isaac Sim, and Dyania Health.
Fiszka 0.3 is public. The model, model card, and a week in production alongside Gemini
Fiszka 0.3 weights, model card, and code are now public, showing a 75% topic agreement with Gemini over a one-week production test.
AI Today: ChatGPT Wallet and DeepSeek API
Discover how ChatGPT Wallet, DeepSeek API, Google WeatherNext 3, and Gemini updates shape AI applications in mid-November 2026.