aipoc-lab
Makieta wizualizacji sieci neuronowej Fiszki: wektor tekstu z 768 liczbami, dwie warstwy (128 i 64 neurony) i wyjścia tematów
/katTech/temEvaluation · LLM · Open Source/typAnalysis

Small model versus Gemini: first trial of an open AI topic classifier

Authorship and approval

[AI-generated content][Approved by a human]

Published and updated

Published: Updated:

Highlights

  • TF-IDF with logistic regression achieved a macro-F1 of 0.34, which is about 79% of the local teacher models' score.
  • Local teachers scored 0.436 compared to 0.439 for previous Gemini topics — practically the same result.
  • TF-IDF performed better than embedding-based models in 93% and 97% of draws, but there is no basis to call it superior; it is not worse.
  • Hidden retries showed that our manual labels often lacked internal consistency.
  • In version 0.2, we plan to refine topic definitions, prepare a new exam, and then release the model and materials.

Why we are building our own classifier

Every day we collect AI texts: arXiv papers, company blog posts, and news. We assign topics to each, such as agents, safety, or robotics. Until now, Gemini in the cloud did this. We want to bring this work in-house, use open models, and release the finished classifier publicly.

We also have a second reason: we want to better understand how models learn in practice. Today we can test this on our own hardware. Our 2011 server is sufficient for experiments that recently required a team and expensive equipment. We gave the model the working name Fiszka. Version 0.1 is a first attempt, not a finished product.

Gemini and Gemma are different models

Fiszka and our chat use embeddings from EmbeddingGemma 2 (embeddinggemma-2:270m). This is an open model under the Apache 2.0 license that runs locally on our server via Ollama, without an account or Google API. We convert each text into a vector composed of 768 numbers.

The knowledge map on the blog uses different embeddings: they come from the Google API and the Gemini Embedding 2 (gemini-embedding-2) model, with 1536 numbers each. This is a separate solution. In this experiment, Gemini serves as a benchmark because we are comparing Fiszka to the topics previously assigned by Gemini. Neither Gemini's responses nor its labels went into Fiszka's training or the input of its models. Fiszka is not trained on Gemini.

How we conducted the test

We prepared 14 topics. Each had a description indicating when a text belongs to it and when it does not. We determined training labels based on the intersection of answers from two models: Ornith 1.5 9B under the MIT license and Qwen 3.5 4B. They acted as teachers and ran via Ollama on a 2011 Intel i5-2400 processor without a graphics card. Labeling 2000 texts took them a total of about 46 hours. Qwen needed about 26 seconds per text, and Ornith about 42 seconds.

Next, we tested four small models. The first always assigned a fixed set of the most frequent topics and served as a baseline, representing zero level. The second used TF-IDF and logistic regression: TF-IDF counts words in the text. The third used logistic regression on EmbeddingGemma 2 vectors. The fourth was a small neural network with layers of 128 and 64 neurons, also working on these vectors.

For training, we used 2000 texts from before October 18, 2026. The exam covered 300 newer texts that were not in the training set. We labeled them manually on a special page; our manual labels served as the ground truth, not the model's response. In 37 cases we chose "do not understand", and in one case "cannot be evaluated". We therefore evaluated 262 texts: 205 from arXiv and 57 from other sources. Rare topics appeared more frequently in the exam, so we weighted the results to match the regular stream of texts.

We report results as weighted macro-F1. This metric calculates F1 separately for each topic and then averages the scores so that each of the 14 topics counts equally. A score of 1 means ideal, and 0 means all errors.

System

macro-F1

Time per text

Fixed set of most frequent topics

0.05

–

TF-IDF + logistic regression

0.34

0.5 ms

EmbeddingGemma 2 + logistic regression

0.30

approx. 1.7 s per vector + 0.02 ms

EmbeddingGemma 2 + small neural network

0.29

approx. 1.7 s per vector + below 0.01 ms

Ornith + Qwen (teachers)

0.44

approx. 70 s

Gemini (previous topics)

0.44

not measured

A separate script checked the fairness of the comparison. It confirmed that the exam does not contain training texts, its labels come from a human, and the training data contains no Gemini labels or responses. It also checked that the model beats the baseline and that the numbers in the report match the data.

What the results say — and what they don't say

Local teachers scored 0.436, and previous Gemini topics scored 0.439. This is practically the same. The best small model, TF-IDF, achieved about 79% of the teachers' score. In 1000 bootstrap draws, the teachers outperformed TF-IDF in 99% of cases, so this advantage is certain.

TF-IDF performed better than embedding-based models in 93% and 97% of draws. This is not enough to consider this advantage certain: the confidence interval touches zero. The conclusion is therefore that TF-IDF is "not worse" than embedding models, rather than better or superior.

The processing time of embedding models resulted mainly from converting text into vectors. EmbeddingGemma 2 needed about 1.7 seconds for this on the CPU. The classification model itself computed the score afterwards in a fraction of a millisecond. Local teachers needed about 70 seconds per text, so the quality score does not mean they operate as fast as small models.

Manual labels also require verification

We showed 30 exam texts for labeling a second time, without warning. Out of 21 pairs that could be compared, an identical set of topics appeared only 7 times. We measured label agreement separately for topics using Cohen's kappa: a value of 1 means always the same choice, and 0 means agreement like a coin toss.

Topics

Kappa

multimodality, medicine, hardware, RAG

1.0

tools, education

0.64–0.69

safety

0.45

language models, evaluation

0.22

agents

0.13

open source

0.0

Models made mistakes in similar places. The best small model assigned the "evaluation" topic in 191 texts where the human did not choose it, although it also found all 36 correct ones. Teachers likely considered any paper with experiments to be evaluation. The model assigned the "tools" topic only 3 times, although the human chose it 61 times, and it assigned the "agents" topic in 93 too many texts.

We also noticed gaps in the topic set. Texts about general machine learning ended up in "language models", and materials on AI policy and regulations, such as the European Commission or cooperation between labs and states, lacked a proper place. The low score, including Gemini's 0.44, stems primarily from blurred definitions of several topics rather than weaknesses in the models themselves.

What we learned and what's next

Topic definition matters more than model choice. Before we evaluate a machine, we must measure the consistency of our own labels. Without hidden retries, we could have blamed the models for the low score, even though part of the difficulty stemmed from the fact that we ourselves did not always assign the same topics to texts.

The experiment showed that open models on an old CPU are sufficient for assigning topics at the level of Gemini. A simple model is a good starting point: it works quickly and is easy to verify. In this trial, it did not perform worse than a neural network, although the results do not give grounds to state that embedding models are worse.

In Fiszka 0.2, we plan to add machine learning, policy, and regulation topics — including lab-state cooperation — as well as business and initiatives. Research papers will be a separate text feature rather than a topic. We also want to tighten definitions for language models, evaluation, agents, open source, and tools topics.

We will prepare a new batch of texts for the exam. We will not use the old exam to improve definitions, because the score of such a test would be biased. After version 0.2, we plan to release the model, code, report, and decision log on GitHub and Hugging Face.

There is no team behind the lab: we prepare experiments jointly as a single person and AI. This is a hobby project, done for science and fun.

Frequently asked questions

Why build your own topic classifier?

Every day we collect AI texts and assign topics to them, and until now Gemini in the cloud did this. Our own classifier aims to shift this work to open models and into our lab. We also build it to learn how model training works and to share the finished solution publicly.

Is Fiszka trained on Gemini?

No. Training labels came from the intersection of responses from Ornith 1.5 9B and Qwen 3.5 4B. Neither Gemini's responses nor its labels went into Fiszka's training or its models' input. Gemini serves exclusively as a benchmark.

Can a small model already replace Gemini?

In version 0.1, no. The best small model, TF-IDF with logistic regression, achieved a macro-F1 of 0.34 compared to 0.44 for Gemini. Local teachers achieved a score practically identical to Gemini, but needed about 70 seconds per text.

How much does it cost?

Nothing besides electricity. The experiment runs on our server with local models executed via Ollama, without a paid API.

Will Fiszka be public?

Yes. After preparing version 0.2, we plan to release the model, code, report, and decision log on GitHub and Hugging Face.

Sources

--- Text prepared by the aipoc-lab (AI) laboratory based on the lab's own notes with measurements and conclusions. Pre-human-editing version.

Provenance manifest of this post (JSON) — which signals it was built from, what decisions the loops made, and when a human approved it.

Related posts

/katEcosystem/temAgents · LLM · Open Source/podMicrosoft · NVIDIA/typNews

AI Today: Nemotron and PDD Kit

Discover how NVIDIA promotes open Nemotron models and Microsoft Research uses PDD Kit for verifying agent-written code.

/katTech/temHardware · LLM · Open Source/podBig Tech (other) · NVIDIA/typNews

AI Today: NVIDIA 6G and AMD InferaSim

Discover NVIDIA's 6G open tools growth, AMD's InferaSim LLM serving predictor, and the European Commission's TikTok DSA findings.