Sign in Subscribe

Why a Small Model Beat Two Frontier LLMs on a Narrow Task

Why a Small Model Beat Two Frontier LLMs on a Narrow Task

On 25 October 2026, the dermatologist Dr Magnus Lynch published a small test on his lab's website. He showed 200 biopsy-proven skin lesions to two frontier LLMs, GPT-6 Astra and Claude Fable 5.1. He also showed them to a fine-tuned convolutional network built on Inception v3, an architecture Google first published in 2015. The old, small network got the diagnosis right on its first choice 62% of the time, and each frontier model managed 42%.

This is Dr Lynch's test, not mine. I did not run it, reproduce it or audit it. It is one clinician's write-up on his own lab site rather than a peer-reviewed study, and it measured one narrow image-classification task, nothing wider. For anyone running their own box, the interesting part is not that two big models lost. It is how they lost, and what the small model was.

What this test says about local AI vs cloud AI

The easy reading is "small beats big", but that misses what happened. On a narrow task with fixed answers, the deciding factor is fit, and size matters much less.

Three things decided this test:

  • The task had six fixed answers. Each model had to name the most likely of six diagnoses (or say "unsure"), give a confidence and a second choice, and say whether the lesion could be safely discharged. A closed list like that is exactly what a classifier is built for.
  • The training data matched the images. The CNN was pretrained on everyday photographs, then fine-tuned on 68,011 dermoscopic and clinical images from the ISIC Archive. The test lesions came from MILK10k, a public dataset released through the International Skin Imaging Collaboration in 2025, which has a clinical close-up and a dermoscopic image of each lesion. The 200 test lesions, and every other photograph of the same lesion or patient, were removed from the training data, including the validation set. They were not used to pick the model or set its threshold.
  • The model was small enough to sit on the device. Fine-tuning took about twenty minutes on a single H100 GPU, and the result runs in a phone browser. More on that below.

The test set was built to be hard. Half the lesions should not be discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas including Bowen's disease, and 20 actinic keratoses. The other half were safe to discharge: 50 benign naevi and 50 seborrhoeic keratoses. Every lesion was biopsied, the benign ones included.

Both LLMs got both images and one prompt at default settings, and neither refused a case. On first choice, the CNN got 125 of 200 right, GPT-6 Astra got 84 and Claude Fable 5.1 got 85. Allowing a second choice, the CNN reached 173 and the two LLMs 134 and 138. Of the 30 melanomas, the CNN named melanoma first 13 times, against 8 and 11 for the LLMs.

The CNN did not win every category. It was one short of GPT-6 Astra on basal cell carcinoma and behind both LLMs on actinic keratosis. It pulled ahead on the harmless lesions, recognising 37 of 50 moles and 26 of 50 seborrhoeic keratoses. GPT-6 Astra managed 16 and 13, and Claude Fable 5.1 managed 33 and 6.

A big model with a favourite answer

This is the part worth remembering. GPT-6 Astra called 75 lesions basal cell carcinoma when only 25 were, including 14 of the 30 melanomas, and it called one melanoma basal cell carcinoma with 95% confidence. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. The CNN made mistakes too, but they went the other way: of the 17 melanomas it missed, it called 8 a naevus and 6 a seborrhoeic keratosis.

Dr Lynch measured the pattern. He did not test the cause. My own reading, in general terms and not a finding from his test, is that a general model asked to pick from a fixed list can settle on a label it leans towards, and a single prompt does nothing to pull it back.

Whatever the cause, the lesson for self-hosters is the same. On any fixed-output task, an API model can collapse onto one label. Its confidence score will not warn you, because the wrong answer arrives just as confident as the right one. The same problem can turn up in jobs many of us already hand to an API:

  • Support-ticket classification, where "billing" quietly swallows half of your bug reports.
  • Photo tagging, where one tag lands on far more images than it should.
  • Log triage, where everything odd gets labelled "network" and the real disk errors hide inside it.
  • Document sorting, where "invoice" becomes the default bucket for anything with a number on it.

There is one cheap check that catches this: compare the distribution of predicted labels with the real one, not just the accuracy. Take a few hundred samples you have labelled by hand, run them through the model, and count how often it predicts each label against how often each label really appears. If the model says "billing" 300 times and your real count is 100, you have found its favourite answer.

A spreadsheet is enough. If you would rather paste something into a script:

# true_labels and predicted_labels: two lists of the same length
from collections import Counter
real = Counter(true_labels)
pred = Counter(predicted_labels)
for label in sorted(set(real) | set(pred)):
    print(f'{label:20} real={real[label]:4}  predicted={pred[label]:4}')

An accuracy score alone would hide a habit like GPT-6 Astra's. Three times the real number of basal cell carcinoma calls jumps out the moment you count. I covered the wider method in how to prove a job can move to a cheaper model, and the label count belongs on that checklist.

The honest cost of being wrong

The safety numbers belong right next to the win. As a screening filter, the frontier models were more cautious: of the 100 lesions that should not be discharged, GPT-6 Astra kept 97 for review and Claude Fable 5.1 kept 96, against 77 for the CNN at its preset threshold. Of the 30 melanomas, the LLMs discharged 3 and 4, and the CNN at that threshold discharged 18. When its threshold was matched to theirs, the CNN kept 98 or 99 unsafe lesions and discharged 2 or 1 melanoma. So the CNN had the higher accuracy and the worse first-pass safety, while the LLMs erred towards sending people to a doctor.

Dr Lynch also asked the LLMs as a patient would, with the clinical photograph alone and the question "do I need to see a doctor?" Both advised seeing a doctor in almost every case, harmless lesions included.

For self-hosters, the lesson here is about the threshold. A classifier gives you a score, and where you draw the line on that score is a policy decision. With an API model, you mostly get whatever caution it came with. When you run the model yourself, the decision is yours: you can move the line towards caution or towards fewer false alarms, and you have to live with the one you pick. That is a feature and a responsibility in equal measure.

Running the small model yourself

Dr Lynch says you can try the CNN yourself at lynchlab.net/llm-skin-cancer, where his full write-up also lives. He says the model is small enough to run entirely in a phone browser using WebGPU, nothing is uploaded, and the random sample button draws from the same 200 lesions. For our purposes, that button is the interesting part: it shows a trained model working on the test images, on a phone, with no server involved.

In practice, that is usually what local AI means: a right-sized model on hardware you already own, rather than a giant model on a rented GPU. It matters for three plain reasons:

  • The model is a file you can keep. Nobody can retire it, swap it for a new version or raise the price.
  • The data never leaves the device. Nothing is sent, so there is nothing for a provider to log or keep. I made the case for why that beats a policy in a privacy promise is not a control.
  • There is no per-call bill. Classify ten items or ten million, and the cost is your electricity.

The limits are real too. A narrow model does one thing, and this one sorts six kinds of skin lesion and nothing else. It also has to be measured on your own data before you trust it. Dr Lynch is clear about the limits of his own test:

  • The numbers are small: 200 lesions, and 30 melanomas leaves a wide uncertainty on the figure that matters most.
  • Each LLM case was run once, and answers vary between runs.
  • These were hard, biopsied cases. On ordinary images from the archive, the CNN separated lesions far better, with an area under the ROC curve of 0.96 against 0.86 here.
  • MILK10k has been public since 2025, so the LLMs could have seen it in training, which if anything would have helped them.
  • Inception v3 is a baseline, not state of the art. State-of-the-art skin AI uses ensembles of image transformers trained on more data, and the regulated products built on them perform considerably better.

The practical rule follows from all of this. For any fixed-output job, spend an afternoon measuring a small local model against the API you are paying for, using your own samples, before you assume the big one wins. Label a few hundred real examples by hand and run both. Compare the accuracy and the label distribution, and set your threshold on purpose. If the small model needs a GPU, my GPU and VRAM buying guide covers what to look for, though a small classifier often runs on hardware you already have.

Pick the model that fits the job

None of this makes frontier models useless. In this test they were the more cautious screening filter. For open-ended work, such as writing, reasoning across messy documents or answering questions nobody planned for, they are still the right tool. A right-sized model is the right tool for a narrow job, and only your own measurement tells you which kind of job you have.

If you want help setting up that measurement, or getting a small model running on your own hardware, have a look at my services page. I also take work through freelance platforms.

Fiverr: hiteshsaini459 · Upwork: hiteshsaini25