Farm Vision — when validation lies
How a model with brilliant validation metrics failed on real photos, and why the only honest judge turned out to be a hand-built golden set.
Farm Vision is a mobile detector of people and animals that runs entirely on an iPhone, with no cloud and no internet. Nine classes in real time: person, cat, dog, horse, cow, sheep, goat, pig, deer. The applied goal is simple — to know who walks the farmyard at night.
But the real story of the project is not about the app. It is about one moment when the model showed excellent metrics and yet did not work — and about how I found that out.
A demo that came together quickly
The technical start was easy. Flutter, the official ultralytics_yolo plugin,
the yolo26n model — the smallest of the family. The plugin handles the camera,
inference and box drawing, so the riskiest step — decoding YOLO’s raw output
tensors by hand — went away on its own. Within a few days there was a working
app: point the camera, boxes with labels appear. Local, offline, real time.
The stock model knows the 80 COCO classes, but a goat is not among them. To tell a goat from a dog — which was the original goal, “no more 300 dogs on the field” — the model had to be fine-tuned on my own nine classes. This is where the real work began.
A model that sat an exam on its own notes
The first version trained in Google Colab and showed brilliant validation:
mAP of 0.872 for goat, 0.982 for cow. By the metrics — a finished product.
I did not trust the numbers and built a separate test: a golden set — 120 photographs, 15 per class, hand-picked from Google in varied styles: pastures, close-ups, different backgrounds, black-and-white. The key condition — these photos never entered training.
The result: 45 out of 120. Goat, cow and sheep — 0 of 15. A model that “knew” a cow at 98% on validation did not recognise it once on real photos.
The cause is called domain shift. The validation split is drawn from the same datasets as training. If every goat in the dataset is a farm shot of one style, the model shines on the same kind of farm validation photos and collapses on studio or “internet” ones. It is an exam sat on your own notes.
The rule I took away from this for good: validation mAP is a direction hint, not a quality score. Decisions are made only on the golden set.
The model learned photo style, not the animal
Digging deeper, I noticed an almost perfect pattern: a class’s success on the golden set matched whether it had a multi-style data source. Classes with a separate “internet” dataset (horse, deer, cat) scored 15 of 15. Classes drawn from a single farm dataset (goat, cow) scored 0 of 15.
This is shortcut learning. Instead of the real feature — “what a goat looks like” — the model found an easier crib: “farm-style photo → probably goat; Google-style photo → probably horse”. When I pointed the camera at a screen the style became “internet”, and every animal became a horse.
The cure was three blows to the crib:
- Multiple domains. Each class must have data from at least 2–3 different sources and styles. Cat was cured (0.67 → 15/15) by simply adding an OIDv4 dataset. Goat and cow were each saved by four different sources.
- Augmentation during training (
mosaic,mixup, colour shifts, rotations): every photo is randomly distorted each epoch, so the model cannot memorise the style of a particular dataset. - A new data checklist item: style diversity within a class, not 3000 near-identical frames from one yard.
Three iterations and the see-saw between classes
Progress on the golden set was monotonic:
| Version | Result | What we did |
|---|---|---|
| v1 | 45/120 | base dataset — domain shift discovered |
| v2 | 74/120 | added “internet” sources; cat 3→15 |
| v3 | 90/120 | multi-style sources for cow and goat; Open Images via FiftyOne |
The main achievement: goat 0 → 14/15, cow 0 → 15/15. The original goal was met — Farm Vision tells a goat from a dog.
But strengthening some classes has a price. In v3 a see-saw between similar classes appeared: I poured in four goat sources and seven thousand cow photos — goat and cow rose, but sheep dropped (10 → 3, confused with goat) and so did pig (7 → 5, confused with cow). Strengthening a class presses on its similar neighbour: goat↔sheep, cow↔pig. Perfect balance does not exist; each iteration shifts the equilibrium.
I deliberately stopped at v3 and documented the boundary rather than hiding it. The fix plan (raise sheep and pig with multi-domain data without aggressively cutting goat and cow) is written down — but doing it now would mean another cycle of Colab pain to polish a goal already met.
A trap that is easy to fall into during testing
One separate lesson is worth mentioning. When v3 “poorly” recognised animals on a laptop screen, that was not a model failure but an invalid test. Re-shooting a screen adds moiré, glare, and shifts the image style to “over-digital”. The model trained on live photos, not on photographs of photographs.
An honest test is a live animal or a flat printout, not a camera aimed at a monitor. The symptom arises on the path of measurement, not in the object being measured.
Takeaways
Technically I went through a full ML cycle that takes real teams months: data collection → domain shift → shortcut-learning diagnosis → honest testing → Open Images via FiftyOne → three balancing iterations.
But the main conclusion is not technical. The hardest part of machine learning is not training the model but knowing whether to believe its metrics. Validation lies when the training data and the deployment data are of different styles. The only defence against that lie is a set built by hand precisely so as not to fall for it.