Electronic nose models get evaluated in two regimes: the way that flatters them, and the way they will actually be used. The gap between the two is where most of this field's inflated numbers come from. This essay is about closing that gap — the specific design choices that make an evaluation recording-fair, the two questions every model must answer separately, and the measured numbers that show what each question actually produces.
The Leak That Inflates Everything
The most common error is subtle, structural, and quiet. A recording is a continuous trace. Models are trained on windows — short slices of that trace, each carrying labels. If you split windows from the same recording randomly into train and test, you leak: adjacent windows from the same physical capture, the same sensor state, the same drift trajectory, appear on both sides of the boundary.
The model then looks brilliant because it memorized a specific afternoon, not because it learned a substance. The tell is the accuracy gap: the leaked model scores 95% on windows it has effectively seen and collapses when you hand it a recording from a different session.
The fix is a grouping rule that sounds like common sense and is violated constantly: never split windows from the same physical recording across the train/test boundary. Group by recording (or by cut, induction, or substance — the natural unit of your dataset), and do grouped cross-validation: GroupKFold, LeaveOneGroupOut, or leave-one-out over the natural grouping. Every headline number in the OpenSmell evaluation suite was produced under this rule, with the null baselines in the same table.
The Two Questions
An e-nose model can be asked two very different things, and an evaluation must answer them separately:
- Can it recognize a substance it was trained on, in a new recording from the same rig? This is session invariance. Measured result: 81.78% accuracy / 80.33% macro-F1 on held-out sessions, against a pre-registered >70% threshold (random baseline 2%). This is a strong, real result.
- Can it recognize a substance it has never seen? This is substance generalization. Measured result: R² = −14.62 under leave-substance-out cross-validation (one fold as bad as −55.71). The model is worse than predicting the mean.
These are different claims with different engineering meaning. Session invariance says: train me on your substances, and I will recognize them next week. Substance generalization says: train me on some substances, and I will recognize ones I never met. The first is achievable and is what most deployed e-noses actually need. The second is a research frontier, and reporting the first as evidence of the second is the category error this field keeps making.
Pre-Registered Thresholds and Fixed Seeds
The other pillar of evaluation is deciding before fitting. The session-invariance work declared a >70% threshold in advance, then ran the experiment, then published the result — so the threshold cannot be moved after the fact to make a near-miss look like a win. The discipline extends to:
- Fixed seeds (the project uses 42) so every run is reproducible bit-for-bit.
- A fixed CV structure, stated before the run, so the grouping choice cannot be tuned to flatter the result.
- Balanced accuracy in addition to raw accuracy, because a 60/40 class split lets a majority-only model claim 60% "accuracy" while doing nothing.
This is the same pre-registration culture that protects clinical trials, and it protects e-nose claims just as well.
What the Adapter Results Actually Say
The perennial hope is that a small "adapter" layer lets a model trained on device A transfer to device B with a few samples. The project tried the main families, and the measured results are mixed enough to be worth printing in full:
| Adapter | Result | Verdict |
|---|---|---|
| MSE adapter | 0.95 cosine on held-out lemon | confirmed (simulation) |
| Cosine-loss adapter | 0.81 | failed |
| Parameterised adapter | 0.879 on held-out config | confirmed |
| Conv1 fine-tune | garlic–ginger cosine 0.409 | partial |
The pattern: adapters help in narrow, well-specified settings (the MSE result is a simulation; the parameterised result is on one held-out configuration), and they stop helping when the gap is structural. The reading the whole experiment arc converged on: a small set of labeled reference samples on the target device, used for reference-point calibration, beats every reference-free adapter family tried. Adapters are a complement to calibration, not a replacement for it.
The Recording-Fair Baseline Table
The strongest habit to borrow from the evaluation suite is its baseline table discipline: every headline number ships next to chance accuracy and the majority-class predictor in the same table, so nobody has to trust a single number in isolation. A sample of the pattern, from six evaluations:
| Task | Result | Balanced / chance | Majority |
|---|---|---|---|
| Gas present/absent | 94.1% | 94.4% / 50% | 65.5% |
| TVC regression | R² 0.793 | ρ 0.849 | mean-pred MAE 0.935 |
| Fine substance identity | 89.4% | 89.5% / 2.0% | 2.3% |
| Perceptual family (LSO) | 40.2% | — / 12.5% | 38.1% |
Read the last row carefully: 40.2% looks substantial until you see the majority baseline is 38.1%. The family result is close to a null result. Without the baseline column the row would be misread; the baseline is why it reads this way.
A Leak Caught and Fixed
The value of the discipline is best shown by the leak that got caught. The taxonomy experiment (U6) initially grouped its cross-validation by recording ID — 250 groups — while claiming leave-one-substance-out. Same-substance recordings would have leaked across the boundary. The run was killed, both evaluation loops were re-grouped by substance (50 groups), and the collapse metric was recomputed from aligned predictions. All published numbers are the post-fix run. The lesson is not "the team was sloppy"; it is that grouping bugs are invisible until you audit the group key itself, and an evaluation pipeline that makes the group key an explicit, versioned argument is what makes the audit possible.
The Checklist
- Group by the natural unit (recording / cut / induction / substance); never by window.
- Report chance and majority baselines in the same table as the headline.
- Pre-register thresholds and seeds; keep the CV structure fixed.
- State which question you answered: session invariance or substance generalization.
- Quote balanced accuracy, not just accuracy.
- Audit the group key — the leak is usually there, hiding.
Sources & Further Reading
- OpenSmell master reference, §7.11 (honesty rules), §8.3 (the session-invariance and leave-substance-out results), §11.7 (ML evaluation hygiene), §12 (the recording-fair evaluation suite and its leak catch).
opensmell/e-nose-evals— the reproducible evaluation suite (harness/evaluate.py) and per-experimentresults/*_metrics.json.- The interoperability essay — why evaluation design and calibration limits are the same story.