What mentoring defence AI students taught me about evaluation
Twelve months of reviewing thesis work on UAV imagery and radar analysis, and the failure mode that showed up almost every time.
Between 2024 and 2025 I was an industry mentor in the SAYZEK Academic Thesis Programme, run jointly by Türkiye’s Presidency of Defence Industries and the Council of Higher Education. The programme took 421 applications from 119 universities, accepted 356, and assigned 90 industry mentors drawn from 20 companies. Each student’s progress went through three interim reports that mentors reviewed and scored.
My students were working on AI-enabled imagery processing for UAV and armed-UAV platforms, and on radar-data analysis for AI-based ground localisation and target classification. Everything below is at a non-operational level. No restricted data, radar parameters or model thresholds appear here, and no student is identified.
The failure mode
Almost every project arrived with the same shape of problem. Not a modelling problem. An evaluation problem.
A student would show me a result on a test set and the result would be genuinely good. Then I would ask where the test set came from, and it would turn out to have been collected under conditions close enough to the training set that the split was measuring memorisation rather than generalisation. Same sensor, same altitude band, same time of day, sometimes the same flight.
This is not carelessness. It is what happens when data is scarce and the evaluation protocol is inherited from a tutorial rather than derived from the deployment condition.
The question that helped
The most useful thing I could offer was usually a single question. Under what condition would this model be wrong, and does your test set contain that condition?
For imagery work the answers came quickly once the question was asked. Low sun angle. Partial occlusion. A sensor the model had never seen. A target class that is rare in training precisely because it is rare in the world, which is often exactly the class that matters.
For radar work it was more subtle, because the intuitions people carry from computer vision transfer badly. Clutter is not noise. Multipath is not an outlier. A model that treats them as nuisance rather than structure will look excellent on a clean set and fail in the way that costs the most.
Why this connects to the rest of the work
I did not expect a mentoring commitment to feed directly into research design, but it did.
The habit I kept pressing on students is the habit that runs through QANTIS. State the operating envelope. Test at the boundary rather than the centre. Report the condition under which the result stops holding, and treat that report as part of the result rather than as an admission against interest.
It is easier to teach than to practise. Writing a paper that says here is where our method degrades is uncomfortable, because a reader can quote that sentence. But a result without a stated boundary is not a weaker claim than one with a boundary. It is an unfalsifiable one, which is worse.
The certificate for that programme is dated 10 July 2025 and signed by the President of Defence Industries. It records a mentoring contribution. It is not a government award or a competitive placement, and I describe it that way because that is what it is.