AIAA SciTech 2027 · to appear

Air Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation

Prompt engineering, architecture, and evaluation for realistic controller transmissions

Ghazanfari, M., Casanova, M., Kam, J., Zongo, A., Wei, P., Darrell, T., and Bayen, A.

UC Berkeley · The George Washington University

Preprint Cite

Can a language model sound like an air traffic controller?

ATC radio is a safety-critical dialogue that is still almost entirely human. We test whether LLMs can generate realistic controller transmissions in a multi-turn setting, against a hand-transcribed general-aviation flight, and identify what helps and what hurts.

Multi-turn evaluation of LLM-generated ATC transmissions (schematic)The model plays the controller against a fixed pilot transcript. When it conditions on its own earlier replies, errors can accumulate and similarity to the ground truth drifts down; when the correct dialogue history is injected, similarity stays high. Radio exchange fixed pilot transcript · model plays ATC · illustrative wording PILOT …over the Golden Gate, request Bay Tour ATC …squawk 4521, maintain VFR PILOT squawk 4521, VFR ATC …radar contact, proceed as requested PILOT proceeding, Bay Tour ATC … ! ! Similarity to ground truth history: model's own replies history: correct transcript injected dialogue turn → Bars are illustrative, not measured values.
Schematic. The model plays the controller against a fixed pilot transcript. Conditioning on its own replies lets errors accumulate; injecting the correct history repairs the drift. Wording and bars are illustrative.
open- and closed-source LLMs
9
prompt structures of increasing constraint
5
hand-transcribed ground-truth flight
1

San Francisco Bay Tour general-aviation flight; a different flight supplies the optional worked example.

01 / Experimental design

A controlled, multi-turn test.

Ground truth

A general-aviation flight over the San Francisco Bay Tour route, transcribed by hand.

Prompts

Five prompt structures of increasing constraint, refined with pilot input.

Factors

With or without a worked example from a different flight; the model's own history vs. the correct history.

02 / Metrics

Lexical, structural, and semantic similarity, plus a judge.

Generated transmissions are compared with the ground truth on lexical, structural, and semantic similarity, and scored by an LLM-as-judge that is validated against human expert annotation.

03 / What matters

Examples help. Stricter prompts do not.

Worked example

Including an example from a different flight improves similarity to the ground truth.

Prompt constraint

The lightest prompts perform best; the most heavily scripted prompt degrades as its own errors accumulate.

Dialogue history

Injecting the correct history repairs that degradation, isolating error accumulation as the cause.

04 / Scope

Realism is a first step, not operational safety.

The study uses one route and flight as ground truth, and similarity to a human transcript is not the same as operational correctness. It complements work on LLMs as decision-makers in the loop (CVPR-W 2026).

Reference

Cite this work

Ghazanfari, M., Casanova, M., Kam, J., Zongo, A., Wei, P., Darrell, T., and Bayen, A. (2027). "Air Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation" Accepted for presentation at AIAA SciTech 2027.