arXiv 2609.01404 / NAVER Cloud
DroneCATS: Commanding, Approaching, Tracking and Searching
Background: four simulated episodes seen from above. Blue is the drone, orange the target and its 5 m arrival radius.
Overview
Multimodal LLMs are being asked to make decisions for drones, but most tests stay in simulation, use a single drone, and give it a target that is in view and standing still. In DroneCATS the model is the only decision maker. Targets can start out of view or move, several drones can share one instruction, and the same agent flies real drones.
Multimodal large language models (MLLMs) are increasingly used to make decisions for drones, but existing evaluations are mostly limited to simulation, to a single drone, and to targets that are visible from the start and do not move. We present DroneCATS, a benchmark in which the MLLM alone controls the drone through four actions, and evaluate ten MLLMs from frontier models to small open models. The benchmark varies target visibility, target motion, scene and simulator, and extends to real drones. Beyond a single drone, it adds a multi-drone suite, where one model drives multiple drones, and a commander + sub-agents suite, where a commander model directs one sub-agent for each of the drones.
Across these settings, we find that frontier models achieve the highest success rates, but their latency causes them to lose moving targets, while small open models often do not declare arrival even when they reach the target. Guided by this failure analysis, we fine-tune a 2B model on drone action data synthesized from public RGB-D images; its success rate rises from 0% to 31% in simulation and outperforms the frontier models in real flight. With multiple drones, we find that every zero-shot model succeeds more often as the commander of one sub-agent per drone than when it drives all of them from one context, and that commanding success follows general VLM ability. On real drones, we observe that the results broadly follow those in simulation, while latency and hardware limits matter more.
The agent
At every step the model gets the instruction, the current camera frame and its last five actions, and answers with one of four actions in JSON.
A rule-based controller turns the action into a flight command, so nothing in the loop is tuned to a particular model.
An episode counts only if the model says finished while it is near the target and can see it.
Swipe to see the whole loop →
One step of the loop. The MLLM gets the current frame and a prompt with the instruction and its last five actions, and replies with one action in JSON. A go reply carries a point (x, y) in the frame, drawn in red, and a distance d in metres. The controller and the flight stack turn the action into motion and are the same for every model.
fly d metres toward the point (x, y)
turn by θ to look for what is not in frame
next reply is a plan schema
declare arrival; the verifier scores it
Task axis
Crossing these two questions gives four single-drone tasks: approach, search, track, and search + track. In every task the drone must reach the target and declare arrival. Episodes are flown in AirSim and Isaac Sim, and on a DJI Tello.
Two questions set the task. Is the target in view at the start, and does it move? A target out of view must first be found; a moving target must be followed.
Organisational axis
DroneCATS goes beyond a single drone. Four drones share one instruction that none of them can finish alone. Either one MLLM drives all four from one prompt, or a commander splits the instruction into one command per drone for four single-drone sub-agents. Both structures fly the same multi-drone tasks, at three levels of difficulty.
Multi-drone tasks. Both structures fly the same tasks, at three levels for the commander. Easy, license plate matching: one of four cars carries the named plate, and each drone reads one car up close. Medium, sign matching: four corridors each end in a sign, and the commander has to divide the corridors. Hard, three-sign sum: the drones read three red signs, add them, and fly into the corridor whose sign shows the sum.
Finding 1
We ran ten MLLMs, from frontier API models down to 2B open models. Gemini 3.7 Flash has the highest mean success, yet the best model changes with the task. Every frontier model recognises the target, so on moving targets the outcome comes down to how fast the next action arrives, and the quicker GPT-5 and Gemini Robotics-ER 2 come out ahead.
| Success rate (%) | Approach | Search | Track | Search + Track | Mean | Real drone |
|---|---|---|---|---|---|---|
| Gemini 3.7 Flash | 87.5 | 67.5 | 55 | 47.5 | 64.4 | 45 |
| Claude Opus 5 | 57.5 | 47.5 | 40 | 40 | 46.2 | 30 |
| Gemini Robotics-ER 2 | 62.5 | 47.5 | 70 | 50 | 57.5 | – |
| GPT-5 | 55 | 37.5 | 62.5 | 65 | 55 | – |
| Qwen3.5-27B | 60 | 45 | 25 | 27.5 | 39.4 | – |
| Qwen3.5-9B | 32.5 | 15 | 32.5 | 5 | 21.2 | 10 |
| Qwen3.5-4B | 12.5 | 5 | 7.5 | 0 | 6.2 | – |
| Qwen3.5-2B | 0 | 0 | 0 | 0 | 0 | – |
| Cosmos3-Edge-2B | 0 | 0 | 0 | 0 | 0 | 0 |
| Cosmos3-Edge-2B (FT, ours) | 35 | 35 | 27.5 | 27.5 | 31.2 | 55 |
Single drone. Simulation: 20 episodes per task, each flown twice. Real drone: mean over the four tasks, 5 episodes each, so one episode moves a real cell by 20 points.
Finding 2
Open models of 9B and below turn less than half of the episodes in which they reach the target into a success. The smallest ones keep going, pass the target and end far from it. How well a model keeps to its system prompt over a long conversation predicts this much better than how often it reaches the target.
finished 2 m from the cyclistThe same Approach episode. The readout turns green inside the 5 m arrival radius; only a finished declared there counts.
Finding 3
Cosmos3-Edge-2B is the fastest model we tested, but it never declared arrival. We fine-tuned it on drone actions synthesised from public RGB-D images, with no benchmark frame in the data. Its success rate went from 0% to 31% in simulation, and on a real drone it beat both frontier models we flew.
Finding 4
Every zero-shot model does better as a commander, writing one order per drone for single-drone sub-agents, than when it returns all four actions itself. The replay below follows a commander through Three-sign sum, the hardest level.
Gemini 3.7 Flash commanding four Gemini Robotics-ER 2 sub-agents, replayed from the run's logs. Map: drone paths, red signs, entrance signs, and the goal with its 5 m radius; × is a rejected declaration, the star the accepted one.
Flown both ways, the commander has Drone 4 at the goal while the four drones of the single model are still near the start.
Gemini 3.7 Flash on the same Three-sign sum episode, cut at the 300 s cap.
The gain holds from the strongest model to the weakest. Between commanders the gap widens with difficulty, and on Three-sign sum only Gemini 3.7 Flash and Claude Opus 5 succeed in most episodes.
Success rate (%) over 30 episodes per model, driving all four drones itself vs. commanding four sub-agents.
Finding 5
We flew the single-drone and commander suites on DJI Tellos. The ranking of models broadly carries over. The video and command links slow every step, though, so the frontier models that tracked well in simulation lose moving targets in the real room. In simulation a step costs little more than the model call; on the Tello the link delay comes on top of it. This is why the fine-tuned 2B, at 0.6 s per call, is the one model that does better on the real drone than in simulation.
| Real drone, SR (%) | Approach | Search | Track | Search + Track | Mean | Sim mean |
|---|---|---|---|---|---|---|
| Gemini 3.7 Flash | 100 | 80 | 0 | 0 | 45 | 64.4 |
| Claude Opus 5 | 40 | 40 | 20 | 20 | 30 | 46.2 |
| Qwen3.5-9B | 40 | 0 | 0 | 0 | 10 | 21.2 |
| Cosmos3-Edge-2B | 0 | 0 | 0 | 0 | 0 | 0 |
| Cosmos3-Edge-2B (FT, ours) | 60 | 60 | 60 | 40 | 55 | 31.2 |
100 real episodes, 5 per cell.
Two Tellos still solve Three-sign sum. Gemini 3.7 Flash, the best commander in simulation, is also the best one here.
Drone 2 reads 20 and 35, Drone 1 reads 5 and finds the chairs, and the commander sends Drone 1 from chair 58 on to chair 60. Frames are the drones' own views, about one every two seconds; ▲ a drone reporting arrival, the star the accepted one. The map is approximate: layout from the setup sheet, paths from the go and turn commands each drone executed.
Benchmark, agent and results in full, with the appendix.
Citation
@article{park2026dronecats,
title = {Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents
for Drone Control: Commanding, Approaching, Tracking and Searching},
author = {Park, Jaewoo and Lee, Minyoung and Seo, Sukmin and Yim, Moonbin and
Yoon, Hyunwook and Ryu, Dohoon and Kim, Daehee and Song, Myungseo and
Byun, Jihyuk and Chang, Seunggyu and Kil, Taeho and Kim, Jiseob and
Lee, Bado and Kim, Geewook},
journal = {arXiv preprint arXiv:2609.01404},
year = {2026}
}