Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Only 1 LLM can fly a drone

github.com

Post reply on HN
Print view View original

51–60 of 100 posts

Re: Show HN: Only 1 LLM can fly a drone

#51

Post by SpyCoder77 »

fsiefken wrote: post #22

I am curious how these models would perform and how much energy they'd take to semi-realtime detect objects: SmolVLM2-500M - Moondream 0.5B/2B/2.5B - Qwen3-VL (3B) https://huggingface.co/collections/Qwen/qwen3-vl I am sure this is already worked on in Russia, Ukraine and The Netherlands. A lot can go wrong with autonomous flying. One could load the VLM on a high end android phone on the drone and have dual control.

A better way would be a VLA as opposed to a VLM. VLAs are meant to take action, where as vlms are for geneeral use. https://cognitivedrone.github.io/

Re: Show HN: Only 1 LLM can fly a drone

#52

Post by storystarling »

bob1029 wrote: post #20
bigfishrunning wrote: post #2

Why would you want an LLM to fly a drone? Seems like the wrong tool for the job -- it's like saying "Only one power drill can pound roofing nails". Maybe that's true, but just get a hammer

The system prompt for the drone is hilarious to me. These models are horrible at spatial reasoning tasks: https://github.com/kxzk/snapbench/blob/main/llm_drone/src/ma... I've been working with integrating GPT-5.2 in Unity. It's fantastic at scripting but completely worthless at managing transforms for scene objects. Even with elaborate planning phases it's going to make a complete jackass of itself in world space eve…

Agreed. I’ve found the only reliable architecture for this is treating the LLM purely as a high-level planner rather than a controller.

We use a state machine (LangGraph) to manage the intent and decision tree, but delegate the actual transform math to deterministic code. You really want the model deciding the strategy and a standard solver handling the vectors, otherwise you're just burning tokens to crash into walls.

Re: Show HN: Only 1 LLM can fly a drone

#53

Post by volkercraig »

pixl97 wrote: post #34
volkercraig wrote: post #31

Earlier quoted context omitted.

I don't think you understand what an "LLM" is. They're text generators. We've had autopilot since the 1930s that relies on measurable things... like PID loops, direct sensor input. You don't need the "language model" part to run an autopilot, that's just silly.

You see to be talking past him and ignoring what they are actually saying. LLMs are a higher level construct than PID loops. With things like autopilot I can give the controller a command like 'Go from A to B', and chain constructs like this to accomplish a task. With an LLM I can give the drone/LLM system complex command that I'd never be able to encode to a controller alone. "Fly a grid over my neighborhood, docume…

They are text generators, and yes they are pretty good, but that really is all they are, they don't actually learn, they don't actually think. Every "intelligence" feature by every major AI company relies on semantic trickery and managing context windows. It even says it right on the tin; Large LANGUAGE Model.

Let me put it this way: What OP built is an airplane in which a pilot doesn't have a control stick, but they have a keyboard, and they type commands into the airplane to run it. It's a silly unnecessary step to involve language.

Now what you're describing is a language problem, which is orchestration, and that is more suited to an LLM.

Re: Show HN: Only 1 LLM can fly a drone

#54

Post by volkercraig »

encrux wrote: post #37
volkercraig wrote: post #30

I don't understand. Surely training an LSTM with sensor input is more practical and reasonable way than trying to get a text generator to speak commands to a drone.

Very much depends on what you want to do. The fact that a language model can „reason“ (in the LLM-slang meaning of the term) about 3D space is an interesting property. If you give a text description of a scene and ask a robot to perform a peg in hole task, modern models are able to solve them fairly easily based on movement primitives. I implemented this on a UR robot arm back in 2023 The next logical step is, instea…

I mean semantically language evolved as an interpretation for the material world, so assuming that you can describe a problem in language, and considering that there exists a solution to said problem that is describable in language, then I'm sure a big enough LLM could do it... but you can also calculate highly detailed orbital maps with epicycles if you just keep adding more... you just don't because it's a waste of time and there's a simpler way.

The latter part is interesting. I'm not sure how the performance of one of those would be once they are working well, but my naive gut feeling is that splitting the language part and the driving part into two delegates is cleaner, safer, faster and more predictable.

Re: Show HN: Only 1 LLM can fly a drone

#55

Post by irl_zebra »

infecto wrote: post #42
irl_zebra wrote: post #39

Earlier quoted context omitted.

SOTA typically refers to achieving the best performance, not using the trendiest thing regardless of performance. There is some subtlety here. At some point an LLM might give the best performance in this task, but that day is not today, so an LLM is not SOTA, just trendy. It's kinda like rewriting something in Rust and calling it SOTA because that's the trend right now. Hope that makes sense.

I don’t think trendy is really the right word and maybe it’s not state of the art but a lot of us in the industry are seeing emerging capabilities that might make it SOTA. Hope that makes sense.

LLMs are indeed the definition of trendy (I've found using Google Trends to dive in is a good entry point to get a broad sense of whether something is "trendy")! Basically the right way to think about it is that something can be promising, and demonstrate emerging capabilities, but but those things don't make something SOTA, nor do they make it trendy. They can be related though (I expect everything SOTA was once promising and emerging, but not everything promising or emerging became SOTA). It's a subtlety that isn't super easy to grasp, but (and here is one area I think an LLM can show promise) an LLM like ChatGPT can help unpick the distinctions here. Still, it's slightly nuanced and I understand the confusion.

Re: Show HN: Only 1 LLM can fly a drone

#56

Post by convolvatron »

volkercraig wrote: post #54
encrux wrote: post #37

Earlier quoted context omitted.

Very much depends on what you want to do. The fact that a language model can „reason“ (in the LLM-slang meaning of the term) about 3D space is an interesting property. If you give a text description of a scene and ask a robot to perform a peg in hole task, modern models are able to solve them fairly easily based on movement primitives. I implemented this on a UR robot arm back in 2023 The next logical step is, instea…

I mean semantically language evolved as an interpretation for the material world, so assuming that you can describe a problem in language, and considering that there exists a solution to said problem that is describable in language, then I'm sure a big enough LLM could do it... but you can also calculate highly detailed orbital maps with epicycles if you just keep adding more... you just don't because it's a waste of…

note that the control systems you were talking about before (i.e. PID) would probably take hold pretty directly in a tiny network, and exactly because of that limitation, be far less likely to contain 'hallucinations'. object avoidance and path planning are likely similar.

since this is a limited and continuous domain, its a far better one for neural training than natural language. I guess this notion that a language model should be used for 3d motion control is a real indicator about the level of thought going into some of these applications.

Re: Show HN: Only 1 LLM can fly a drone

#57

Post by famouswaffles »

irl_zebra wrote: post #39
avaer wrote: post #25

Earlier quoted context omitted.

Using an LLM is the SOTA way to turn plain text instructions into embodied world behavior. Charitably, I guess you can question why you would ever want to use text to command a machine in the world (simulated or not). But I don't see how it's the wrong tool given the goal.

SOTA typically refers to achieving the best performance, not using the trendiest thing regardless of performance. There is some subtlety here. At some point an LLM might give the best performance in this task, but that day is not today, so an LLM is not SOTA, just trendy. It's kinda like rewriting something in Rust and calling it SOTA because that's the trend right now. Hope that makes sense.

>Using an LLM is the SOTA way to turn plain text instructions into embodied world behavior.

>SOTA typically refers to achieving the best performance

Multimodal Transformers are the best way to turn plain text instructions to embodied world behavior. Nothing to do with being 'trendy'. A Vision Language Action model would probably have done much better but really the only difference between that and the models trialed above is training data. Same technology.

Re: Show HN: Only 1 LLM can fly a drone

#58

Post by mbreese »

I can’t really take this too seriously. This seems to me to be a case of asking “can an LLM do X?” Instead, the question is like to see is: “I want to do X, is an LLM this right tool?”

But that said, I think the author missed something. LLMs aren’t great at this type of reasoning/state task, but they are good at writing programs. Instead of asking the LLM to search with a drone, it would be very interesting to know how they performed if you asked them to write a program to search with a drone.

This is more aligned with the strengths of LLMs, so I could see this as having more success.

Re: Show HN: Only 1 LLM can fly a drone

#59

Post by pixl97 »

volkercraig wrote: post #53
pixl97 wrote: post #34

Earlier quoted context omitted.

You see to be talking past him and ignoring what they are actually saying. LLMs are a higher level construct than PID loops. With things like autopilot I can give the controller a command like 'Go from A to B', and chain constructs like this to accomplish a task. With an LLM I can give the drone/LLM system complex command that I'd never be able to encode to a controller alone. "Fly a grid over my neighborhood, docume…

They are text generators, and yes they are pretty good, but that really is all they are, they don't actually learn, they don't actually think. Every "intelligence" feature by every major AI company relies on semantic trickery and managing context windows. It even says it right on the tin; Large LANGUAGE Model. Let me put it this way: What OP built is an airplane in which a pilot doesn't have a control stick, but they…

[dead]

Re: Show HN: Only 1 LLM can fly a drone

#60

Post by zahlman »

> I gave 7 frontier LLMs a simple task: pilot a drone through a 3D voxel world and find 3 creatures.

> Only one could do it.

If I understood the chart correctly, even the successful one only found 1/6 of the creatures across multiple runs.

Post reply on HN