Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Only 1 LLM can fly a drone

github.com

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Show HN: Only 1 LLM can fly a drone

#71

Post by notepad0x90 »

volkercraig wrote: post #31
notepad0x90 wrote: post #13

Earlier quoted context omitted.

We don't need a lot of things, but new tech should also address what people want, not just needs. I don't know how to pilot drones, nor do I care to learn how to, but I want to do things with drones, does that qualify as a need? Tech is there to do things for us we're too lazy to do.

I don't think you understand what an "LLM" is. They're text generators. We've had autopilot since the 1930s that relies on measurable things... like PID loops, direct sensor input. You don't need the "language model" part to run an autopilot, that's just silly.

LLMs can do chat-completion, they don't do only chat completion. There are LLMs for image generation, voice generation, video generation and possibly more. The camera of a drone inputs images for the LLM, then it determines what action take based on that. Similar to if you asked ChatGPT "there is a tree in this picture, if you were operating a drone, what action would you take to avoid collision", except the "there is a tree" part is done by the LLMs image recognition, and the sys prompt is "recognize objects and avoid collision", of course I'm simplifying it a lot but it is essentially generating navigational directions under a visual context using image recognition.

Re: Show HN: Only 1 LLM can fly a drone

#72

Post by notepad0x90 »

iso1631 wrote: post #18
notepad0x90 wrote: post #10

Earlier quoted context omitted.

There are almost endless reasons why. It's like asking why would you want a self-driving car. Having a drone to transport things would be amazing, or to patrol an area. LLMs can be helpful with object identification, reacting to different events, and taking commands from users. The first thought I had was those security guard robots that are popping up all over the place. if they were drones instead, and LLM talked t…

You want a self driving car You don't want an LLM to drive a car There is more to "AI" than LLMs

I don't mind someone trying LLMs to see if they can do better than existing ML solutions.

Re: Show HN: Only 1 LLM can fly a drone

#73

Post by cheema33 »

volkercraig wrote: post #31
notepad0x90 wrote: post #13

Earlier quoted context omitted.

We don't need a lot of things, but new tech should also address what people want, not just needs. I don't know how to pilot drones, nor do I care to learn how to, but I want to do things with drones, does that qualify as a need? Tech is there to do things for us we're too lazy to do.

I don't think you understand what an "LLM" is. They're text generators. We've had autopilot since the 1930s that relies on measurable things... like PID loops, direct sensor input. You don't need the "language model" part to run an autopilot, that's just silly.

"You don't need the "language model" part to run an autopilot, that's just silly."

I think most of us understood that reproducing what existing autopilot can do was not the goal. My inexpensive DJI quadcopter has an impressive abilities in this area as well. But, I cannot give it a mission in natural language and expect it to execute it. Not even close.

Re: Show HN: Only 1 LLM can fly a drone

#74

Post by infecto »

bigfishrunning wrote: post #68
infecto wrote: post #15

Earlier quoted context omitted.

What’s the right tool then? This looks like a pretty fun project and in my rough estimation a fun hacker project.

The right tool would likely be some conventional autopilot software; if you want AI cred you could train a Neural Network which maps some kind of path to the control features of the drone. LLMs are language models -- good for language, but not good for spacial reasoning or navigation or many of the other things you need to pilot a drone.

So you are suggesting building a full featured package that is nontrivial compared to this fun excitement?

Vision models do a pretty decent job with spatial reasoning. It’s not there yet but you’re dismissing some interesting work going on.

Re: Show HN: Only 1 LLM can fly a drone

#75

Post by beigebrucewayne »

SpyCoder77 wrote: post #49
modeless wrote: post #40

This is what VLA models are for. They would work much better. Would need a bit of fine tuning but probably not much. Lots of literature out there on using VLAs to control drones.

Did some research, found a model that is exactly that. https://cognitivedrone.github.io/

Thanks will check this out!

Re: Show HN: Only 1 LLM can fly a drone

#76

Post by dimatura »

This is neat! It's a bit amusing in that I worked on a somewhat similar project for my phd thesis almost 10 years ago, although in that case we got it working on a real drone (heavily customized, based on DJI matrice) in the field, with only onboard compute. Back then it was just a fairly lightweight CNN for the perception, not that we could've gotten much more out of the jetson TX2.

Re: Show HN: Only 1 LLM can fly a drone

#77

Post by uoaei »

zahlman wrote: post #60

> I gave 7 frontier LLMs a simple task: pilot a drone through a 3D voxel world and find 3 creatures. > Only one could do it. If I understood the chart correctly, even the successful one only found 1/6 of the creatures across multiple runs.

No science detected.

Without comparison to some null hypothesis (a random policy), this article is hogwash.

Re: Show HN: Only 1 LLM can fly a drone

#78

Post by zahlman »

uoaei wrote: post #77
zahlman wrote: post #60

> I gave 7 frontier LLMs a simple task: pilot a drone through a 3D voxel world and find 3 creatures. > Only one could do it. If I understood the chart correctly, even the successful one only found 1/6 of the creatures across multiple runs.

No science detected. Without comparison to some null hypothesis (a random policy), this article is hogwash.

Given that all the other agents failed to find any creatures, it's hard to imagine that a random policy would except by extreme coincidence.
Post reply on HN