Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Enough AI copilots, we need AI HUDs

geoffreylitt.com

Post reply on HN
Print view View original

51–60 of 290 posts

Re: Enough AI copilots, we need AI HUDs

#51

Post by skydhash »

CGamesPlay wrote: post #46
wewewedxfgdf wrote: post #34

Kind of a weird article because the computer systems that is "invisible" i.e. an integrated part of the flight control systems - is exactly what we have now. He's sort of arguing for .... computer software. Like, we have HUDs - that's what a HUD is - it's a computer program.

A HUD is typically non-interactive, which is the core distinction he’s advocating for. The “copilot” responds to your requests, the “HUD” surfaces relevant information passively.

Isn't this one of the core selling of an IDE compared to a simple editor? You have the analyzer running in the background and telling you things. And a host of buttons around the editor to act on stuff, all within the context of a project.

Re: Enough AI copilots, we need AI HUDs

#52

Post by skydhash »

jpm_sd wrote: post #15

This is how ship's AI is depicted in The Expanse (TV series) and I think it's really compelling. Quiet and unobtrusive, but Alex can ask the Rocinante to plot a new course or display the tactical situation and it's fast, effective and effortlessly superhuman with no back-talk or unnecessary personality. Compare another sci-fi depiction taken to the opposite extreme: Sirius Cybernetics products in the Hitchhikers Guid…

I may remember wrongly, but I don't believe the expanse was depicting AI. It was more powerful computation and unobtrusive interfaces. There was nothing like Jarvis. Rocinante was a war vessel and all its feature was tailored to that. I believe even the mechanical suits of the Martians were very manual (no cortana a la Master Chief).

Re: Enough AI copilots, we need AI HUDs

#53

Post by energy123 »

hi_hi wrote: post #8

Doesn't it all come down to "what is the ideal interface for humans to deal with digital information"? We're getting more and more information thrown at us each day, and the AIs are adding to that, not reducing it. The ability to summarise dense and specialist information (I'm thinking error logs, but could be anything really) just means more ways for people to access and view that information who previously wouldn't…

The designers of 6th gen fighter jets are confronting the same challenge. The cockpit, which is an interface between the pilot and the airframe, will be optionally manned. If the cockpit is manned, the pilot will take on a reduced set of roles focused on higher-level decision making.

By the 7th generation it's hard to see how humans will still be value-add, unless it's for international law reasons to keep a human in the loop before executing the kill chain, or to reduce Skynet-like tail risks in line with Paul Christiano's arms race doom scenario.

Perhaps interfaces in every domain will evolve this way. The interface will shrink in complexity, until it's only humans describing what they want to the system, at higher and higher levels of abstraction. That doesn't necessarily have to be an English-language interface if precision in specification is required.

Re: Enough AI copilots, we need AI HUDs

#54

Post by skydhash »

JimDabell wrote: post #39
ThunderSizzle wrote: post #35

Earlier quoted context omitted.

As in, a developer would write something in e.g. gherkin, and AI would automatically create the matching unit tests and the production code? That would be interesting. Of course, gherkin tends to just be transpiled into generated code that is customized for the particular test, so I'm not sure how AI can really abstract it away too much.

I’m talking higher level than that. Think about the acceptance criteria you would put in a user story. I’m specifically responding to this: > When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat. You don’t need to personally write code that mechanically iterates over every possible state to remain in the driver’s seat. You need to desc…

I think your perspective is heavily influenced by the imperative paradigm where you actually write the state transition. Compare that to functional programming where you only describe the relation between the initial and final state. Or logic programming where you describe the properties of the final state and where it would find the elements with those properties in the initial state.

Those does not involves writing state transitions. You are merely describing the acceptance criteria. Imperative is the norm because that's how computers works, but there are other abstractions that maps more to how people thinks. Or how the problem is already solved.

Re: Enough AI copilots, we need AI HUDs

#55

Post by cellis »

cjonas wrote: post #44
cadamsdotcom wrote: post #4

Love the idea & spitballing ways to generalize to coding.. Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take The tests could appear in a separated panel next to your code, and pass/fail status in the gutter of that panel. As simple as red and green dots for tests t…

Then do you need tests to validate your tests are correct, otherwise the LLM might just generate passing code even if the test is bad? Or write code that games the system because it's easier to hardcode an output value then to do the actual work. There probably is a setup where this works well, but the LLM and humans need to be able to move across the respective boundaries fluidly... Writing clear requirements and le…

The harder part is “test invalidation”. For instance if a feature no longer makes sense, the human / test validator must painstakingly go through and delete obsolete specs. An idea I’d like to try is to “separate” the concerns; only QA agents can delete specs, engineer agents must conform to the suite, and make a strong case to the qa agent for deletion.

Re: Enough AI copilots, we need AI HUDs

#56

Post by ravila4 »

I think one key reason HUDs haven’t taken off more broadly is the fundamental limitation of our current display medium - computer screens and mobile devices are terrible at providing ambient, peripheral information without being intrusive. When I launch an AI agent to fix a bug or handle a complex task, there’s this awkward wait time where it takes too long for me to sit there staring at the screen waiting for output, but it’s too short for me to disengage and do something else meaningful. A HUD approach would give me a much shorter feedback loop. I could see what the AI is doing in my peripheral vision and decide moment-to-moment whether to jump in and take over the coding myself, or let the agent continue while I work on something else. Instead of being locked into either “full attention on the agent” or “completely disengaged,” I’d have that ambient awareness that lets me dynamically choose my level of involvement. This makes me think VR/AR could be the killer application for AI HUDs. Spatial computing gives us the display paradigm where AI assistance can be truly ambient rather than demanding your full visual attention on a 2D screen. I picture that this would be especially helpful for help on more physical tasks, such as cooking, or fixing a bike.

Re: Enough AI copilots, we need AI HUDs

#57

Post by elliotec »

ravila4 wrote: post #56

I think one key reason HUDs haven’t taken off more broadly is the fundamental limitation of our current display medium - computer screens and mobile devices are terrible at providing ambient, peripheral information without being intrusive. When I launch an AI agent to fix a bug or handle a complex task, there’s this awkward wait time where it takes too long for me to sit there staring at the screen waiting for output…

You just described what I do with my ultrawide monitor and laptop screen.

I can be fully immersed in a game or anything and keep Claude in a corner of a tmux window next to a browser on the other monitor and jump in whenever I see it get to the next step or whatever.

Re: Enough AI copilots, we need AI HUDs

#59

Post by skydhash »

benjaminwootton wrote: post #19

I think there is a third and distinct model which is AI that runs in the background autonomously amd over a long period and pushes things to you. It can detect situations intelligently, do the filtering, summarisation of what’s happening and possibly a recommendation. This feels a lot more natural to me, especially in a business context when you want to monitor for 100 situations about thousands of customers.

Actually defining those situations and collecting the data (which should help identify those situations) are the hard parts. Having an autonomous system that do it has been solved for ages.

Re: Enough AI copilots, we need AI HUDs

#60

Post by ravila4 »

elliotec wrote: post #57
ravila4 wrote: post #56

I think one key reason HUDs haven’t taken off more broadly is the fundamental limitation of our current display medium - computer screens and mobile devices are terrible at providing ambient, peripheral information without being intrusive. When I launch an AI agent to fix a bug or handle a complex task, there’s this awkward wait time where it takes too long for me to sit there staring at the screen waiting for output…

You just described what I do with my ultrawide monitor and laptop screen. I can be fully immersed in a game or anything and keep Claude in a corner of a tmux window next to a browser on the other monitor and jump in whenever I see it get to the next step or whatever.

It’s a similar idea, but imagine you could fire off a task, and go for a run, or do the dishes. Then be notified when it completes, and have the option to review the changes, or see a summary of tests that are failing, without having to be at your workstation.
Post reply on HN