Enough AI copilots, we need AI HUDs • Hacker News Board

Re: Enough AI copilots, we need AI HUDs

#221

Post by cleverwebble » Mon, Jul 28, 2025, 5:25 PM UTC

_jab wrote: post #218

There’s a lot of ideation for coding HUDs in the comments, but ironically I think the core feature of most coding copilots is already best described as a HUD: tab completion. And interestingly, that is indeed the feature I find most compelling from Cursor. I particularly love when I’m doing a small refactor, like changing a naming convention for a few variables, and after I make the first edit manually Cursor will ju…

Agreed!

I've spent the last few months using Claude Code and Cursor - experimenting with both. For simple tasks, both are pretty good (like identifying a bug given console output) - but when it comes to making a big change, like adding a brand new feature to existing code that requires changes to lots of files, writing tests, etc - it often will make at least a few mistakes I catch on review, and then prompting the model to fix those mistakes often causes it to fix things in strange ways.

A few days ago, I had a bug I just couldn't figure out. I prompted Claude to diagnose and fix the issue - but after 5 minutes or so of it trying out different ideas, rerunning the test, and getting stuck just like I did - it just turned off the test and called it complete. If I wasn't watching what it was doing, I could have missed that it did that and deployed bad code.

The last week or so, I've totally switched from relying on prompting to just writing the code myself and using tab complete to autocomplete like 80% of it. It is slower, but I have more control and honestly, it's much more enjoyable of an experience.

Re: Enough AI copilots, we need AI HUDs

#222

Post by cleverwebble » Mon, Jul 28, 2025, 5:34 PM UTC

irthomasthomas wrote: post #179

newswasboring wrote: post #126

Earlier quoted context omitted.

Can you elaborate on how would one do this calculation?

import openai, math, os, textwrap, json, sys query = 'Paris is the capital of' # short demo input os.environ['OPENAI_API_KEY'] # check key early client = openai.OpenAI() resp = client.chat.completions.create( model='gpt-3.5-turbo', messages=[{'role': 'user', 'content': query}], max_tokens=12, logprobs=True, top_logprobs=1 ) logprobs = [t.logprob for t in resp.choices[0].logprobs.content] perplexity = math.exp(-sum(lo…

If you want to generate a heatmap of existing text, you will have to take a different approach here.

The naive solution I could come up with would be really expensive with openai, but if you have an open source model, you can write up custom inference that goes one-token-at-a-time through the text, and on each token you look up the difference in logprobs between the token that the LLM predicted vs what was actually there, and use that to color the token.

The downside I imagine to this approach is it would probably tend to highlight the beginning of bad code, and not the entire block - because once you commit to a mistake, the model will generally roll with it - ie, a 'hallucination' - so logprobs of tokens after the bug happened might only be slightly higher than normal.

Another option might be to use a diffusion based model, adding some noise to the input and having it iterate a few times through, then measuring the parts of the text that changed the most. I have only a light theory understanding of these models though, so I'm not sure how well that would work

Re: Enough AI copilots, we need AI HUDs

#223

Post by timeinput » Mon, Jul 28, 2025, 5:43 PM UTC

There is a giant confusing tech stack here, but opening up this simple plain text page resulted in like 100% CPU usage from from gnome-shell when visiting from firefox on a pretty non-customized Ubuntu 24 install.

In the firefox task manager nothing really looked odd, but opening that tab and displaying it is insanely CPU intensive.

Pausing the autoplaying video makes it seem like a sane web page in terms of CPU usage. I'm surprised how much CPU playing that video consumed.

Re: Enough AI copilots, we need AI HUDs

#224

Post by GuB-42 » Mon, Jul 28, 2025, 5:46 PM UTC

furyofantares wrote: post #68

I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.

This! That's what I wanted since LLMs learned how to code.

And in fact, I think I saw a paper / blog post that showed exactly this, and then... nothing. For the last few years, the tech world became crazy with code generation, with forks of VSCode hooked to LLMs worth billions of dollars and all that. But AI-based code analysis is remarkably poor. The only thing I have seen resembling this is bug report generators, which is I believe is one of the worst approach.

The idea you have, that I also had and I am sure many thousands of other people had seem so obvious, why is no one talking about it? Is there something wrong with it?

The thing is, using such a feature requires a brain between the keyboard and the chair. A "surprising" token can mean many things: a bug, but also a unique feature, anyways, something you should pay attention to. Too much "green" should also be seen as a signal. Maybe you reinvented the wheel and you should use a library instead, or maybe you failed to take into account a use case specific to your application.

Maybe such tools don't make good marketing. You need to be a competent programmer to use them. It won't help you write more lines faster. It doesn't fit the fantasy of making anyone into a programmer with no effort (hint: learning a programming language is not the hard part). It doesn't generate the busywork of AI 1 introducing bugs for AI 2 to create tickets for.

Re: Enough AI copilots, we need AI HUDs

#225

Post by dreamcompiler » Mon, Jul 28, 2025, 5:48 PM UTC

The main reasons I don't have "smart" devices in my house:

1. They introduce new and fascinating failure modes that never happened with the old, dumb devices (e.g. if your router fails you lose the ability to control your lights).

2. They demand human attention at the slightest provocation (e.g. the microwave beeps loudly forever when your food is done, every app on your phone insists on interacting with you whenever the company would like to upsell you something, etc.)

Item 2 above is what TFA is about. Yes you can often turn this shit off, but that's not the point. The point is you shouldn't have to. Useful technology should never call attention to itself in the manner of someone with narcissistic personality disorder.

But what about emergency situations? Glad you asked. Many airplane crashes in modern aircraft have happened because of "warning buzzer overload" which happens when one important system on the aircraft fails and then causes a cascade of secondary warnings, while giving the pilot no insight as to the root cause. A true AI assistant would reason about such situations and guide the pilot toward the root solution.

A true coding assistant would do the same kind of reasoning about program errors and suppress multipage error dumps in favor of flagging the root issue.

Re: Enough AI copilots, we need AI HUDs

#226

Post by itslennysfault » Mon, Jul 28, 2025, 5:53 PM UTC

timeinput wrote: post #223

There is a giant confusing tech stack here, but opening up this simple plain text page resulted in like 100% CPU usage from from gnome-shell when visiting from firefox on a pretty non-customized Ubuntu 24 install. In the firefox task manager nothing really looked odd, but opening that tab and displaying it is insanely CPU intensive. Pausing the autoplaying video makes it seem like a sane web page in terms of CPU usag…

Very odd. This must be some kind of Ubuntu issue. Out of curiosity I tried it in Chrome, Firefox and Safari on MacOS 15.5 and saw very little CPU usage and no difference with/without the video playing. I don't have a Ubuntu Desktop handy to confirm. Looking at the video it's nothing special. Just a fairly small mp4 using a native html5 video element to play it. Really no reason this should be causing issues.

Re: Enough AI copilots, we need AI HUDs

#227

Post by furyofantares » Mon, Jul 28, 2025, 6:13 PM UTC

GuB-42 wrote: post #224

furyofantares wrote: post #68

I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.

This! That's what I wanted since LLMs learned how to code. And in fact, I think I saw a paper / blog post that showed exactly this, and then... nothing. For the last few years, the tech world became crazy with code generation, with forks of VSCode hooked to LLMs worth billions of dollars and all that. But AI-based code analysis is remarkably poor. The only thing I have seen resembling this is bug report generators, w…

> The idea you have, that I also had and I am sure many thousands of other people had seem so obvious, why is no one talking about it? Is there something wrong with it?

I expect it definitely requires some iteration, I don't think you can just map logits to heat, you get a lot of noise that way.

Re: Enough AI copilots, we need AI HUDs

#228

Post by jonchurch_ » Mon, Jul 28, 2025, 6:22 PM UTC

cleverwebble wrote: post #221

_jab wrote: post #218

There’s a lot of ideation for coding HUDs in the comments, but ironically I think the core feature of most coding copilots is already best described as a HUD: tab completion. And interestingly, that is indeed the feature I find most compelling from Cursor. I particularly love when I’m doing a small refactor, like changing a naming convention for a few variables, and after I make the first edit manually Cursor will ju…

Agreed! I've spent the last few months using Claude Code and Cursor - experimenting with both. For simple tasks, both are pretty good (like identifying a bug given console output) - but when it comes to making a big change, like adding a brand new feature to existing code that requires changes to lots of files, writing tests, etc - it often will make at least a few mistakes I catch on review, and then prompting the m…

Drop in a lint rule to fail on skipped tests. Ive added these at a previous job after finding that tests skipped during dev sometimes slipped through review and got merged.

Re: Enough AI copilots, we need AI HUDs

#229

Post by jFriedensreich » Mon, Jul 28, 2025, 6:27 PM UTC

My aha moment into this direction happened when i got XR glasses and tried embracing voice agents to see where this leads. Did not expect the HUD aspect be so cool of how this setup works. At first i tried using the glasses as a replacement for my main screen and using voice agents as a replacement to the sidebars and chat windows we are used to, which worked ok but then i started using the main screen and main agent interfaces again but kept the glasses with a second screen on a different visual focal plane connected to the voice agents. Dark mode gets a totally new importance there because black is transparent in XR which is important to work well without obscuring the main screen. I can switch between screens by shifting eye focus and have a sense what the other screen is doing a far better than just how a physical second screen is in the visual periphery. When i need to combine images more i move the focal planes closer together so i can see both screens better at the same time. The voice agents can answer side questions, start research that is needed for the next step of the main workflow or fix minor issues that are not important enough to interrupt the main workflow. Its so obvious how HUD, voice ai, XR and agents can grow together into this new computing environment but i am afraid what happens once android and ios shape that reality. I want this to be part of the web.

Re: Enough AI copilots, we need AI HUDs

#230

Post by marcosdumay » Mon, Jul 28, 2025, 7:01 PM UTC

GuB-42 wrote: post #224

furyofantares wrote: post #68

I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.

This! That's what I wanted since LLMs learned how to code. And in fact, I think I saw a paper / blog post that showed exactly this, and then... nothing. For the last few years, the tech world became crazy with code generation, with forks of VSCode hooked to LLMs worth billions of dollars and all that. But AI-based code analysis is remarkably poor. The only thing I have seen resembling this is bug report generators, w…

Just to point...

> Is there something wrong with it?

> Maybe such tools don't make good marketing.

You had the answer the entire time :)

Features that require a brain between the AI and key-presses just don't sell. Don't expect to see them for sale. (But we can still get them for free.)