I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.
That’s actually fantastic as an idea
Enough AI copilots, we need AI HUDs
71–80 of 290 posts
Re: Enough AI copilots, we need AI HUDs
#72I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.
That's a really cool idea. Also the inverse, where suggestions from the AI were similarly heat mapped for confidence would be extremely useful.
Re: Enough AI copilots, we need AI HUDs
#73I'd rather flip this around and be in a fully immersive environment watching agents do things in such a way that I am able to interject and guide in realtime. How do we build that, and build it in such a way that the content and delivery of my guidance becomes critical to what they learn? The best teacher gets the best AI students.
> fully immersive environment
Start with a snapshot of what you are envisioning using Blender.
Re: Enough AI copilots, we need AI HUDs
#74I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.
previously undefined variable and function names would be red as well
Re: Enough AI copilots, we need AI HUDs
#75Love the idea & spitballing ways to generalize to coding.. Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take The tests could appear in a separated panel next to your code, and pass/fail status in the gutter of that panel. As simple as red and green dots for tests t…
> Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take I think this is a bad approach. Tests enforce invariants, and they are exactly the type of code we don't want LLMs to touch willy-nilly.
You want your tests to only change if you explicitly want them to, and even then only the tests should change.
Once you adopt that constraint, you'll quickly realize ever single detail of your thought experiment is already a mundane workflow in any developer's day-to-day activities.
Consider the fact that watch mode is a staple of any JavaScript testing framework, and those even found their way into .NET a couple of years ago.
So, your thought experiment is something professional software developers have been doing for what? A decade now?
Re: Enough AI copilots, we need AI HUDs
#76Love the idea & spitballing ways to generalize to coding.. Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take The tests could appear in a separated panel next to your code, and pass/fail status in the gutter of that panel. As simple as red and green dots for tests t…
There's no way this would work for any serious C++ codebase. Compile times alone make this impossible I'm also not sure how LLM could guess what the tests should be without having written all of the code, e.g. imagine writing code for a new data structure
> There's no way this would work for any serious C++ codebase. Compile times alone make this impossible
There's nothing in C++ that prevents this. If build times are your bogeyman, you'd be pleased to know that all mainstream build systems support incremental builds.
Re: Enough AI copilots, we need AI HUDs
#77Earlier quoted context omitted.
Humans writing the test first and LLM writing the code is much better than the reverse. And that is because tests are simply the “truth” and “intention” of the code as a contract. When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat.
> When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat. You don’t need to write tests for that, you need to write acceptance criteria.
> You don’t need to write tests for that, you need to write acceptance criteria.
Sir, those are called tests.
Re: Enough AI copilots, we need AI HUDs
#78I might be wrong but isn't the HUD the author suggesting for coding is basically AREPL? For debugging I can see it work, but chatbox and inline q&a I feel has awider application.
On a wider note, I buy the argument for alternative interfaces other than chat, but chat permeates our lives every day, smartphone is full of chat interfaces. HUD might be good for AR glasses though, literal HUD.
Re: Enough AI copilots, we need AI HUDs
#79I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.
I want that in an editor. It's also a good way to check if your writing is too predictable or cliche.
The perplexity calculation isn't difficult; just need to incorporate it into the editor interface.
Re: Enough AI copilots, we need AI HUDs
#80Earlier quoted context omitted.
There's no way this would work for any serious C++ codebase. Compile times alone make this impossible I'm also not sure how LLM could guess what the tests should be without having written all of the code, e.g. imagine writing code for a new data structure
> There's no way this would work for any serious C++ codebase. Compile times alone make this impossible There's nothing in C++ that prevents this. If build times are your bogeyman, you'd be pleased to know that all mainstream build systems support incremental builds.
The original example was (paraphrasing) "rerunning 10-100 tests that take 1ms after each keystroke".
Even with incremental builds, that surely does not sound plausible? I only mentioned C++ because that's my main working language, but this wouldn't sound reasonable for Rust either, no?