31–40 of 50 posts
Can I throw my hat in the ring? I’ve been working on this for.. a long time. AR VR iOS and macOS app for arbitrary code rendering in 3D space. Terminal like rendered glyph by glyph means perfect control over ever mesh and texture. The iOS demo is fun. You can walk around your code like an art museum, draw lines of executed traces, and perform visual hierarchical search. https://github.com/tikimcfee/LookAtThat
Can I throw my hat in the ring? I’ve been working on this for.. a long time. AR VR iOS and macOS app for arbitrary code rendering in 3D space. Terminal like rendered glyph by glyph means perfect control over ever mesh and texture. The iOS demo is fun. You can walk around your code like an art museum, draw lines of executed traces, and perform visual hierarchical search. https://github.com/tikimcfee/LookAtThat
Looks very interesting, thanks for sharing this.
Thanks for asking around and letting me share
Earlier quoted context omitted.
Sounds like a good reason to try to do better. Unless you suppose the UI of programming is solved and everyone who wants to have control over a machine just needs to bite down and learn vim ?
Grabbing AST nodes and dragging them around is never the bottleneck when I'm programming. Cut/paste is plenty efficient even in notepad and doesn't require gross arm movements. Feel free to try it, but I maintain doubt that a literal forest of code is going to be anything more than a gimmick. By all means, prove me wrong!
While all the VR goggles try to embed your body into virtual space (putting the graphics "around you") I'm prototyping with a stereo-microscope, with it's nice heavy knobs for course and fine adjustment, such that you're looking into the virtual world from the outside. No illusions to pull off, no motion sickness, and nothing strapped to your face.
Most programming work is still text editing with various amounts of debugging, refactoring, and autocompletion. It is not obvious to me how an AR interface would make a difference other than more virtual screen real estate. You would still need a way to enter text, build code, run tests, etc. This means a keyboard and pointing device (mouse, trackpad, etc.) are still needed unless something else can do it better. Gra…
> Most programming work is still text editing [...] Text no longer needs to be the primary way of conveying programs. There are practical reasons text works best on screens, but if your coding environment is boundless then there’s no reason to believe you can’t do fancier things like direct manipulation of ASTs pretty easily. Imagine "grabbing" an AST node and attaching it to a different parent, all in space. Beyond…
How do you deal with references? Like defining a variable and using it later in multiple places?
Most programming work is still text editing with various amounts of debugging, refactoring, and autocompletion. It is not obvious to me how an AR interface would make a difference other than more virtual screen real estate. You would still need a way to enter text, build code, run tests, etc. This means a keyboard and pointing device (mouse, trackpad, etc.) are still needed unless something else can do it better. Gra…
But that setup definitely doesn't fit in a backpack. So the idea of duplicating this inside a spatial computing environment is really exciting, even though everything released so far doesn't quite cut it.
But aside from the portability angle, another way to leverage the possibilities would be make the OS, probably using applied statistics trained-model techniques, really good at helping move the 2D planes through space (basically, re-arranging the windows or "screens"). 4 monitors is really all I can get on my desk. In theory I could like, ceiling-mount a row of three more above the main row of left-center-right but IRL that would be way too much work and too low-benefit to be worth doing.
But in a virtual, spatial-compute environment, all kinds of things might be worth doing. Maybe I have 40 or 50 2D planes for editors, debuggers, design schematics, and a team of flying robot ninjas with jetpacks who can instantly move them in and out of position.
The main rationale for a multi-monitor setup like mine is to "see everything at once". I do know how to press Alt-Tab and use virtual desktops, and I understand some people even prefer that workflow.
But for me the ability to keep the main work front and center, but everything in my field of view and glanceable, eliminates hundreds of instances of what would otherwise be window/desktop switching, every day.
If I had Jedi powers, would I just fling a few more monitors to hover in the corner of my room near the ceiling? Sure I would. Why do I need btop taking up space on my main screens? Or my chat thing that I will only use a couple times?
The only reason is that I only have these monitors, right on my desk. In theory I could like, wall-mount more displays for those things I don't really care about, but the effort and time investment to do that is totally not worth it.
But in theory a spatial compute environment could go hog-wild putting less relevant info farther away, but still in view, using space in the room that we just can't practically make use of today. All my screens are 60cm or so from my face.
I suspect we will see novel ideas around how to position these 2D planes, and use applied statistics/trained-model techniques to make the OS better at helping manage them (in a useful, but perhaps also delightful, way) before we transcend the 2D-plane "window" metaphor.
I also don't see avoiding keyboard, for programming or other text-input-heavy tasks. The mouse, though... I mean, we'll see how the eye tracking works once they really ship it. I could see the mouse becoming obsolete on computers that are 20mm away from your eyeballs and know exactly what you are looking at....
just look at what IDEs that runs exclusively on ios are like: crap. nobody invest into a closed platform. i expect jetbrains to came up with something marvelous for ar/vr, but it will run on the upcoming version of Microsoft or HP glasses. you know, the only ones today that works just like an external monitor without a locked in ecosystem like apple or facebook. the silly apps and games and such will net millions tho…
With tools like Remote Development extensions, VS Code already works superbly from a desktop or laptop, editing locally but using a remote machine (your own or some cloud thing) for the filesystem, git, compilation, dependency management, development server if its a web/network app, etc.
It also works pretty great on the iPad, other than the tiny screen (which yeah, is a dealbreaker IMHO, but wouldn't be on a many-screens device like Vision Pro).
It also works pretty fabulously in a web browser with that same setup (e.g. Github Codespaces, but you can roll your own, too).
I doubt we will see a great IDE that does everything locally on the device. But a great IDE client is eminently doable; it's basically already done.
The IDE is one application type where the tooling around offloading all the compute, and just keeping the UI local, has flourished. VS Code was the first to really do it so it was just as good or better as developing 100% locally. JetBrains is right behind them. Others are doing it too.
This was the only way iPad OS became usable as development machines. I don't expect there to be any good native IDEs for Vision OS, either.
But if you can deal with requiring a network connection, and using VS Code, you will be able to use Vision Pro to develop on a remote Linux machine (or Mac or Windows, if that is your thing).
Whether JetBrains will make an IDE client for it, enable third parties to make that, or not support it at all remains to be seen, I guess.
Can I throw my hat in the ring? I’ve been working on this for.. a long time. AR VR iOS and macOS app for arbitrary code rendering in 3D space. Terminal like rendered glyph by glyph means perfect control over ever mesh and texture. The iOS demo is fun. You can walk around your code like an art museum, draw lines of executed traces, and perform visual hierarchical search. https://github.com/tikimcfee/LookAtThat
Most programming work is still text editing with various amounts of debugging, refactoring, and autocompletion. It is not obvious to me how an AR interface would make a difference other than more virtual screen real estate. You would still need a way to enter text, build code, run tests, etc. This means a keyboard and pointing device (mouse, trackpad, etc.) are still needed unless something else can do it better. Gra…
> Most programming work is still text editing [...] Text no longer needs to be the primary way of conveying programs. There are practical reasons text works best on screens, but if your coding environment is boundless then there’s no reason to believe you can’t do fancier things like direct manipulation of ASTs pretty easily. Imagine "grabbing" an AST node and attaching it to a different parent, all in space. Beyond…
I think it boils down to it being a “programming language”
What we need for AR / VR is “programming gestures”
This way there is no syntax but visual mediums you manipulate via gestures. And this would get compiled to a binary which can be executed.