Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Natural language is an unnatural interface

varunshenoy.substack.com

Post reply on HN
Print view View original

81–90 of 94 posts

Re: Natural language is an unnatural interface

#81

Post by Verdex »

TeMPOraL wrote: post #40
iamflimflam1 wrote: post #13

Earlier quoted context omitted.

Yet this is how we communicate with designers and end up with good results.

I don't think they see it the same way. At least not given that instructions in the style GP mentions: > Instead of just seeing the entire palette and choosing, you need to say "I want a gold color", "lighter than that" and "darker" and "less like urine" etc. have meme status among designers, and not in a positive way. Some years ago, when I hanged out with a couple designers, I was introduced to Facebook groups exch…

In the deep past, I considered design to be an industry full of poets and philosophers. I suspect I got this impression from home improvement shows where they bring in an interior decorator to toss throw pillows around. Then I ended up working with three high quality designers in a row.

At this point I consider the design industry to be cousins or even siblings to the software engineering industry. All those incomprehensible design decisions that pop up in popular software don't come from designers debating faux marx or freud in coffee shops at 3am. They show up from management and other stakeholders who at the 11th hour decide that suddenly everything has to be flat because they read something in a magazine.

The bad decisions are fought by designers tooth and nail and the fact that anything looks halfway descent at all is due to their herculean efforts. If anything they deserve more sympathy than we do because we can always retreat into low level communication protocols or type theory when we need to get the muggles off our backs. But everyone has an opinion on how that button looks.

This comes to mind: https://theoatmeal.com/comics/design_hell

Re: Natural language is an unnatural interface

#82

Post by ericol »

parpfish wrote: post #79
ericol wrote: post #74

As opposed to what? I don't know about you, but for me on a personal level ChatGPT is a SERIOUS paradigm shift. I work on OSX, I use an app (MacGPT) that is always a keyboard shortcut away, and more often than not the responses are several times better than what Google will give me. I know there are a lot of areas with high friction, but those will go away eventually and compared to what I was doing before, the frict…

for an app like MacGPT, how much do you end up paying each month in API fees? I'd love to try it out, but I know that I'd constantly be second-guessing myself and saying "nah, i guess I don't really need to look that up" because I'd that pay-per-token API fee would always be sitting in the back of my mind.

TBH I have no idea [1] but I have quite a low limit of expenses using the API (10 bucks) and I'm yet to reach it.

Besides that, you can as well log in via web (That I haven't done, but it's possible)

As a side note, even thought the friction might be too high I couple it with the same programmer's MacWhisper, so that when I need a large text I just speak to the computer.

[1] Geez. I just checked, and my cumulative expenses for June are a staggering 0.27 USD

Re: Natural language is an unnatural interface

#84

Post by TeMPOraL »

parpfish wrote: post #78
TeMPOraL wrote: post #64

Earlier quoted context omitted.

> we were familiar with what could it do, and what were the “magic prompts” for achieving that. That's the thing though: in my system, there were no "magic prompts". What Speech API gave me, instead, is the ability to use "controlled language" - constrain the set of possible words at any given moment. That, and as a user, to train the living hell out of them in Windows settings. Yes, today's systems "are expected to…

i think by "magic prompts", what the previous post meant was that you knew all the possible commands (e.g., "Music, playlist alpha"). in theory, you could just look at a manpage for the speech api and know every keyword. there's no manpage for siri/alexa so you don't know what the commands are -- you just have to guess and when it works it supposedly "feels like magic"

And this is a mistake, IMHO. I mean, it sort of works with ChatGPT - now, and only somewhat reliably since past 3 months. It didn't work and doesn't work with voice assistants.

There is fun in exploration, in discovering new and useful or interesting functionality on your own. At least, when you're young and have ample free time for it. For adults... well, between blog posts and in-app examples, they gave us scattered map of the language anyway. Might have just compiled it into a reference guide from the start.

After all, those voice assistants still have a command grammar, similar to that of my system. Users end up having to learn that grammar anyway. Hiding the grammar, adding some fuzziness in command matching, and then putting an unconstrained voice-to-text engine in front... didn't really improve anything, and only made the problem much, much harder. A self-goal. And the only way it "feels like magic" is that it feels like your phone's being haunted by an angry poltergeist.

Re: Natural language is an unnatural interface

#85

Post by cubefox »

hbn wrote: post #80
cubefox wrote: post #66

Earlier quoted context omitted.

Current Voice assistants aren't much more than hardcoded GOFAI software. The only relevant ML involved is in the speech-to-text model. Modern language models are on a completely different level. Problem is that they need absurd amounts of VRAM, so running them locally on a phone is out of the question. > It isn’t at all clear what these interfaces can and cannot do, all you can do is keep trying different things unti…

> Problem is that they need absurd amounts of VRAM, so running them locally on a phone is out of the question. Weren't most voice assistants running on a server until recent years anyway?

Yeah. I'm pretty sure this will return. But this might cost a subscription, as language models are expensive.

Re: Natural language is an unnatural interface

#86

Post by TeMPOraL »

Verdex wrote: post #81
TeMPOraL wrote: post #40

Earlier quoted context omitted.

I don't think they see it the same way. At least not given that instructions in the style GP mentions: > Instead of just seeing the entire palette and choosing, you need to say "I want a gold color", "lighter than that" and "darker" and "less like urine" etc. have meme status among designers, and not in a positive way. Some years ago, when I hanged out with a couple designers, I was introduced to Facebook groups exch…

In the deep past, I considered design to be an industry full of poets and philosophers. I suspect I got this impression from home improvement shows where they bring in an interior decorator to toss throw pillows around. Then I ended up working with three high quality designers in a row. At this point I consider the design industry to be cousins or even siblings to the software engineering industry. All those incompre…

Yes, this matches what I heard and saw when hanging around the designers I mentioned.

BTW, quoting from the penultimate panel of that excellent Oatmeal piece (which drives home just how similar are the experiences of designers and programmers):

> You are no longer a web designer. You are now a mouse cursor inside a graphics program which the client can control by speaking, emailing and instant messaging.

This gains a new meaning, or at least becomes an interesting parallel, with LLMs in the picture. Many of us - myself included - already use GPT-4 as, paraphrasing, "a keyboard inside an editor program, which you can control by instant messaging". Ignoring that diffusion models can spit out parts of the design wholesale, someone is bound to eventually hook GPT-4 up to Photoshop or Gimp and get a graphics program you can drive by texting it.

... just remembered, I think someone already did that to Blender, made easy thanks to Blender being able to eat Python code and spit out 3D graphics.

Re: Natural language is an unnatural interface

#87

Post by Terr_ »

TeMPOraL wrote: post #86
Verdex wrote: post #81

Earlier quoted context omitted.

In the deep past, I considered design to be an industry full of poets and philosophers. I suspect I got this impression from home improvement shows where they bring in an interior decorator to toss throw pillows around. Then I ended up working with three high quality designers in a row. At this point I consider the design industry to be cousins or even siblings to the software engineering industry. All those incompre…

Yes, this matches what I heard and saw when hanging around the designers I mentioned. BTW, quoting from the penultimate panel of that excellent Oatmeal piece (which drives home just how similar are the experiences of designers and programmers): > You are no longer a web designer. You are now a mouse cursor inside a graphics program which the client can control by speaking, emailing and instant messaging. This gains a…

Tangentially, earlier talk a "text-box interface" made me think of Blender's "type the name of the immediate action you know should be possible but can't quickly find in hierarchical menus" box--a feature also present in some IDEs--and I'd like to emphasize that those things are (A) totally different than all this AI stuff and (B) generally awesome.

Re: Natural language is an unnatural interface

#88

Post by esafak »

grumbel wrote: post #45
Planktonne wrote: post #42

Earlier quoted context omitted.

> You ask it to show you a color picker, it generates HTML code for a color picker, you copy that into your browser and you can pick your color, which you can then copy&paste back into the LLM for further processing. This is slower, more awkward, and less efficient than just picking a colour from an existing colour picker.

The point is that nobody had to program this. Nobody had think up front "Will the user need a color picker?". Nobody had to find a spot in the UI to place it. You can just will it into existing as a user with nothing but the power of the LLM. No classic app has anywhere near that amount of expressiveness. Future versions of chatbots will of course have support for or similar to display this kind of stuff inline, that…

One duty of the programmer and product manager is to think about the likely uses of the program and to build a UI to enable it. If users wanted a blank slate they could write the program themselves, or have chatGPT write it.

Maximum expressiveness is not the goal, because it comes with a price. There is a balance to be struck between expressiveness, and economy of effort and cognition.

Re: Natural language is an unnatural interface

#89

Post by TeMPOraL »

Terr_ wrote: post #87
TeMPOraL wrote: post #86

Earlier quoted context omitted.

Yes, this matches what I heard and saw when hanging around the designers I mentioned. BTW, quoting from the penultimate panel of that excellent Oatmeal piece (which drives home just how similar are the experiences of designers and programmers): > You are no longer a web designer. You are now a mouse cursor inside a graphics program which the client can control by speaking, emailing and instant messaging. This gains a…

Tangentially, earlier talk a "text-box interface" made me think of Blender's "type the name of the immediate action you know should be possible but can't quickly find in hierarchical menus" box--a feature also present in some IDEs--and I'd like to emphasize that those things are (A) totally different than all this AI stuff and (B) generally awesome.

Agreed. Unlike the AI stuff with its "empty textbox" problem, fast incremental search is capital-A Awesome! Pretty much my favorite UI paradigm ever, at least out of those that gained adoption after I started using computers.

The best incremental search UIs are those that respond near-instantly, and have a stable list of candidates that is (or at least feels like) being filtered, and not like every keystroke re-runs some search from scratch. Prime example, which made me love this UI paradigm, is Foobar2000 - even back in early 2000s, I could have hundreds or thousands of entries in the music library, and then I would type into the magic textbox and watch that huge list (or tree) get instantly trimmed with each keystroke.

Re: Natural language is an unnatural interface

#90

Post by parpfish »

ericol wrote: post #82
parpfish wrote: post #79

Earlier quoted context omitted.

for an app like MacGPT, how much do you end up paying each month in API fees? I'd love to try it out, but I know that I'd constantly be second-guessing myself and saying "nah, i guess I don't really need to look that up" because I'd that pay-per-token API fee would always be sitting in the back of my mind.

TBH I have no idea [1] but I have quite a low limit of expenses using the API (10 bucks) and I'm yet to reach it. Besides that, you can as well log in via web (That I haven't done, but it's possible) As a side note, even thought the friction might be too high I couple it with the same programmer's MacWhisper, so that when I need a large text I just speak to the computer. [1] Geez. I just checked, and my cumulative ex…

Oof. By the end of the month that may climb all the way up to 0.30USD. /s
Post reply on HN