What will LLM-powered software look like in the medium-term future?

This space is surprisingly under-explored. Any idea in this space is basically new.

Ok. I appreciate they didn't use Medium as a blog post. There's a lot of good engineering knowledge currently locked behind the Medium "sign in" interface. It's dumb and stupid.

But just to voice an opinion, please kill the lengthy paragraph levels of fluff in blog posts. I don't want to say my opinion is influenced by shitty blog recipes and research papers, but it is. So stop it.

Just say what you need to say in bullet points at the beginning and fill in the details further on.

> I don’t believe that natural language is an adequate medium for conveying instructions with the precision required for many applications. Moreover, the “inefficiency” involved in carrying out many digital tasks “manually” is a core driver of the joy that comes with using computers (for me and, I assume, for at least some subset of others). Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes.

But searching for places to eat and recipes to make is very much not a precise search.

IMO the reason chat and not input text and get an answer is so powerful is that it allows messy search with iterative refinement - just like talking to an expert. Just "chat input, result given" doesn't have that.

I want to tell a recipe search I'm after a healthy light thing as it's hot. I want to get back options and then say "I don't really like cucumber though" and have it get rid of cucumber heavy recipes but leave in some salads and say "look this recipe has cucumber in but it'll be fine without it". Or "you asked for a chicken recipe but here's one with pork that should sub in just fine".

For restaurants I want to get back some options and tell it "that's too expensive, this is a quick thing" and get back more hole-in-the-wall things. Tell it "Hmm, something spicy sounds good but I've been to those Indian restaurants before, I'm after something new" and get a recommendation for the Ethiopian place in town.

> The current state of having a chat-ish UX that’s specific to each tool or website (e.g. a Documentation Chat on a library documentation page, a VSCode extension, a Google Bard integration, a really-badly-implemented chatbot in my banking app, etc.) doesn’t make any single one of those experiences more enjoyable, effective, or entertaining;

Coding chat in my editor absolutely makes coding more effective. I want heavy integration around how it edits text, not a general chat widget.

> The idealized role of persistence in LLM UX is also fairly obvious: it’s easy to imagine an LLM-powered experience that remembers and “understands” all my previous interactions with it, and uses that information to better help me with whatever my current task is

I sort of agree, but I absolutely detest hidden state about me. I should not alter my behaviour just because I'm worried about how that'll impact things. You see this regularly, I may avoid some weird youtube video (e.g. to see a weird flat earth take) because I don't really want the hassle of having loads of weird conspiracy stuff promoted or having to manually remove it from some list.

Having said that, recipe search that remembers I hate cucumbers is great.

I wonder if manually curated context will in general be better? Or maybe just for me.

> I’m interacting with UXes that can remember and utilize my previously expressed preferences, desires, goals, and information, using that underlying memory across different use cases seems like low-hanging fruit for drastically reducing friction.

The tricky part here is I switch contexts and don't want them bleeding into each other. My preferences for interaction around my kids and while at work are very different.

> A small-scale and developer-centric example: I use GitHub Copilot in VSCode, and I was recently implementing a library with documentation that featured LLM-powered Q&A, and it felt bizarre to have two LLM-mediated experiences open that each had exactly half the info needed to solve my problem.

I think here a split between data sources and frontends is key. This interaction is awkward, and should be combined (copilot should be able to reach out to that documentation).

> locally runnable and open models are no match for GPT-4 (etc.),

It's going to be more efficient to move compute to central places, the less they're used per person the more efficient it will be to have one central location process everyones things. A short taxi ride per week is cheaper than owning a car. However as uses grow (e.g. proactive llms), will this shift the equation towards locally runnable ones? A few queries a day and you're obviously better off not buying a h100. Constantly running things all day and if prices fall maybe that'll change.

I'm actively trying to scaffold an LLM for business usecases and my experience totally echoes this. I can tweak the prompt for a very representative set of training data, and then it vomits at a seemingly normal new case.

I'm not overly familiar with it but isn't this already a genre of gaming YouTuber?