Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

121–130 of 148 posts

Re: What we've learned from a year of building with LLMs

#121

Post by idf00 »

threeseed wrote: post #49
fnordpiglet wrote: post #41

Earlier quoted context omitted.

We use LLMs in dozens of different production applications for critical business flows. They allow for a lot of dynamism in our flows that aren’t amenable to direct quantitative reasoning or structured workflows. Double digit percents of our growth in the last year are entirely due to them. The biggest challenge is tool chain, limits on inference capacity, and developer understanding of the abilities, limits, and tec…

No one is saying that all of AI is hype. It clearly isn't. But the facts are that today LLMs are not suitable for use cases that need accurate results. And there is no evidence or research that suggests this is changing anytime soon. Maybe for ever. There are very strong parallels to crypto in that (a) people are starting with the technology and trying to find problems and (b) there is a cult like atmosphere where no…

I didn't see anything in the article that indicated the authors believed that those who don't see use cases for LLMs are anti-progress or anti-technology. Is that comment related to the authors of this article, or just a general grievance you have unrelated to this article?

Re: What we've learned from a year of building with LLMs

#122

Post by exhaze »

jakubmazanec wrote: post #119
qeternity wrote: post #113

Earlier quoted context omitted.

I think you're going about it backwards. You don't take a tool, and then try to figure out what to do with it. You take a problem, and then figure out which tool you can use to solve it.

But it seems to me that's what they're doing: "We have LLMs, what to do with them?" But anyway, I'm seriously just looking for an example of app that is build with stuff described in the article. Me personally, I only used LLM for one "serious" application: I used GPT-3.5Turbo for transforming unstructured text into JSON; it was basically just ad-hoc Node.js script that called API (prompt was few examples of input-ou…

https://github.com/hrishioa/lumentis

Since you seem to have not noticed my comment above, here's another example of a project that implements many of these techniques. Me and many others have used this to transcribe hour long videos into a well organized "docs site" that makes the content easy to read.

Example: https://matadoc.vercel.app/

This was completely auto-generated in a few minutes. The author of the library reviewed it and said that it's nearly 100% correct and people in the company where it was built rely on these docs.

Tell me how long it would take you to write these docs. I'm really confused where your dismissive mentality is coming from in the face of what I think is overwhelming evidence to the contrary. I'm happy to provide example after example after example. I'm sorry, but you are utterly, completely wrong in your conclusions.

Re: What we've learned from a year of building with LLMs

#123

Post by phillipcarter »

mattmanser wrote: post #81
hamelsmu wrote: post #64

Earlier quoted context omitted.

Hi, Hamel here. I'm one of the co-authors. I'm an independent consultant and not all clients allow me to talk about their work. However, I have two that do, which I've discussed in the article. These are two production use cases that I have supported (which again, are explicitly mentioned in the article): 1. https://www.honeycomb.io/blog/introducing-query-assistant 2. https://www.youtube.com/watch?v=B_DMMlDuJB0 Other…

You've linked to a query generator for a custom programming language and a 1 hour video about LLM tools. The cynic in me feels like the former could probably be done by chatgpt off the shelf. But those do not seem to be real world business cases. Can you expand a bit more why you think they are? We don't have hours to spend reading, and you say you've been allowed to talk about them. So can you summarise the business…

> The cynic in me feels like the former could probably be done by chatgpt off the shelf.

Hello! I'm the owner of the feature in question who experimented with chatgpt last year in the course of building the feature (and working with Hamel to improve it via fine-tuning later).

Even today, it could not work with ChatGPT. To generate valid queries, you need to know which subset of a user's dataset schema is relevant to their query, which makes it equally a retrieval problem as it does a generation problem.

Beyond that, though, the details of "what makes a good query" are quite tricky and subtle. Honeycomb as a querying tool is unique in the market because it lets you arbitrarily group and filter by any column/value in your schema without pre-indexing and without any cost w.r.t. cardinality. And so there are many cases where you can quite literally answer someone's question, but there are multitudes of ways you can be even more helpful, often by introducing a grouping that they didn't directly ask for. For example, "count my errors" is just a COUNT where the error column exists, but if you group by something like the HTTP route, the name of the operation, etc. -- or the name of a child operation and its calling HTTP route for requests -- you end up actually showing people where and how these errors come from. In my experience, the large majority of power users already do this themselves (it's how you use HNY effectively), and the large majority of new users who know little about the tool simply have no idea it's this flexible. Query Assistant helps them with that and they have a pretty good activation rate when they use it.

Unfortunately, ChatGPT and even just good old fashioned RAG is often not up to the task. That's why fine-tuning is so important for this use case.

Re: What we've learned from a year of building with LLMs

#124

Post by e1g »

gpsx wrote: post #15
Multicomp wrote: post #8

Anyone have a convenience solution for doing multi-step workflows? For example, I'm filling out the basics of an NPC character sheet on my game prep. I'm using a certain rule system, give the enemy certain tactics, certain stats, certain types of weapons, right now I have a 'god prompt' trying to walk the LLM through creating the basic character sheet, but the responses get squeezed down into what one or two prompt r…

One option for doing this is to incrementally build up the "document" using isolated prompts for each section. I say document because I am not exactly sure what the character sheet looks like, but I am assuming it can be constructed one section at a time. You create a prompt to create the first section. Then, you create a second prompt that gives the agent your existing document and prompts it to create the next sect…

Perplexity recently released something like this https://www.perplexity.ai/hub/blog/perplexity-pages

Re: What we've learned from a year of building with LLMs

#126

Post by jakubmazanec »

exhaze wrote: post #122
jakubmazanec wrote: post #119

Earlier quoted context omitted.

But it seems to me that's what they're doing: "We have LLMs, what to do with them?" But anyway, I'm seriously just looking for an example of app that is build with stuff described in the article. Me personally, I only used LLM for one "serious" application: I used GPT-3.5Turbo for transforming unstructured text into JSON; it was basically just ad-hoc Node.js script that called API (prompt was few examples of input-ou…

https://github.com/hrishioa/lumentis Since you seem to have not noticed my comment above, here's another example of a project that implements many of these techniques. Me and many others have used this to transcribe hour long videos into a well organized "docs site" that makes the content easy to read. Example: https://matadoc.vercel.app/ This was completely auto-generated in a few minutes. The author of the library…

But that seems to belong to the category "text transformation" (e.g. translating, converting unstructed notes into structured data, etc.), which I acknowledge LLMs are good at; instead of category "I'll magically debug your SQL wish!".

Re: What we've learned from a year of building with LLMs

#127

Post by janpieterz »

jncfhnb wrote: post #93
janpieterz wrote: post #58

Earlier quoted context omitted.

How would I do this reliably? Eg give me 10 different values, all in one prompt for performance reasons? Might not need JSON but whatever format it outputs, it needs to be reliable.

Don’t do it all in one prompt.

Right, but now I’m basically running a huge performance hit, need to parallelize my queries etc.

I was parsing a document recently, 10-ish questions for 1 document, would make things expensive.

Might be what’s needed but not ideal.

Re: What we've learned from a year of building with LLMs

#128

Post by robbiep »

exhaze wrote: post #117
jakubmazanec wrote: post #111

I'm not saying the content of the article is wrong, but what apps are people/companies writing articles like this actually building? I'm seriously unable to imagine any useful app. I only use GPT via API (as better Google for documentations, and its output is never usable without heavy editing). This week I tried to use "AI" in Notion: I needed to generate 84 check boxes for each day starting with specific date. I go…

I've built many production applications using a lot of these techniques and others - it's made money either by increasing sales or decreasing operational costs. Here's a more dramatic example: https://www.grey-wing.com/ This company provides deeply integrated LLM-powered software for operating freight ships. There are a lot of people who are doing this and achieving very good results. Sorry, if it's not working for y…

That’s really interesting. Surely the crewing roster stuff is actually using linear algebra rather than AI though?

Re: What we've learned from a year of building with LLMs

#129

Post by gengstrand »

Interesting blog. It seems to be a compendium of advice for all kinds of folks ranging from end user to integration partner. For a slightly different take on how to use LLMs to build software, you might be interested in https://www.infoq.com/articles/llm-productivity-experiment/ which documents an experiment where the same prompt was given to various prominent LLMs asking to write two unit tests for an already existing code base. The results were collected, metrics were analyzed, then comparisons were made. No advice on how to write better prompts but some insight on how to work with and what you can expect from LLMs in order to improve developer productivity.

Re: What we've learned from a year of building with LLMs

#130

Post by solidasparagus »

altdataseller wrote: post #104
solidasparagus wrote: post #56

Earlier quoted context omitted.

Continuous retraining and deployment maybe? But I'm actually not anti-RAG (although I think it is overrated because the retrieval problem is still handled extremely naively), I just think that fine-tuning should also be in your toolkit.

Why is the retrieval part overrated? There isnt even a single way to retrieve. It could be a simple keyword sesrch, a vector sesrch, a combo, or just simply retrieving a single doc and stuffing it in the context

People will disagree, but my problem with retrieval is that every technique that is popular uses one-hop thinking - you retrieve information that is directly related to the prompt using old-school techniques (even though the embeddings are new, text similarity is old). LLMs are most powerful, IMO, at horizontal thinking. Building a prompt using one-hop narrow AI techniques and then feeding it into a powerful generally capable model is like building a drone but only letting it fly over streets that already exist - not worthless, but only using a fraction of the technology's power.

A concrete example is something like a tool for querying an internal company wiki and the query "tell me about the Backend Team's approach to sprint planning". Normal retrieval approaches will pull information directly related to that query. But what if there is no information about the backend team's practices? As a human, you would do multi-hop/horizontal information extraction - you would retrieve information about who makes up the backend team, you would then retrieve information about them and their backgrounds/practices. You might might have a hypothesis that people carry over their practices from previous experiences, so you look at the previous teams and their practices. Then you would have the context necessary to give a good answer. I don't know of many people implementing RAG like that. And what I described is 100% possible for AI to do today.

Techniques that would get around this like iterative retrieval or retrieval-as-a-tool don't seem popular.

Post reply on HN