Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

11–20 of 148 posts

Re: What we've learned from a year of building with LLMs

#11

Post by CuriouslyC »

solidasparagus wrote: post #2

No offense, but I'd love to see what they've successfully built using LLMs before taking their advice too seriously. The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

Fine tuning has been on the way out for a while. It's hard to do right and costly. LoRAs are better for influencing output style as they don't dumb down the model, and they're easier to create. This is on top of RAG just being better for new facts like the other reply mentioned.

Re: What we've learned from a year of building with LLMs

#12

Post by mentos »

Multicomp wrote: post #8

Anyone have a convenience solution for doing multi-step workflows? For example, I'm filling out the basics of an NPC character sheet on my game prep. I'm using a certain rule system, give the enemy certain tactics, certain stats, certain types of weapons, right now I have a 'god prompt' trying to walk the LLM through creating the basic character sheet, but the responses get squeezed down into what one or two prompt r…

I still haven’t played with using one LLM to oversee another.

“You are in charge of game prep and must work with an LLM over many prompts to…”

Re: What we've learned from a year of building with LLMs

#13

Post by hugocbp »

Multicomp wrote: post #8

Anyone have a convenience solution for doing multi-step workflows? For example, I'm filling out the basics of an NPC character sheet on my game prep. I'm using a certain rule system, give the enemy certain tactics, certain stats, certain types of weapons, right now I have a 'god prompt' trying to walk the LLM through creating the basic character sheet, but the responses get squeezed down into what one or two prompt r…

For me, a very simple "breakdown tasks into a queue and store in a DB" solution has help tremendously with most requests.

Instead of trying to do everything into a single chat or chain, add steps to ask the LLM to break down the next tasks, with context, and store that into SQLite or something. Then start new chats/chains on each of those tasks.

Then just loop them back into LLM.

I find that long chats or chains just confuse most models and we start seeing gibberish.

Right now I'm favoring something like:

"We're going to do task {task}. The current situation and context is {context}.

Break down what individual steps we need to perform to achieve {goal} and output these steps with their necessary context as {standard_task_json}. If the output is already enough to satisfy {goal}, just output the result as text."

I find that leaving everything to LLM in a sequence is not as effective as using LLM to break things down and having a DB and code logic to support the development of more complex outcomes.

Re: What we've learned from a year of building with LLMs

#14

Post by OutOfHere »

solidasparagus wrote: post #2

No offense, but I'd love to see what they've successfully built using LLMs before taking their advice too seriously. The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

Fine-tuning is an absolutely necessary for true AI, and even if it's desirable, it's unfeasible to do for now for any large model considering how expensive GPUs are. If I had infinite money, I'd throw it at continuous fine-tuning and would throw away the RAG. Fine-tuning also requires appropriate measures to prevent forgetting of older concepts.

Re: What we've learned from a year of building with LLMs

#15

Post by gpsx »

Multicomp wrote: post #8

Anyone have a convenience solution for doing multi-step workflows? For example, I'm filling out the basics of an NPC character sheet on my game prep. I'm using a certain rule system, give the enemy certain tactics, certain stats, certain types of weapons, right now I have a 'god prompt' trying to walk the LLM through creating the basic character sheet, but the responses get squeezed down into what one or two prompt r…

One option for doing this is to incrementally build up the "document" using isolated prompts for each section. I say document because I am not exactly sure what the character sheet looks like, but I am assuming it can be constructed one section at a time. You create a prompt to create the first section. Then, you create a second prompt that gives the agent your existing document and prompts it to create the next section. You continue until all the sections are finished. In some cases this works better than doing a single conversation.

Re: What we've learned from a year of building with LLMs

#16

Post by [deleted] »

CuriouslyC wrote: post #9
Multicomp wrote: post #8

Anyone have a convenience solution for doing multi-step workflows? For example, I'm filling out the basics of an NPC character sheet on my game prep. I'm using a certain rule system, give the enemy certain tactics, certain stats, certain types of weapons, right now I have a 'god prompt' trying to walk the LLM through creating the basic character sheet, but the responses get squeezed down into what one or two prompt r…

You can do multi shot workflows pretty easy, I like to have the model produce markdown, then add code blocks (```json/yaml```) to extract the interim results. You can lay out multiple "phases" in your prompt and have it perform each one in turn, and have each one reference prior phases. Then at the end you just pull out the code blocks for each phase and you have your structured result.

[deleted]

Re: What we've learned from a year of building with LLMs

#18

Post by solidasparagus »

OutOfHere wrote: post #14
solidasparagus wrote: post #2

No offense, but I'd love to see what they've successfully built using LLMs before taking their advice too seriously. The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

Fine-tuning is an absolutely necessary for true AI, and even if it's desirable, it's unfeasible to do for now for any large model considering how expensive GPUs are. If I had infinite money, I'd throw it at continuous fine-tuning and would throw away the RAG. Fine-tuning also requires appropriate measures to prevent forgetting of older concepts.

It is not unfeasible. It is absolutely realistic to do distributed finetuning of an 8B text model on previous generation hardware. You can add finetuning to your set of options for about the cost of one FTE - up to you whether that tradeoff is worth it, but in many places it is. The expertise to pull it off is expensive, but to get a mid-level AI SME capable of helping a company adopt finetuning, you are only going to pay about the equivalent of 1-3 senior engineers.

Expensive? Sure, all of AI is crazy expensive. Unfeasible? No

Re: What we've learned from a year of building with LLMs

#19

Post by proc0 »

Multicomp wrote: post #8

Anyone have a convenience solution for doing multi-step workflows? For example, I'm filling out the basics of an NPC character sheet on my game prep. I'm using a certain rule system, give the enemy certain tactics, certain stats, certain types of weapons, right now I have a 'god prompt' trying to walk the LLM through creating the basic character sheet, but the responses get squeezed down into what one or two prompt r…

Sounds like you need an agent system, some libs are mentioned here: https://lilianweng.github.io/posts/2023-06-23-agent/

Re: What we've learned from a year of building with LLMs

#20

Post by solidasparagus »

CuriouslyC wrote: post #11
solidasparagus wrote: post #2

No offense, but I'd love to see what they've successfully built using LLMs before taking their advice too seriously. The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

Fine tuning has been on the way out for a while. It's hard to do right and costly. LoRAs are better for influencing output style as they don't dumb down the model, and they're easier to create. This is on top of RAG just being better for new facts like the other reply mentioned.

How much of that is just the flood of traditional engineers into the space and the fact that collecting data and then fine-tuning models is orders of magnitude more complex than just throwing in RAG? I suspect a huge amount of RAG's popularity is just that any engineer can do a version of it + ChatGPT API calls in a day.

As for lora - in the context of my comment, that's just splitting hairs IMO. It falls in the category of finetuning for me, although I understand why you might disagree. But it's not like the article mentions lora either, nor am I aware of people doing lora without GPUs which the article is against (No GPUs before PMF)

Post reply on HN