Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are you actually using LLMs for in production?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: Ask HN: What are you actually using LLMs for in production?

#51

Post by BowBun »

captainbland wrote: post #25
petercooper wrote: post #16

Analyzing firehoses of data. RSS feeds, releases, stuff like that. My job involves curating information and while I still do that process by hand, LLMs make my net larger and help me find more signals. This means hallucinations or mistakes aren't a big deal, since it all ends up with me anyway. I'm quite bullish on using LLMs as extra eyes, rather than as extra hands where they can run into trouble.

Is cost a major consideration for you here? Like if you're dealing with firehose data which I'm assuming is fairly high throughput, do you see an incentive for potentially switching to a more specific NLP classifier model rather than sticking with generative LLMs? Or is it that this is good enough/the ROI of switching isn't attractive? Or is the generative aspect adding something else here?

I don't think everyone's using the term 'firehose' the same here. A child comment refers to half a billion tokens for $20.

I did some really basic napkin math with some Rails logs. One request with some extra junk in it was about 400 tokens according to the OpenAI tokenizer[0]. 500M/400 = ~1.25 million log lines.

Paying linearly for logs at $20 per 1.25 million lines is not reasonable for mid-to-high scale tech environments.

I think this would be sufficient if a 'firehose of data' is a bunch of news/media/content feeds that needs to be summarized/parsed/guessed at.

[0] https://platform.openai.com/tokenizer

Re: Ask HN: What are you actually using LLMs for in production?

#52

Post by notjoemama »

binarymax wrote: post #2

So many things. I have built several customer facing products, a web research platform that works better than the RAG you get from Google, and lots of small tools. For example, I wrote a recent blog post on how I use LLMs to generate excel files with a prompt (less about the actual product and more about how to improve outcomes): https://maxirwin.com/articles/persona-enriched-prompting/

Thank you for the link! That was a nice read through. I'm just familiarizing myself with using AI in software development and this gives me some structure around how to scaffold up a domain knowledge response. Very cool.

Re: Ask HN: What are you actually using LLMs for in production?

#53

Post by diggan »

Vegenoid wrote: post #29
actinium226 wrote: post #3

We have a prompt that takes a job description and categorizes it based on whether it's an individual contributor role, manager, leadership, or executive, and also tags it based on whether it's software, mechanical, etc. We scrape job sites and use that prompt to create tags which are then searchable by users in our interface. It was a bit surprising to see how Karpathy described software 3.0 in his recent presentatio…

Can you elaborate on what makes this “software 3.0”? I didn’t really understand what the distinction was in Karpathy’s talk, and felt like I needed a more concrete example. What you describe sounds cool, but I still feel like I’m not understanding what makes it “3.0”. I’m not trying to criticize, I really am trying to understand this concept.

> Can you elaborate on what makes this “software 3.0”?

Software 2.0: We need to parse a bunch of different job ads. We'll have a rule engine, decide based on keywords what to return, do some filtering, maybe even semantic similarity to descriptions we know match with a certain position, and so on

Software 3.0: We need to parse a bunch of different job ads. Create a system prompt that says "You are a job description parser. Based on the user message, return a JSON structure with title, description, salary-range, company, position, experience-level" and etc, pass it the JSON schema of the structure you want and you have a parser that is slow, sometimes incorrect but (most likely) covers much broader range than your Software 2.0 parser.

Of course, this is wildly simplified and doesn't include everything, but that's the difference Karpathy is trying to highlight. Instead of programming those rules for the parser ourselves, you "program" the LLM via prompts to do that thing.

Re: Ask HN: What are you actually using LLMs for in production?

#54

Post by on_the_train »

polishdude20 wrote: post #22
yamalight wrote: post #10

Built vaporlens.app in my free time using LLMs (specifically gemini, first 2.0-flash, recently moved to 2.5-flash). It processes Steam game reviews and provides one page summary of what people thing about the game. Have been gradually improving it and adding some features from community feedback. Has been good fun.

I usually find that if a game is rated overwhelmingly positive, I'm gonna like it. The moment it's just mostly positive, it doesn't stay as a favorite for me.

That rating is not (just) a function of positive to negative ratio. Small number of reviews (ie small games) can't reach that rating although they might be equally well received.

Re: Ask HN: What are you actually using LLMs for in production?

#55

Post by on_the_train »

impure wrote: post #26

Pretty much all of my productivity apps has LLM integration now. My language learning app uses them to break down phrases and get detailed definitions. My RSS app generates summaries. And recently I released an email app that's like Google Inbox in that it uses bundles. It also summarizes emails and extracts expiry and due dates.

And all your users hate it

Re: Ask HN: What are you actually using LLMs for in production?

#56

Post by jackthetab »

Which LLMs and plans are you guys using for all of these cool ideas?

ATM I use ChatGPT Plus for everything except coding inside my Jetbrains IDEs.

I'm starting to look around at other LLMs for non-coding purposes (brainstorming, docs, being a project manager, summarizing, learning new subjects, etc.).

Re: Ask HN: What are you actually using LLMs for in production?

#57

Post by Vegenoid »

diggan wrote: post #53
Vegenoid wrote: post #29

Earlier quoted context omitted.

Can you elaborate on what makes this “software 3.0”? I didn’t really understand what the distinction was in Karpathy’s talk, and felt like I needed a more concrete example. What you describe sounds cool, but I still feel like I’m not understanding what makes it “3.0”. I’m not trying to criticize, I really am trying to understand this concept.

> Can you elaborate on what makes this “software 3.0”? Software 2.0: We need to parse a bunch of different job ads. We'll have a rule engine, decide based on keywords what to return, do some filtering, maybe even semantic similarity to descriptions we know match with a certain position, and so on Software 3.0: We need to parse a bunch of different job ads. Create a system prompt that says "You are a job description p…

Thank you for the explanation, I appreciate it.

Re: Ask HN: What are you actually using LLMs for in production?

#58

Post by petercooper »

captainbland wrote: post #25
petercooper wrote: post #16

Analyzing firehoses of data. RSS feeds, releases, stuff like that. My job involves curating information and while I still do that process by hand, LLMs make my net larger and help me find more signals. This means hallucinations or mistakes aren't a big deal, since it all ends up with me anyway. I'm quite bullish on using LLMs as extra eyes, rather than as extra hands where they can run into trouble.

Is cost a major consideration for you here? Like if you're dealing with firehose data which I'm assuming is fairly high throughput, do you see an incentive for potentially switching to a more specific NLP classifier model rather than sticking with generative LLMs? Or is it that this is good enough/the ROI of switching isn't attractive? Or is the generative aspect adding something else here?

No. It's a tiny expense. I mostly use GPT 4.1 Mini for what I'm doing as it's the best balance between results and cost, but Gemini Flash can do the job just as well for a little less if I need it.

As other commenters have mentioned, a firehose can mean many things. For me it might be thousands of different reasonably small things a day which is dollars a day even in the worst case. If you were processing the raw X feed or the whole of Reddit or something, then all of your questions certainly become more relevant :-)

Re: Ask HN: What are you actually using LLMs for in production?

#59

Post by muzani »

polishdude20 wrote: post #22
yamalight wrote: post #10

Built vaporlens.app in my free time using LLMs (specifically gemini, first 2.0-flash, recently moved to 2.5-flash). It processes Steam game reviews and provides one page summary of what people thing about the game. Have been gradually improving it and adding some features from community feedback. Has been good fun.

I usually find that if a game is rated overwhelmingly positive, I'm gonna like it. The moment it's just mostly positive, it doesn't stay as a favorite for me.

There's plenty I don't like, like Factorio. It's not bad enough to downvote, but not likeable enough to play.

However, review positivity is usually the best indicator of sales - it's so accurate that there's algorithms that rely entirely on it.

Re: Ask HN: What are you actually using LLMs for in production?

#60

Post by captainbland »

petercooper wrote: post #58
captainbland wrote: post #25

Earlier quoted context omitted.

Is cost a major consideration for you here? Like if you're dealing with firehose data which I'm assuming is fairly high throughput, do you see an incentive for potentially switching to a more specific NLP classifier model rather than sticking with generative LLMs? Or is it that this is good enough/the ROI of switching isn't attractive? Or is the generative aspect adding something else here?

No. It's a tiny expense. I mostly use GPT 4.1 Mini for what I'm doing as it's the best balance between results and cost, but Gemini Flash can do the job just as well for a little less if I need it. As other commenters have mentioned, a firehose can mean many things. For me it might be thousands of different reasonably small things a day which is dollars a day even in the worst case. If you were processing the raw X f…

Yeah that makes sense based on those specifics, thanks
Post reply on HN