Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Numbers every LLM Developer should know

anyscale.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Numbers every LLM Developer should know

#11

Post by PeterStuer »

fullstackchris wrote: post #6

I'm curious about the point on the embedding lookup cost... in my experience for an embedding lookup to be accurate, you have to include your entire document dataset to be queried against... obviously this can be just as expensive as querying a full cloud model if your dataset is very large. Interested if anyone had thoughts about this.

Yes. I think the point is that the price per token for creating the embeddings using e.g. OpenAI's text-embedding-ada-002 api might be low, this will add up to some significant cost for a large document corpus. The suggestion to roll your own based on freely available embedding models is sound IMHO.

Now how to chunk those documents into semantically coherent pieces for context retrieval, that is the real challange though.

Re: Numbers every LLM Developer should know

#13

Post by luckyt »

vikp wrote: post #5

I clicked because I thought they were defining LLM developer as "someone training LLMs", but instead they define it as "someone integrating LLMs into their application". If you also had the same initial thought as me, this is an excellent article - https://blog.eleuther.ai/transformer-math/ .

I had the same thought, but overall, there is probably an order of magnitude more people using LLMs in applications or fine-tuning them compared to those trying to pretrain LLMs from scratch.

Re: Numbers every LLM Developer should know

#14

Post by Roark66 »

The main thing every LLM should know is that ARM will eat x86_64's lunch in ML. Why? Because of the shared/unified memory model. M2 Ultra from apple can use up to 192GB of RAM. Even your smartphone thanks to this model can run networks a lot bigger than you would expect.

Re: Numbers every LLM Developer should know

#16

Post by Havoc »

Don't think I've ever heard anyone call it "GRAM" instead of VRAM.

Another cost saving tip: On API, do combo calls where possible to dual use the input tokens. e.g.

"""You are an AI assistant that summarizes text given.

After the summarized text, add the word END.

After that answer the following questions with Yes or NO:

Is the text about Donald Trump?

Is the text about Space? """

Down side is now you need code to parse the output pieces & error handling around that

Re: Numbers every LLM Developer should know

#17

Post by swores »

Havoc wrote: post #16

Don't think I've ever heard anyone call it "GRAM" instead of VRAM. Another cost saving tip: On API, do combo calls where possible to dual use the input tokens. e.g. """You are an AI assistant that summarizes text given. After the summarized text, add the word END. After that answer the following questions with Yes or NO: Is the text about Donald Trump? Is the text about Space? """ Down side is now you need code to pa…

I've found - when using ChatGPT with GPT4 - that sometimes when I ask it to do two things like that it will ignore my request to do one before the other and try to do both at the same time before providing a combined answer, unless I give even more specific instructions along the lines of "do not until after you have entirely finished and answered to completion ".

Just FYI in case anyone reading your comment tries your suggestion and has same issue, that with more firm instructions the problem can be avoided. Though I've not felt the need to experiment enough to understand exactly where the line is to avoid it trying to start one task too early without being wastefully verbose in the prompt.

Re: Numbers every LLM Developer should know

#18

Post by golergka »

swores wrote: post #17
Havoc wrote: post #16

Don't think I've ever heard anyone call it "GRAM" instead of VRAM. Another cost saving tip: On API, do combo calls where possible to dual use the input tokens. e.g. """You are an AI assistant that summarizes text given. After the summarized text, add the word END. After that answer the following questions with Yes or NO: Is the text about Donald Trump? Is the text about Space? """ Down side is now you need code to pa…

I've found - when using ChatGPT with GPT4 - that sometimes when I ask it to do two things like that it will ignore my request to do one before the other and try to do both at the same time before providing a combined answer, unless I give even more specific instructions along the lines of "do not until after you have entirely finished and answered to completion ". Just FYI in case anyone reading your comment tries yo…

Works much better with function calling when you just give it one function it has to call with two non optional arguments.

Re: Numbers every LLM Developer should know

#19

Post by bhickey »

hiddencost wrote: post #3

This is honestly a bit gross, as it's just a marketing piece. The original numbers every programmer should know is a profound piece of pedagogy, aimed at helping programmers be better at their craft. This is just an excerpt from a pitch deck.

Flag it and move on.

Re: Numbers every LLM Developer should know

#20

Post by fweimer »

Roark66 wrote: post #14

The main thing every LLM should know is that ARM will eat x86_64's lunch in ML. Why? Because of the shared/unified memory model. M2 Ultra from apple can use up to 192GB of RAM. Even your smartphone thanks to this model can run networks a lot bigger than you would expect.

AMD is reportedly working on APUs with decent amounts of HBM directly on the chip. It's not an x86-64 vs AArch64 thing.
Post reply on HN