Numbers every LLM developer should know
github.com
Numbers every LLM developer should know
1–10 of 107 posts
Re: Numbers every LLM developer should know
#2How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?
Re: Numbers every LLM developer should know
#3How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?
That is how I understood it, a token is on average a 3/4 of a word. "Token to word". So if you want to buy 1000 tokens you would get effectively 750 words.
Re: Numbers every LLM developer should know
#4I think that it would be helpful to add a fine-tuning costs for an open source model (think LLaMA to Alpaca).
From the phrasing around fine tuning right now it seems like it's using openai's fine tuning api to determine that cost, but it's not very clear.
Also this would be helpful for other foundation models if that doesn't already exist - how much VRAM to run Stable Diffusion v2.1 at different resolutions, running Whisper or Bark for audio, etc.
Re: Numbers every LLM developer should know
#5How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?
[deleted]
Re: Numbers every LLM developer should know
#6How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?
It's the token to word multiplier, yeah. i.e. x tokens = 0.75x words.
Re: Numbers every LLM developer should know
#7> There’s usually no need to go beyond 16-bit accuracy, and most of the time when you go to 8-bit accuracy there is too much loss of resolution.
I'm not sure this is accurate. From what I have seen, 8-bit quantization is usually fine, and even 4-bit is a viable tradeoff. Here are some benchmarks from TextSynth showing no significant degradation between 16 and 8 bit:
https://textsynth.com/technology.html
8-bit uses half as much memory and doubles the throughput for limited quality loss.
Re: Numbers every LLM developer should know
#8How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?
I think all the ratios given are x:1 and they tell you x.
Re: Numbers every LLM developer should know
#9Re: Numbers every LLM developer should know
#10> LLM Developer
This is the fastest I've rolled my eyes in a long time!