Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyLlama: An Open-Source Small Language Model

arxiv.org

Post reply on HN
Print view View original

21–30 of 46 posts

Re: TinyLlama: An Open-Source Small Language Model

#21

Post by sp332 »

rnd0 wrote: post #4
andy99 wrote: post #3

I've been using one of the earlier checkpoints for benchmarking a Llama implementation. Completely anecdotally I feel at least as good or better about this one than the earlier openllama 3B. I wouldn't use either of them for RAG or anything requiring more power, just to say that it's competitive as a smaller model, whatever you use those for, and easy to run on CPU at FP16 (meaning without serious quantization).

>I wouldn't use either of them for RAG What's RAG?

Since the models have a limited context size, you pre-process a bunch of data that might be related to the task (documentation, say) and generate a semantic vector for each piece. The when you ask a question, look up just the few pieces that are semantically most simlar and load them into the context along with the question. Then the LLM can generate a new answer with the most relevant pieces of data.

Re: TinyLlama: An Open-Source Small Language Model

#22

Post by WhitneyLand »

minimaxir wrote: post #13

It was fun to follow the public TinyLlama loss curves in near real-time, although it showed that it can be frustrating since the loss curves barely moved down even after an extra trillion tokens: https://wandb.ai/lance777/lightning_logs/reports/metric-trai... (note the log-scaled X-axis) But they did move down and that's what's important. There should probably be more aggressive learning rate annealing for models try…

More aggressive learning rate? Wouldn’t we need more information on why they decided to stop training at this point to conclude that?

My current understanding of the story is, to recap:

- First the game was increase model size massively

- For example GPT3 had 175B parameters, but less than 0.5T tokens of training data

- Then Chinchilla showed for a given compute budget we can scale better by increasing training data

- Now we have models like this, and Phi, that have over 1T trained tokens

For any model, the loss curve going down could mean it’s learning, or could mean it’s overfitting, we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before.

So getting back to your comment, I thought they were actually a multitude of indicators that should be used, not just validation loss, to determine what would be gained with more training.

Re: TinyLlama: An Open-Source Small Language Model

#23

Post by fergal_reid »

andy99 wrote: post #16
sroussey wrote: post #11

Earlier quoted context omitted.

What is good for RAG?

The smallest model your users agree meets their needs. It really depends. The retrieval part is way more important. I've used the original 13B instruction tuned llama2, quantized, and found it gives coherent answers about the context provided, ie the bottleneck was mostly getting good context. When I played with long context models (like 16k tokens, and this was a few months ago, maybe they improved) they sucked.

>The retrieval part is way more important.

I don't agree with this - at Intercom we've put a lot of work into our Fin chatbot, which uses a RAG architecture, and we're still using GPT-4 for the generation part.

GPT-4 is a really powerful and expensive model but we find we need this power to 1) reduce hallucinations acceptably, and 2) keep the quality of inferences made using the retrieved text high.

Now, our bot is answering customer support questions unsupervised - maybe it'd be different for a human in the loop system - but at least in our case, we feel we need a very powerful generation model to reduce errors, even after having benchmarked this thoroughly.

We've also done work on the retrieval end of things, including a customised model, but found the generation side is where we need the most capable models.

Re: TinyLlama: An Open-Source Small Language Model

#24

Post by minimaxir »

WhitneyLand wrote: post #22
minimaxir wrote: post #13

It was fun to follow the public TinyLlama loss curves in near real-time, although it showed that it can be frustrating since the loss curves barely moved down even after an extra trillion tokens: https://wandb.ai/lance777/lightning_logs/reports/metric-trai... (note the log-scaled X-axis) But they did move down and that's what's important. There should probably be more aggressive learning rate annealing for models try…

More aggressive learning rate? Wouldn’t we need more information on why they decided to stop training at this point to conclude that? My current understanding of the story is, to recap: - First the game was increase model size massively - For example GPT3 had 175B parameters, but less than 0.5T tokens of training data - Then Chinchilla showed for a given compute budget we can scale better by increasing training data…

> Wouldn’t we need more information on why they decided to stop training at this point to conclude that?

The experiment was fixed at 3 epochs on 1T tokens, they didn't decide to "stop" at a given criterion.

> we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before.

The data I linked shows the validation loss, which has the same behavior as the training loss.

Re: TinyLlama: An Open-Source Small Language Model

#25

Post by refulgentis »

andy99 wrote: post #18
andy99 wrote: post #3

I've been using one of the earlier checkpoints for benchmarking a Llama implementation. Completely anecdotally I feel at least as good or better about this one than the earlier openllama 3B. I wouldn't use either of them for RAG or anything requiring more power, just to say that it's competitive as a smaller model, whatever you use those for, and easy to run on CPU at FP16 (meaning without serious quantization).

Also, I should promote the code I wrote for running this. It runs models in ggml format, the one I made available is an older checkpoint though. It's easy to convert the newer one. And it's in Fortran but it should be easy to get gfortran if you don't have it installed. https://github.com/rbitr/llm.f90/tree/optimize16/purefortran

Man I didn't recognize your username but once you said Fortran I recognized you immediately. You are an inspiration, an example of true software engineering vis a vis what in day to day becomes what you can hire for.

Edit: you have some rare knowledge, I'm curious if you have any thoughts on small models good enough for RAG. Mistral 7B is in my testing buts it's laughably slow and 7B is just too much for mobile, both iOS and Android get crashy. (4 tkns/s on Pixel Fold, similar on iOS). Similar problems on web from a good-enough 2 year old i7.

I'd try Phi-2 but I want to charge for my app and the non-commercial usage license bars that. (all these hours building ain't free! And I can't responsibly give search away, scraping locally is too risky for the user, and the free search API I know of has laudable goals, but ultimately, is "trust me bro" as far as privacy goes)

I'm starting to think we might not get an open, RAG capable model sub 7B without a concerted open source effort. Stabilitys distracted and spread thin, MS is all in on AI PCs(tm), and it's too commercially valuable for the big boys to give away

Re: TinyLlama: An Open-Source Small Language Model

#26

Post by minimaxir »

ssivark wrote: post #20
minimaxir wrote: post #13

It was fun to follow the public TinyLlama loss curves in near real-time, although it showed that it can be frustrating since the loss curves barely moved down even after an extra trillion tokens: https://wandb.ai/lance777/lightning_logs/reports/metric-trai... (note the log-scaled X-axis) But they did move down and that's what's important. There should probably be more aggressive learning rate annealing for models try…

How crucial is it to freeze the learning rate schedule a priori, instead of tweaking it on the fly?

Constant learning rates were the default in older ML implementations, but linear decay became an obvious optimization, and now we have both warmup and cosine decay to handle common training patterns, especially with the AdamW optimizer.

If the learning rate is too high at a given point in training, it can result in either a) the model stopping learning or b) exploding gradients, which is very bad.

Re: TinyLlama: An Open-Source Small Language Model

#27

Post by xml »

andy99 wrote: post #18
andy99 wrote: post #3

I've been using one of the earlier checkpoints for benchmarking a Llama implementation. Completely anecdotally I feel at least as good or better about this one than the earlier openllama 3B. I wouldn't use either of them for RAG or anything requiring more power, just to say that it's competitive as a smaller model, whatever you use those for, and easy to run on CPU at FP16 (meaning without serious quantization).

Also, I should promote the code I wrote for running this. It runs models in ggml format, the one I made available is an older checkpoint though. It's easy to convert the newer one. And it's in Fortran but it should be easy to get gfortran if you don't have it installed. https://github.com/rbitr/llm.f90/tree/optimize16/purefortran

Here is another inference implementation in Python (only dependency is PyTorch).

https://github.com/99991/SimpleTinyLlama

The new checkpoints did not seem much better and they changed the chat format for some reason, so I did not port the new checkpoints yet. Perhaps I'll get to it this weekend.

Re: TinyLlama: An Open-Source Small Language Model

#28

Post by zozbot234 »

WhitneyLand wrote: post #22
minimaxir wrote: post #13

It was fun to follow the public TinyLlama loss curves in near real-time, although it showed that it can be frustrating since the loss curves barely moved down even after an extra trillion tokens: https://wandb.ai/lance777/lightning_logs/reports/metric-trai... (note the log-scaled X-axis) But they did move down and that's what's important. There should probably be more aggressive learning rate annealing for models try…

More aggressive learning rate? Wouldn’t we need more information on why they decided to stop training at this point to conclude that? My current understanding of the story is, to recap: - First the game was increase model size massively - For example GPT3 had 175B parameters, but less than 0.5T tokens of training data - Then Chinchilla showed for a given compute budget we can scale better by increasing training data…

Overfitting is quite unlikely with a smaller model though. Model parsimony provides a kind of regularization "for free", in fact with the extra benefit of saving on compute costs.

Re: TinyLlama: An Open-Source Small Language Model

#29

Post by grandma_tea »

ssivark wrote: post #20
minimaxir wrote: post #13

It was fun to follow the public TinyLlama loss curves in near real-time, although it showed that it can be frustrating since the loss curves barely moved down even after an extra trillion tokens: https://wandb.ai/lance777/lightning_logs/reports/metric-trai... (note the log-scaled X-axis) But they did move down and that's what's important. There should probably be more aggressive learning rate annealing for models try…

How crucial is it to freeze the learning rate schedule a priori, instead of tweaking it on the fly?

Adaptive learning rate is a thing. For example, one scheme I've used before is to decrease the learning rate if the validation loss stops decreasing.

It's not clear to me if this is applicable to LLMs though.

Re: TinyLlama: An Open-Source Small Language Model

#30

Post by andy99 »

fergal_reid wrote: post #23
andy99 wrote: post #16

Earlier quoted context omitted.

The smallest model your users agree meets their needs. It really depends. The retrieval part is way more important. I've used the original 13B instruction tuned llama2, quantized, and found it gives coherent answers about the context provided, ie the bottleneck was mostly getting good context. When I played with long context models (like 16k tokens, and this was a few months ago, maybe they improved) they sucked.

>The retrieval part is way more important. I don't agree with this - at Intercom we've put a lot of work into our Fin chatbot, which uses a RAG architecture, and we're still using GPT-4 for the generation part. GPT-4 is a really powerful and expensive model but we find we need this power to 1) reduce hallucinations acceptably, and 2) keep the quality of inferences made using the retrieved text high. Now, our bot is a…

That's interesting, thanks. My experience is with technical documentation Q&A, returning summaries and relevant passages. My takeaway was that the summary is basically as good as the passages. I do think overall response quality is very subjective and really depends on how it's being used, so whatever users do best with wins the day.
Post reply on HN