Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyLlama: An Open-Source Small Language Model

arxiv.org

Post reply on HN
Print view View original

41–46 of 46 posts

Re: TinyLlama: An Open-Source Small Language Model

#41

Post by ukuina »

matt1 wrote: post #12
ukuina wrote: post #7

Earlier quoted context omitted.

I visit your site every day. Thank you for creating it and evolving it past simple summaries to show paper details! I recall you were looking to sell it at some point. Was wondering what that process looked like, and why you ended up holding on to the site.

Hey, thanks for the kind words. To answer your question: an earlier version of the site focused on surfacing AI news, but that space is super competitive and I don't think Emergent Mind did a better job than the other resources out there. I tried selling it instead of just shutting it down, but ultimately decided to keep it. I recently decided to pivot to covering arXiv papers, which is a much better fit than AI news…

Thank you for the detailed response!

Pivoting into arXiv is a good idea. It helps you have focused prompts and templates.

A natural progression is aggregation, categorization, and related paper suggestions. Since arXiv has HTML versions of papers now, you can also consider allowing deeplinked citations directly from the LLM summaries.

A GPT-curated comments section for papers would also be nice, automatically filtering out any spam that gets past the regular Disqus filters, then scoring/hiding comments based on usefulness or insight.

Re: TinyLlama: An Open-Source Small Language Model

#42

Post by kiratp »

minimaxir wrote: post #33
kiratp wrote: post #31

Earlier quoted context omitted.

> For any model, the loss curve going down could mean it’s learning, or could mean it’s overfitting, we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before. You want to look at validation accuracy.

Accuracy is a bad metric for LLMs, especially since a LLM tokenizer can have thousands of "classes": 32,000 in the case of TinyLlama.

I guess it comes down to whether your usecase has a single correct answer vs multiple possible ones. For example a lot of what we do has one and only one correct sequence of tokens. Need to look at both but so much of the learning material out there just focuses on loss. YMMV.

Re: TinyLlama: An Open-Source Small Language Model

#43

Post by chrismorgan »

TheCoreh wrote: post #40

From the GitHub repo Readme: > we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs I knew the computational power required to train LLMs was absurd, but seeing the figures of larger networks (which are just too large to intuitively understand) it didn't really register. With this one I could actually imagine the 16 machines with A100 GPUs sitting on a server room running at full blast for 90 da…

So, four A100-years. Unit cost $8,000 (from a quick search) and electricity cost under $2,000. If you reckon the useful life of an A100 to be four years then that’s a training cost approaching $10,000. I have no idea of the forecast useful life of the GPU, but I’d hope it’d be a lot longer; if it was about ten years, then this training cost would be around $5,000.

Of course, we’re probably both simplifying thing too much, but if these numbers are good enough it’s an interesting perspective.

At these sorts of costs and a final size of 2.2GB, each MB cost a few dollars to produce.

Re: TinyLlama: An Open-Source Small Language Model

#45

Post by refulgentis »

refulgentis wrote: post #25
andy99 wrote: post #18

Earlier quoted context omitted.

Also, I should promote the code I wrote for running this. It runs models in ggml format, the one I made available is an older checkpoint though. It's easy to convert the newer one. And it's in Fortran but it should be easy to get gfortran if you don't have it installed. https://github.com/rbitr/llm.f90/tree/optimize16/purefortran

Man I didn't recognize your username but once you said Fortran I recognized you immediately. You are an inspiration, an example of true software engineering vis a vis what in day to day becomes what you can hire for. Edit: you have some rare knowledge, I'm curious if you have any thoughts on small models good enough for RAG. Mistral 7B is in my testing buts it's laughably slow and 7B is just too much for mobile, both…

So much changed in a day. What a field!

Re: TinyLlama: An Open-Source Small Language Model

#46

Post by minimaxir »

kiratp wrote: post #42
minimaxir wrote: post #33

Earlier quoted context omitted.

Accuracy is a bad metric for LLMs, especially since a LLM tokenizer can have thousands of "classes": 32,000 in the case of TinyLlama.

I guess it comes down to whether your usecase has a single correct answer vs multiple possible ones. For example a lot of what we do has one and only one correct sequence of tokens. Need to look at both but so much of the learning material out there just focuses on loss. YMMV.

That is already accounted for with categorical cross-entropy loss.
Post reply on HN