Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training LLMs from ground zero as a startup

yitay.net

Post reply on HN
Print view View original

121–125 of 125 posts

Re: Training LLMs from ground zero as a startup

#121

Post by imtringued »

dauertewigkeit wrote: post #7
joe_the_user wrote: post #6

So essentially a startup in this context has a small number of people and a large amount of money for training clusters. The article describes many operation leasing servers - that you assume to go many startups (or existing firms). So it seems like you have the various LLM creators all doing roughly the same sort of thing (training with text and image data) with similar hardware and similar data. Each of these natur…

I don't think most of them have any kind of secret sauce. I think the founders hope to get bought out simply for being able to train "near-SOTA" LLMs. I guess achieving that level of skill and infra could be valuable enough to build upon.

There was a guy who followed a tutorial about how to fine tune mistral with DPO, who has zero computer science skills and his model ended up at the top of the hugging face leader board among the opensource models with 7 billion parameters. Some random guy managed to outdo the creators of the LLM.

Re: Training LLMs from ground zero as a startup

#122

Post by mleonhard »

> I think this could be more about the competency of the hardware team that manages your accelerators rather than the underlying chip.

Google's systems are reliable because of the tens of billions of dollars that Google has invested into developing datacenter hardware, software, and processes over 25 years. Highly-competent teams at smaller and less-mature organizations will always deliver a much worse product.

Another thing to consider is priorities. Google prioritizes reliability. They retire parts that fail repeatedly, even if the failures are relatively infrequent. Smaller and less-sophisticated datacenters keep parts in service even with frequent failures, or don't even monitor failure rates of certain parts. Smaller datacenters buy and use Google's old parts and unreliable parts.

Therefore unreliable machines does not imply anything about the competency of the hardware team.

If the low reliability of the hardware is making your work slow, then how about improving the software so it can tolerate the unreliable hardware, or switching to a more reliable (more expensive) hardware provider?

Re: Training LLMs from ground zero as a startup

#123

Post by pests »

eru wrote: post #120
friendzis wrote: post #119

Earlier quoted context omitted.

Since the entity releasing the model obviously has certain goals aligning/censoring model in some ways is good for their particular short-term goal. In the grand scheme these alignments are harmful as they place a reality distortion field. Authors create model of what language is and then contort that model to fit an opinionated idea of what language should be. Smells a bit Orwellian, right?

> Smells a bit Orwellian, right? No, seems perfectly fine by me. You are already shaping your results by your selection of training data. Eg do you want to train a model that speaks English, or German, or both? Do you want to run your training data past a spam filter first? Do you want to do a character based model, or one of those weird encodings that is popular with LLMs these days? Doing some other procedures afte…

I agree but this entire conversation misses my point that "alignment" originally only meant making the LLM act as you want it.

A GPT that hasn't been aligned does not work how we expect - you give it a prompt, and it will autogenerate until it reaches an end state.

To even make the GPT answer the question in the prompt, and not autocomplete it into nonsense, is an example of alignment.

It took a lot of fine tuning and data curation to get ChatGPT up to its current chat-like interface.

But this is not the only alignment you can do. The original Transformer paper was about machine translation, turning the prompt into the translated text. Once it was done it was done.

We could choose to have the model do something else, say translate the prompt into 5 languages at once instead of one, just as an example. This would be another alignment decision.

There is nothing political or selection bais or anything inherent to the original definition, its only recently "alignment" has morphed into this "align with human morals" concept.

Even in the Andrej Karpathy's build-your-own-gpt YT video, which is highly talked about around here, he uses the phase like this. The end of the video you are left with a GPT, but not a question-and-response model, and he says it would need to be aligned to answer questions like ChatGPT.

Re: Training LLMs from ground zero as a startup

#124

Post by londons_explore »

PeterStuer wrote: post #91
joe_the_user wrote: post #6

So essentially a startup in this context has a small number of people and a large amount of money for training clusters. The article describes many operation leasing servers - that you assume to go many startups (or existing firms). So it seems like you have the various LLM creators all doing roughly the same sort of thing (training with text and image data) with similar hardware and similar data. Each of these natur…

"this seems like a massive, energy intensive exercise in redundancy" This is commonly refered to as a market working as intended. Yes, the waste from this type of redundency can be massive , especially if you realize that ultimately just a tiny percentage of these efforts will result in even moderate success. But it is the price to pay at the edge of progress. A planned monopoly might be more efficient (despite popul…

There is a middle ground....

Still have many teams trying to achieve the goal, but prevent corporate secrecy - effectively allowing competitors to look over each others shoulders and copy good data and ideas.

Such a system probably wants to compensate those whose ideas were copied, but that isn't strictly necessary - another approach is to simply make it illegal not to share data/results. Your compensation is your freedom from prison.

Re: Training LLMs from ground zero as a startup

#125

Post by icpmacdo »

gwern wrote: post #69
ganeshkrishnan wrote: post #52

Earlier quoted context omitted.

OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.". In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4. GPUs have memory constraint i…

> And we are using the older Tesla T4. That's an undemanding and well-debugged chip by this point (6 years ago!). So you aren't experiencing any of the pain people using A100s or H100s (never mind people who have to stand up clusters with B100s soon) are going through now.

> never mind people who have to stand up clusters with B100s soon

Gwern(or anyone else) do you have any resources on this?

Post reply on HN