Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persimmon-8B

adept.ai

Post reply on HN
Print view View original

11–20 of 61 posts

Re: Persimmon-8B

#11

Post by TrueDuality »

automatistredo wrote: post #5
automatistist wrote: post #2

> The standard practice for achieving fast inference is to rewrite the entire model inference loop in C++, as in FasterTransformer, and call out to special fused kernels in CUDA. But this means that any changes to the model require painfully reimplementing every feature twice: once in Python / PyTorch in the training code and again in C++ in the inference codebase. We found this process too cumbersome and error prone…

Can someone explain the down votes? What exactly is incorrect in OPs comment?

I would have to guess it has something to do with that task not actually being suitable for language models at their current stage. Even if they could be trusted to perform the task, its actually not that much work to just... write code to handle keeping this kind of thing in sync. It's really really not that much more work. You really don't even need to do it, both training and inference can be done within PyTorch or in C++.

If it was necessary for some reason... Running a language model to keep something like this is sync over long term training and iteration would likely be more expensive than a developer's time AND block the researcher in a verification loop on the output which still probably needs to be checked by the developer (they could be the same person which will just deepen the frustration they experience).

The use of a lot of garbage accounts in this thread and lack of model details also looks pretty shady...

Re: Persimmon-8B

#12

Post by solverist »

TrueDuality wrote: post #11
automatistredo wrote: post #5

Earlier quoted context omitted.

Can someone explain the down votes? What exactly is incorrect in OPs comment?

I would have to guess it has something to do with that task not actually being suitable for language models at their current stage. Even if they could be trusted to perform the task, its actually not that much work to just... write code to handle keeping this kind of thing in sync. It's really really not that much more work. You really don't even need to do it, both training and inference can be done within PyTorch o…

I'm confused. If these tools aren't good enough for AI research then why would they be good enough for consumer applications? If language models can not help with the AI development loop then the technology is not going to be useful for consumer use cases. Code can be very easily verified by linters and type systems so the problem of verification is much simpler than in consumer use cases without linters and type systems.

Re: Persimmon-8B

#13

Post by 123yawaworht456 »

YetAnotherNick wrote: post #8
123yawaworht456 wrote: post #6

I applaud you guys for not including any nauseating gibberish in this press release or seemingly anywhere else on your website. It's like a breath of fresh air comparing to every other AI-related resource I saw recently. Please, keep it up.

This is the least detailed foundational model release I have seen. Llama paper offers lot more details like ablations, loss curves etc. Falcon has data preparation details etc. Google's model release papers like T5 are some of the best and includes many ablations.

I mean "I am become death, destroyer of worlds" bullshit about AI safety/ethics/etc that is included in every press release from Google/Meta/OpenAI and even much smaller players.

Re: Persimmon-8B

#15

Post by ironrabbit »

automatistist wrote: post #2

> The standard practice for achieving fast inference is to rewrite the entire model inference loop in C++, as in FasterTransformer, and call out to special fused kernels in CUDA. But this means that any changes to the model require painfully reimplementing every feature twice: once in Python / PyTorch in the training code and again in C++ in the inference codebase. We found this process too cumbersome and error prone…

Automatic kernel fusion (compilation) is a very active field, and most major frameworks support some easy-to-use compilation (e.g. jax's jit, or torch.compile which iirc uses openai's triton under the hood). Often you can still do better than the compiler by writing fused kernels yourself (either in cuda c++ or in something like triton (python which compiles down to cuda) but compilers are getting pretty good.

edit: not sure why op is getting downvotes, this is a very reasonable question imo; maybe the characterization of kernel compilation as "AI" vs. just "software"?

Re: Persimmon-8B

#16

Post by TrueDuality »

solverist wrote: post #12
TrueDuality wrote: post #11

Earlier quoted context omitted.

I would have to guess it has something to do with that task not actually being suitable for language models at their current stage. Even if they could be trusted to perform the task, its actually not that much work to just... write code to handle keeping this kind of thing in sync. It's really really not that much more work. You really don't even need to do it, both training and inference can be done within PyTorch o…

I'm confused. If these tools aren't good enough for AI research then why would they be good enough for consumer applications? If language models can not help with the AI development loop then the technology is not going to be useful for consumer use cases. Code can be very easily verified by linters and type systems so the problem of verification is much simpler than in consumer use cases without linters and type sys…

I don't think you read what I said, or you don't know what you're talking about. Language models are actively being researched to be useful, but as of right now _they are not ready, capable, or trustworthy enough_ to perform tasks without supervision, they're especially bad at complex code and code that doesn't have many examples in the internet corpuses such as optimized CUDA kernels...

Re: Persimmon-8B

#17

Post by lambda »

solverist wrote: post #9
snissn wrote: post #7

Earlier quoted context omitted.

i don't know why people downvote, but writing highly performant gpu code across multiple languages is still in the realm of only a few people with a lot of the right experience can do well, and while ai can help assist those people it's not a problem that can be fully solved by an ai at this moment, maybe a few years with a large feedback loop of iterating, testing, benchmarking, repeating. i guess one day but not no…

If the bottleneck is writing performant code then seems like that's the first thing AI companies should solve with AI. If that's solved then building applications on top of that foundation is very easy. Are there any companies working on this problem?

Did you just create three new accounts to ask this series of questions? These questions, and account names, all seem to share similar patterns.

Re: Persimmon-8B

#18

Post by gardnr »

Two important takeaways on the base model:

* scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2

* was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k.

Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe during pretraining?

Re: Persimmon-8B

#19

Post by stale2002 »

solverist wrote: post #12
TrueDuality wrote: post #11

Earlier quoted context omitted.

I would have to guess it has something to do with that task not actually being suitable for language models at their current stage. Even if they could be trusted to perform the task, its actually not that much work to just... write code to handle keeping this kind of thing in sync. It's really really not that much more work. You really don't even need to do it, both training and inference can be done within PyTorch o…

I'm confused. If these tools aren't good enough for AI research then why would they be good enough for consumer applications? If language models can not help with the AI development loop then the technology is not going to be useful for consumer use cases. Code can be very easily verified by linters and type systems so the problem of verification is much simpler than in consumer use cases without linters and type sys…

> If language models can not help with the AI development loop then the technology is not going to be useful for consumer use cases.

it quite literally is useful for consumer usecases though.

For example, one consumer usecase that is being used by a lot of students right now is cheating on their homework.

It is right now being used for all sorts of consumer things like that.

Also, if you have an opinion you can just say what your opinion is. You don't have to hide it behind questions and alt accounts.

You can just have an opinion and say it.

Re: Persimmon-8B

#20

Post by YetAnotherNick »

solverist wrote: post #10
YetAnotherNick wrote: post #8

Earlier quoted context omitted.

This is the least detailed foundational model release I have seen. Llama paper offers lot more details like ablations, loss curves etc. Falcon has data preparation details etc. Google's model release papers like T5 are some of the best and includes many ablations.

Why are ablations useful? Their release report seemed very informative to me without getting bogged down in jargon.

Ablations are important because they tell why the model is better. Here the model is of similar size of llama 7b, trained on 1/3rd the dataset still their claim is that the performance is better. Now this could happen due to lot of things like relu squared or better dataset or 16k tokens. We just don't know why it performed better.
Post reply on HN