Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persimmon-8B

adept.ai

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Persimmon-8B

#41

Post by ekelsen »

deckar01 wrote: post #40

The docker container fails installing flash-attn… but honestly a giant API container on top of a custom model generation framework loses all the benefits of Torch’s standard interfaces. It doesn’t really matter how optimized your model runtime is if it’s cemented into a synchronous monolith. The metric that should be optimized is time to first decoded token, because that is how speed is perceived by humans reading th…

Can you share details of the build failure on the github? We'll try to help.

The inference code is shared as a proof of concept, it is not meant to be a production ready deploy. Also worth noting that not all LLMs are used to produce text which is read by humans.

Re: Persimmon-8B

#42

Post by Havoc »

>The model has 70k unused embeddings for multimodal extensions,

Could someone briefly explain what this means? multimodal as in picture, but if unused then presumably that part is somehow untrained...so it wouldn't know what to do with the picture?

Re: Persimmon-8B

#43

Post by loopist »

ironrabbit wrote: post #15
automatistist wrote: post #2

> The standard practice for achieving fast inference is to rewrite the entire model inference loop in C++, as in FasterTransformer, and call out to special fused kernels in CUDA. But this means that any changes to the model require painfully reimplementing every feature twice: once in Python / PyTorch in the training code and again in C++ in the inference codebase. We found this process too cumbersome and error prone…

Automatic kernel fusion (compilation) is a very active field, and most major frameworks support some easy-to-use compilation (e.g. jax's jit, or torch.compile which iirc uses openai's triton under the hood). Often you can still do better than the compiler by writing fused kernels yourself (either in cuda c++ or in something like triton (python which compiles down to cuda) but compilers are getting pretty good. edit:…

Both AI and compilers are just software and right now the optimizers are written manually which is kinda weird because the whole point of LLMs is to generate sequences of tokens that minimize some scalar valued loss function. In the case of compilers the input is some high level code in python expressing tensor operations and the output is whatever is executable by GPUs as fast as possible by combination of kernels which are formally equivalent to the tensor operations expressed in Python (or whatever higher level language is used to write the tensor specifications to be optimized for the task at hand). Everything in this loop has a well defined input with a well defined output and an associated scalar valued metric (execution time) and even a normalization factor (output length with shorter sequences being "better").

The whole thing seems obviously amenable to gradient based optimization and data augmentation with synthetic code generators. It is surprising that no one is pursuing such approaches to improving the optimization pipeline in kernel compilation/fusion/optimization because it is just another symbol game with much better defined metrics than natural language models.

Re: Persimmon-8B

#44

Post by [deleted] »

developist wrote: post #31
stale2002 wrote: post #19

Earlier quoted context omitted.

> If language models can not help with the AI development loop then the technology is not going to be useful for consumer use cases. it quite literally is useful for consumer usecases though. For example, one consumer usecase that is being used by a lot of students right now is cheating on their homework. It is right now being used for all sorts of consumer things like that. Also, if you have an opinion you can just…

I don't have an opinion. I am legitimately surprised that very easy problems in AI research have not already been solved with some foundation model. Translating and optimization of code from one formal language to another seems like a very obvious application of AI and yet most of the work is still done manually.

[deleted]

Re: Persimmon-8B

#45

Post by theLiminator »

richdougherty wrote: post #39
theLiminator wrote: post #36

What kind of use cases do these sub 10B param models serve? Are they mostly useful for code completion?

You can run them either for general purpose inference. You can also fine-tune them and get improved performance for specific use cases. It's safe to assume they're worse at every task than larger models, so I wouldn't look at use cases in terms of what tasks they can do compared to larger models. But what's good about them is they're smaller so they can run on smaller and cheaper hardware. So an example would be to f…

Yeah, my point is moreso is are smaller models ever "smart" enough to perform useful tasks?

Perhaps for basic code completion and simple writing tasks?

Re: Persimmon-8B

#46

Post by sthatipamala »

Havoc wrote: post #42

>The model has 70k unused embeddings for multimodal extensions, Could someone briefly explain what this means? multimodal as in picture, but if unused then presumably that part is somehow untrained...so it wouldn't know what to do with the picture?

Yes, it wouldn't know what to do with the picture unless you fine-tune the model (which is why they are permissively releasing it).

The embeddings form the vocabulary of the model. The vocabulary "namespace" has 70k empty slots so you could introduce your own tokens and train on top of that, where token = some patch of multimodal data.

Re: Persimmon-8B

#47

Post by coder543 »

gardnr wrote: post #18

Two important takeaways on the base model: * scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2 * was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k. Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe du…

> scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2

The article claims 18.9 for the base model, but also claims 20.7 for the fine tuned model.

Re: Persimmon-8B

#48

Post by swyx »

gardnr wrote: post #18

Two important takeaways on the base model: * scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2 * was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k. Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe du…

its not so much about benefit, as it is a design goal to want large context windows.

https://twitter.com/suchenzang/status/1699926157028897078?s=... notes some issues directly comparing the 16k context number. the odd choice of tokenizer means its effectively like a 10-12k model (? ballpark, not calculated)

Re: Persimmon-8B

#49

Post by Havoc »

sthatipamala wrote: post #46
Havoc wrote: post #42

>The model has 70k unused embeddings for multimodal extensions, Could someone briefly explain what this means? multimodal as in picture, but if unused then presumably that part is somehow untrained...so it wouldn't know what to do with the picture?

Yes, it wouldn't know what to do with the picture unless you fine-tune the model (which is why they are permissively releasing it). The embeddings form the vocabulary of the model. The vocabulary "namespace" has 70k empty slots so you could introduce your own tokens and train on top of that, where token = some patch of multimodal data.

Gotcha. Thanks for explaining
Post reply on HN