Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persimmon-8B

adept.ai

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Persimmon-8B

#51

Post by gremlinsinc »

theLiminator wrote: post #36

What kind of use cases do these sub 10B param models serve? Are they mostly useful for code completion?

say you had very vertical trained models, such that you had like 1000 separate LLMs trained on specialized data and then others LLMs trained on which LLM is most likely to have the data you need, sort of like the way Wikipedia is interlinked, or hierarchical, or essentially like a db index, over nested LLMs, performance would scale higher with many more highly focused models, at least that's my understanding of what possible here.

Re: Persimmon-8B

#52

Post by deckar01 »

ekelsen wrote: post #41
deckar01 wrote: post #40

The docker container fails installing flash-attn… but honestly a giant API container on top of a custom model generation framework loses all the benefits of Torch’s standard interfaces. It doesn’t really matter how optimized your model runtime is if it’s cemented into a synchronous monolith. The metric that should be optimized is time to first decoded token, because that is how speed is perceived by humans reading th…

Can you share details of the build failure on the github? We'll try to help. The inference code is shared as a proof of concept, it is not meant to be a production ready deploy. Also worth noting that not all LLMs are used to produce text which is read by humans.

https://github.com/persimmon-ai-labs/adept-inference/issues/...

It’s funny you say production, because all of the errors I ran into suggest the container is expecting your production architecture.

My advice is stream first then make synchronous convenience wrappers on top of that. Also, lean on community standards for PoC. I’m guessing your investors are interested in making this scale as cheaply as possible, but that is probably the least important feature for people evaluating your model’s quality locally.

Re: Persimmon-8B

#53

Post by craigacp »

swyx wrote: post #48
gardnr wrote: post #18

Two important takeaways on the base model: * scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2 * was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k. Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe du…

its not so much about benefit, as it is a design goal to want large context windows. https://twitter.com/suchenzang/status/1699926157028897078?s=... notes some issues directly comparing the 16k context number. the odd choice of tokenizer means its effectively like a 10-12k model (? ballpark, not calculated)

There's a correction to that tweet, larger vocab means fewer tokens for any given sequence (usually, assuming it's not to add other languages or character sets).

Re: Persimmon-8B

#54

Post by YetAnotherNick »

ekelsen wrote: post #38
imjonse wrote: post #33

Congrats on the release! Two questions. 1) In the results table, Llama2 base is being compared to Persimmon base and finetuned, and only the latter performs better. Would a comparison to Llama2-chat be possible/fair? 2) The Llama-2 numbers for MMLU in that table seem different from those in the HF leaderboard and the Llama-2 webpage presentation. Is it the 1-shot variant that is different or are these measurements no…

Llama2 chat performs worse and wasn't included for that reason. The numbers are different because the measurement is different. The blog post explains that we sample from the models and expect answers rather than relying on perplexity measurements.

Could you share the results with standard way of benchmarking(accuracy of top selection). While the approach you guys took is reasonable, but it would be more informative to see to see how much better/worse it is with standard benchmark.

Re: Persimmon-8B

#55

Post by visarga »

Since it is coming from Adept, maybe they are building 8B models for UI automation, the inputs are usually large and latency required is low. It's basically a task of information extraction and UI action generation.

Re: Persimmon-8B

#56

Post by selfhoster11 »

thewataccount wrote: post #4

Awesome! I applaud everyone training new models and attempting different techniques! I'm concerned about the current download's availability - its two URLs to some object storage. I find that these go dark rather quickly for many different reasons (accidentally moving it, bandwidth limits, deleting it later, etc). I'm curious if there's a reason it's not also hosted on huggingface? I'm not saying they're the best pla…

If this model can be made to work as GGUF, TheBloke will probably have a set of quantizations in a day or two at most.

Re: Persimmon-8B

#57

Post by hansonw »

AaronFriel wrote: post #35

I hope this is only a slight tangent; since the authors talk about their model serving throughput and I hope I can get a gut-check on my understanding of the state-of-the-art of model serving. The success of ChatGPT and my current work has had me thinking a lot about the "product" applications of large language models. I work at Pulumi on www.pulumi.com/ai; it's a GPT-3.5 and GPT-4 interface using retrieval augmented…

This is the best comparison I've found that benchmarks the current OSS inference solutions: https://hamel.dev/notes/llm/inference/03_inference.html

IME the streaming API in text-generation-inference works fine in production. (Though some of the other solutions may be better). I've used it with Starcoder (15B) and the time-to-first-token / tokens per second all seem quite reasonable out of the box.

Re: Persimmon-8B

#58

Post by euclaise »

sbierwagen wrote: post #28
gardnr wrote: post #18

Two important takeaways on the base model: * scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2 * was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k. Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe du…

phi-1 supposedly does 50.6 on HumanEval with 1.3B parameters. (Python only) https://arxiv.org/abs/2306.11644 Weights haven't been released, though.

phi-1 is a code-specific base model, with further finetuning on top of that. This is a general language base model, not really comparable.

Re: Persimmon-8B

#59

Post by andai »

sunshadow wrote: post #29

Do you have any explanations on why this performed better than Llama 2?

They did several things differently from Llama 2.

From my understanding, you'd have to repeat the experiment isolating each variable to see what difference each one actually makes, no?

Re: Persimmon-8B

#60

Post by euclaise »

swyx wrote: post #48
gardnr wrote: post #18

Two important takeaways on the base model: * scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2 * was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k. Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe du…

its not so much about benefit, as it is a design goal to want large context windows. https://twitter.com/suchenzang/status/1699926157028897078?s=... notes some issues directly comparing the 16k context number. the odd choice of tokenizer means its effectively like a 10-12k model (? ballpark, not calculated)

That tweet had it backwards, more tokens in tokenizer means that the 16k token context window typically allows for even longer passages than if LLaMA were 16k
Post reply on HN