Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Persimmon-8B

adept.ai

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Persimmon-8B

#31

Post by developist »

stale2002 wrote: post #19
solverist wrote: post #12

Earlier quoted context omitted.

I'm confused. If these tools aren't good enough for AI research then why would they be good enough for consumer applications? If language models can not help with the AI development loop then the technology is not going to be useful for consumer use cases. Code can be very easily verified by linters and type systems so the problem of verification is much simpler than in consumer use cases without linters and type sys…

> If language models can not help with the AI development loop then the technology is not going to be useful for consumer use cases. it quite literally is useful for consumer usecases though. For example, one consumer usecase that is being used by a lot of students right now is cheating on their homework. It is right now being used for all sorts of consumer things like that. Also, if you have an opinion you can just…

I don't have an opinion. I am legitimately surprised that very easy problems in AI research have not already been solved with some foundation model. Translating and optimization of code from one formal language to another seems like a very obvious application of AI and yet most of the work is still done manually.

Re: Persimmon-8B

#32

Post by developist »

lambda wrote: post #17
solverist wrote: post #9

Earlier quoted context omitted.

If the bottleneck is writing performant code then seems like that's the first thing AI companies should solve with AI. If that's solved then building applications on top of that foundation is very easy. Are there any companies working on this problem?

Did you just create three new accounts to ask this series of questions? These questions, and account names, all seem to share similar patterns.

HN keeps throttling after 2 comments. Maybe they need more AI to detect bots instead of whatever algorithm they are using at the moment.

Back to the question at hand. How exactly do AI companies plan to build AGI if they can not optimize the AI development loop with their tools and techniques?

Re: Persimmon-8B

#33

Post by imjonse »

Congrats on the release! Two questions.

1) In the results table, Llama2 base is being compared to Persimmon base and finetuned, and only the latter performs better. Would a comparison to Llama2-chat be possible/fair?

2) The Llama-2 numbers for MMLU in that table seem different from those in the HF leaderboard and the Llama-2 webpage presentation. Is it the 1-shot variant that is different or are these measurements not 100% standard and reproducible?

Re: Persimmon-8B

#34

Post by imjonse »

sbierwagen wrote: post #28
gardnr wrote: post #18

Two important takeaways on the base model: * scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2 * was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k. Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe du…

phi-1 supposedly does 50.6 on HumanEval with 1.3B parameters. (Python only) https://arxiv.org/abs/2306.11644 Weights haven't been released, though.

no code or dataset either for phi-1.

Re: Persimmon-8B

#35

Post by AaronFriel »

I hope this is only a slight tangent; since the authors talk about their model serving throughput and I hope I can get a gut-check on my understanding of the state-of-the-art of model serving.

The success of ChatGPT and my current work has had me thinking a lot about the "product" applications of large language models. I work at Pulumi on www.pulumi.com/ai; it's a GPT-3.5 and GPT-4 interface using retrieval augmented generation to generate Pulumi programs, and user experience is top of mind for me.

(Fingers crossed this doesn't hug our site to death here for the reasons I'm about to explain.)

To be blunt: I have found it surprisingly difficult to find the right tools to host models without dramatically worsening the UX. In theory we should be able to fine-tune a model against our own SDKs and synthetically generated code to improve the model's output and to guard against hallucination when retrieval fails. In practice, self-hosted model serving APIs have really poor time-to-first-token or even completely lack streaming behavior. It's a non-starter to build a product on something where a user has to sit and watch a spinner for a minute or more. I've been looking at the vLLM project with great interest, but haven't found much else.

---

For folks in MLops, deploying models with streaming APIs:

1. Is it mostly accurate that none of the model serving tools created prior to ChatGPT are great for streaming, interactive use cases?

2. How are you currently serving these models as an API and what upcoming tools are you exploring?

For the authors: How does your inference optimization compare to vLLM, or other tools using techniques such as continuous batching and paged attention?

Re: Persimmon-8B

#37

Post by Bnjoroge »

ironrabbit wrote: post #15
automatistist wrote: post #2

> The standard practice for achieving fast inference is to rewrite the entire model inference loop in C++, as in FasterTransformer, and call out to special fused kernels in CUDA. But this means that any changes to the model require painfully reimplementing every feature twice: once in Python / PyTorch in the training code and again in C++ in the inference codebase. We found this process too cumbersome and error prone…

Automatic kernel fusion (compilation) is a very active field, and most major frameworks support some easy-to-use compilation (e.g. jax's jit, or torch.compile which iirc uses openai's triton under the hood). Often you can still do better than the compiler by writing fused kernels yourself (either in cuda c++ or in something like triton (python which compiles down to cuda) but compilers are getting pretty good. edit:…

thanks for explaining pretty concisely w/out being rude :)

Re: Persimmon-8B

#38

Post by ekelsen »

imjonse wrote: post #33

Congrats on the release! Two questions. 1) In the results table, Llama2 base is being compared to Persimmon base and finetuned, and only the latter performs better. Would a comparison to Llama2-chat be possible/fair? 2) The Llama-2 numbers for MMLU in that table seem different from those in the HF leaderboard and the Llama-2 webpage presentation. Is it the 1-shot variant that is different or are these measurements no…

Llama2 chat performs worse and wasn't included for that reason.

The numbers are different because the measurement is different. The blog post explains that we sample from the models and expect answers rather than relying on perplexity measurements.

Re: Persimmon-8B

#39

Post by richdougherty »

theLiminator wrote: post #36

What kind of use cases do these sub 10B param models serve? Are they mostly useful for code completion?

You can run them either for general purpose inference. You can also fine-tune them and get improved performance for specific use cases.

It's safe to assume they're worse at every task than larger models, so I wouldn't look at use cases in terms of what tasks they can do compared to larger models.

But what's good about them is they're smaller so they can run on smaller and cheaper hardware. So an example would be to fine-tune and then run on some sort of local user device rather than in the cloud. This might become more practical in the future as hardware improves.

Re: Persimmon-8B

#40

Post by deckar01 »

The docker container fails installing flash-attn… but honestly a giant API container on top of a custom model generation framework loses all the benefits of Torch’s standard interfaces. It doesn’t really matter how optimized your model runtime is if it’s cemented into a synchronous monolith. The metric that should be optimized is time to first decoded token, because that is how speed is perceived by humans reading the output.
Post reply on HN