Minor point: something about the HTML input is causing iOS’s auto correct to be disabled; making input very frustrating!
Groqchat
51–60 of 131 posts
Re: Groqchat
#52Earlier quoted context omitted.
It’s just running bog standard Llama2-70B by all appearances. I don’t know why so many people here are interested in the outputs. The whole point of this demo is that the company is trying to show off how fast their hardware could host one of your models, not the model itself.
Doesn't the speed also depend on the number of people currently accessing it?
I assume it queues up requests if there are too many in flight to handle all of them at the same time, but it shows you the tokens per second (T/s) for every response, which is the number that matters (and presumably won't include the time spent in the queue).
Re: Groqchat
#53It’s very fast at telling me it can’t tell me things! I asked about creating illicit substances — an obvious (and reasonable) target for censorship. And, admirably, it suggested getting help instead. That’s fine. But I asked for a poem about pumping gas in the style of Charles Bukowski, and it moaned that I shouldn’t ask for such mean-spirited, rude things. It wouldn’t dare create such a travesty.
It seems to reject all lyrics requests as well (In my experience, LLMs are good at the first one or two lines, and then just make it up as they go along, with sometimes hilarious results).
Re: Groqchat
#54I posed this question to GPT-4 and Groq: "I am building an api in spring boot that persists users documents. This would be for an hr system. There are folders, and documents, which might have very sensitive data. I will need somewhere to store metadata about those documents. I was thinking of using postgres for the emtadata, and s3 for the actual documents. Any better ideas? or off the shelf libraries for this?" Both…
[deleted]
Re: Groqchat
#55This isn't running on one chip. It's running on 128, or two racks worth of their kit. https://news.ycombinator.com/item?id=38739106 This doesn't mean much without comparing $ or watts of GPU equivalents
This article from less than a month ago says that it is on 576 chips https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...
Re: Groqchat
#56It’s very fast at telling me it can’t tell me things! I asked about creating illicit substances — an obvious (and reasonable) target for censorship. And, admirably, it suggested getting help instead. That’s fine. But I asked for a poem about pumping gas in the style of Charles Bukowski, and it moaned that I shouldn’t ask for such mean-spirited, rude things. It wouldn’t dare create such a travesty.
It seems like it must be using Llama-2-chat, which has had 'safety' training.
To test which underlying model I asked it what a good sexy message for my girlfriend for Valentine's Day would be, and it lectured me about objectification.
It makes sense the chat interface is using the chat model, I just wish that people were more consistent about labeling the use of Llama-2-chat vs Llama-2 as the fine tuning really does lead to significant underlying differences.
Re: Groqchat
#57Yeah, it’s fast but almost always wrong. I asked it a few things (recipes, trivia etc…) and it completely made up the answers. These things don’t really know how to say “I don’t know” and pretend to know everything.
Yeah, that's raw llama2 for you. Fine tuning, LoRa, that's how you get the nice responses.
I suspect this is Llama-2-chat and not the base model. It's very attuned to safety fine tuning, and it isn't having issues with completion in a chat format.
They probably weren't specific enough in mentioning what the model it was built on was, referring to Llama-2-chat as being a Llama-2 model (which is kind of correct).
Re: Groqchat
#58Lots of comments talking about the model itself. This is Llama 2 70B, a model that has been around for a while now, so we're not seeing anything in terms of model quality (or model flaws) we haven't seen before. What's interesting about this demo is the speed at which it is running, which demonstrates the "Groq LPU™ Inference Engine". That's explained here: https://groq.com/lpu-inference-engine/ > This is the world’s…
This is really impressive. For reference, inference for llama 70b on together’s api generates text at roughly 60 tokens/second.
I can’t find any information about an api, though I’m guessing that the costs are eye watering.
If they offered a Mixtral endpoint that did 300-400 tokens per second at a reasonable cost, I can’t imagine ever using another provider.
Re: Groqchat
#59Lots of comments talking about the model itself. This is Llama 2 70B, a model that has been around for a while now, so we're not seeing anything in terms of model quality (or model flaws) we haven't seen before. What's interesting about this demo is the speed at which it is running, which demonstrates the "Groq LPU™ Inference Engine". That's explained here: https://groq.com/lpu-inference-engine/ > This is the world’s…
this is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202... EDIT: i work at Groq, but i’m commenting in a personal capacity. happy to answer clarifying questions or forward them along to folks who can :)
what’s the cost?
Re: Groqchat
#60I asked "How up to date is your information about the world?" It said December 2022, but the answers to another question was not correct for that time or now. It also went into some kind of repeating loop to its maximum response length. Still pretty cool that our standards for chat programs have risen.
LLMs don’t answer based on truth. They are a glorified autocomplete trained on a large corpus of text.