Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI GPT-4 vs. Groq Mistral-8x7B

serpapi.com

Post reply on HN
Print view View original

101–110 of 139 posts

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#101

Post by osigurdson »

londons_explore wrote: post #62
dns_snek wrote: post #56

For all the posturing and crypto hate on HN, we're entering a world where it's socially acceptable to use 1000W of computing power and 5 seconds of inference time to parse a tiny HTML fragment which would take microseconds with traditional methods - and people are cheering about it. Time for some self-reflection? That's not very green.

While energy remains cheap and human minds remain expensive, it always makes sense to use AI to reduce human effort. If one cares about the environment, a carbon cap/tax is what you should campaign for. Then carbon-based energy sources will be curtailled, energy costs will go up, and AI like this will be encouraged to become more energy efficient or other methods used instead.

It is a nice idea in principle but ends up being a political tool and a tariff on goods and services of your own country. A global and corruption free carbon tax might work but that is impossible to achieve.

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#102

Post by Eisenstein »

m348e912 wrote: post #49
Drakim wrote: post #22

Earlier quoted context omitted.

If I ask an LLM a very complex and specific question 500 times, if it just doesn't know the facts you'll still get the wrong answer 500 times. That's understandable. The real problem is when the AI lies/hallucinates another answer with confidence instead of saying "I don't know".

The weird problem is with LLM hallucinations is that it usually will acknowledge its mistake and correct itself if you call it out. My question is why can't LLMs included a sub-routine to check itself before answering. Simply asking itself something like "this answer may not be correct, are you sure you're right?"

> My question is why can't LLMs included a sub-routine to check itself before answering.

Because LLMs don't work in a way for that to be possible if you operate them on their own.

Here is the debug output of my local instance of Mistral-Instruct 8x7B. The prompt from me was 'What is poop spelled backwards?'. It answered 'puoP'. Let's see how it got there starting with it processing my prompt into tokens:

   'What (3195)', ' is (349)', ' po (1627)', 'op (410)', ' sp (668)', 'elled (6099)', ' backwards (24324)', '? (28804)', '\n (13)', '### (27332)', ' Response (12107)', ': (28747)', '\n (13)',
It tokenized 'poop' as two tokens: 'po', number 1627, and 'op', number 410.

Next it comes up with its response:

   Generating (1 / 512 tokens) [(pu 4.43%) (The 66.62%) (po 11.96%) (p 4.99%)]
   Generating (2 / 512 tokens) [(o 89.90%) (op 10.10%)]
   Generating (3 / 512 tokens) [(P 100.00%)]
   Generating (4 / 512 tokens) [( 100.00%)]
It picked 'pu' even though it was only a ~4% chance of being correct, then instead of picking 'op' it picked 'o'. The last token was a 100% probability of being 'P'.

   Output: puoP
At no time did it write 'puoP' as a complete word nor does it know what 'puoP' is. It has no way of evaluating whether that is the right answer or not. You would need a different process to do that.

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#103

Post by infecto »

This test is interesting from a general high level metric/test but overall the way they are extracting data using a LLM is suboptimal so I don't think the takeaway means much. You could extract this type of data using a low-end model like 8x7B with a high degree of accuracy.

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#104

Post by ogogmad »

dns_snek wrote: post #56

For all the posturing and crypto hate on HN, we're entering a world where it's socially acceptable to use 1000W of computing power and 5 seconds of inference time to parse a tiny HTML fragment which would take microseconds with traditional methods - and people are cheering about it. Time for some self-reflection? That's not very green.

You're partially right. It's obvious that the solution is to combine traditional programming with AI, using traditional programming wherever possible because it's greener. Assuming you want things to turn out well in every possible future scenario, your decisions only matter if AGI isn't right around the corner. So assume it isn't right around the corner. Then there's going to be some interesting combining-together of manual human intervention, traditional software, and AI. We'll need to charge more for some uses of electricity, to incentivise turning AI into traditional software wherever possible.

Crypto is nearly pure waste.

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#105

Post by Jensson »

malux85 wrote: post #14
selcuka wrote: post #11

Earlier quoted context omitted.

The first entry on Google is Wikipedia [1] for me: > N/A (or sometimes n/a or N.A.) is a common abbreviation in tables and lists for the phrase not applicable, not available, not assessed, or no answer. [1] https://en.wikipedia.org/wiki/N/A

Thats interesting, wikipedia is not on the first page for me, my first hit is Cambridge dict: (and then a bunch of other dicts) - Im flying right now but IP geolocation puts me in the US Meaning of n/a in English written abbreviation for not applicable: used on a form to show that you are not giving the information asked for because the question is not intended for you or your situation: If a question does not apply…

In a data table "not available" is usually the right word for it, like if you have a list of national statistics then some of the values wont be available due to political reasons etc. But all of those means basically the same thing to the end user, this value isn't there.

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#106

Post by darthrupert »

RUnconcerned wrote: post #30
feintruled wrote: post #9

Brave new world, where our machines are sometimes wrong but by gum they are quick about it.

I too am a big fan of having my computer hallucinate incorrect information.

Yesterday I asked my locally running gpt4all "What model are you running on?"

Answer: "I'm running on Toyota Corolla"

Which was perhaps the funniest thing I heard that day.

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#107

Post by mhuffman »

infecto wrote: post #96
wruza wrote: post #60

The prompt, for those interested. I find it pretty underspecified, but maybe that's the point. For example, "Business operating hours" could be expanded a little, because "Closed - Opens at XX" is still non-processable in both cases. You are an expert in Web Scraping, so you are capable to find the information in HTML and label them accordingly. Please return the final result in JSON. Data to scrape: title: Name of t…

This should be higher up. This whole blog post is mostly worthless because the way they are extracting data is less than optimal. Lower end models do not have the attention to complete tasks like this, GPT4Turbo will generally have the capability. But to have an optimal pipeline you should really be splitting up these tasks into individual units. You extract each attribute you want independently and then combine it b…

Do you have an example of a more optimal prompt to share?

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#108

Post by emporas »

Mixtral works very well with json output in my personal experience. Gpt family are excellent of course, and i would bet Claude and Gemini are pretty good. Mixtral however is the smallest of the models and the most efficient.

Especially running on Groq's infrastructure it's blazing fast. Some examples i ran on Groq's API, the query was completed in 70ms. Groq has released API libraries for Python and Javascript, i wrote a simple Rust example here, of how to use the API [1].

Groq's API documents how long it takes to generate the tokens for each request. 70ms for a page of document, are well over 100 times faster than GPT, and the fastest of every other capable model. Accounting for internet's latency and some queue that might exist, then the user receives the request in a second, but how fast would this model run locally? Fast enough to generate natural language tokens, generate a synthetic voice, listen again and decode the next request the user might talk to it, all in real time.

With a technology like that, why not talk to internet services with just APIs and no web interface at all? Just functions exposed on the internet, take json as an input, validate it, and send the json back to the user? Or every other interface and button around. Why pressing buttons for every electric appliance, and not just talk to the machine using a json schema? Why should users on an internet forum, every time a comment is added, have to press the add comment button, instead of just talking and saying "post it"? Pretty annoying actually.

[1] https://github.com/pramatias/groq_test

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#109

Post by infecto »

mhuffman wrote: post #107
infecto wrote: post #96

Earlier quoted context omitted.

This should be higher up. This whole blog post is mostly worthless because the way they are extracting data is less than optimal. Lower end models do not have the attention to complete tasks like this, GPT4Turbo will generally have the capability. But to have an optimal pipeline you should really be splitting up these tasks into individual units. You extract each attribute you want independently and then combine it b…

Do you have an example of a more optimal prompt to share?

The prompt does not matter as much as the workflow which is describe above. 1) Extract one attribute at a time. 2) Don't ask for json during extraction, but on binary small attributes it might not matter as much.. 3) Combine the data later.

There are differences that can be marked on how different models perform against the same raw prompt but generally the workflow is what matters more. The raw text prompt will be dependent on what model you are using as there are those differences but I don't think its a level of "prompt engineering" like we had a year ago.

Re: OpenAI GPT-4 vs. Groq Mistral-8x7B

#110

Post by CaptainFever »

samlinnfer wrote: post #80
dns_snek wrote: post #56

For all the posturing and crypto hate on HN, we're entering a world where it's socially acceptable to use 1000W of computing power and 5 seconds of inference time to parse a tiny HTML fragment which would take microseconds with traditional methods - and people are cheering about it. Time for some self-reflection? That's not very green.

Any amount of energy spent useful work is vastly superior than whatever “POW” crypto burn does. >For all the posturing and forest fire hate on HN, it’s now socially acceptable to run a toy steam engine to power a model car? Not very green of you.

It's almost a fallacy at this point to declare something bad simply because of the existence of carbon emissions, without first comparing the benefits of what is being produced, and the alternative tradeoffs.

To be fair to GP, they did compare it to alternatives (dumb HTML parsing), but failed to consider versatile HTML parsing or other uses for Groq LLM.

Post reply on HN