Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

151–160 of 512 posts

Re: Large Enough

#151

Post by wesleyyue »

ijustlovemath wrote: post #146
wesleyyue wrote: post #138

I'm building a ai coding assistant ( https://double.bot ) so I've tried pretty much all the frontier models. I added it this morning to play around with it and it's probably the worst model I've ever played with. Less coherent than 8B models. Worst case of benchmark hacking I've ever seen. example: https://x.com/WesleyYue/status/1816153964934750691

What was the expected outcome for you? AFAIK, Python doesn't have a const dictionary. Were you wanting it to refactor into a dataclass?

Yes, there's a few things wrong: 1. If it assumes typescript, it should do `as const` in the first msg 2. If it is python, it should be something like https://x.com/WesleyYue/status/1816157147413278811 which is what I wanted but I didn't want to bother with the typing.

Re: Large Enough

#152

Post by ThinkBeat »

A side note about the ever increasing costs to advance the models. I feel certain that some branch of what may be connected to the NSA is running and advancing models that probably exceed what the open market provides today.

Maybe they are running it on proprietary or semi proprietary hardware but if they dont, how much does the market no where various shipments of NVIDEA processors ends up?

I imagine most intelligence agencies are in need of vast quantities.

I presume is M$ announces new availability of AI compute it means they have received and put into production X Nvidiam, which might make it possible to guesstimate within some bounds how many.

Same with other open market compute facilities.

Is it likely that a significant share of NVIDEA processors are going to government / intelligent / fronts?

Re: Large Enough

#153

Post by moffkalast »

RyanAdamas wrote: post #134

Personally, language diversity should be the last thing on the list. If we had optimized every software from the get-go for a dozen languages our forward progress would have been dead in the water.

You'd think so, but 3.5-turbo was multilingual from the get go and benefitted massively from it. If you want to position yourself as a global leader, then excluding 95% of the world who aren't English native speakers seems like a bad idea.

Re: Large Enough

#154

Post by omneity »

rkwasny wrote: post #90

All evals we have are just far too easy! We need to figure out how to measure intelligence that is greater than human.

Give it problems most/all humans can't solve on their own, but that are easy to verify.

Math problems being one of them, if only LLMs were good at pure math. Another possibility is graph problems. Haven't tested this much though.

Re: Large Enough

#157

Post by RyanAdamas »

moffkalast wrote: post #153
RyanAdamas wrote: post #134

Personally, language diversity should be the last thing on the list. If we had optimized every software from the get-go for a dozen languages our forward progress would have been dead in the water.

You'd think so, but 3.5-turbo was multilingual from the get go and benefitted massively from it. If you want to position yourself as a global leader, then excluding 95% of the world who aren't English native speakers seems like a bad idea.

Yeah clearly, OpenAI is rocketing forward and beyond.

Re: Large Enough

#158

Post by RicoElectrico »

lolinder wrote: post #129
Liquix wrote: post #10

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.

> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs Yes. This is exactly why I'm skeptical of AI doomerism/saviorism. Too many people have been looking at the pace of LLM development over the last two (2) years, modeled it as an exponential growth function, and come to the conclusion that AGI is inevitable in the next ${1-5} years and we're headed for ${(dys|u)topia}. B…

I don't think we are approaching limits, if you take off the English-centric glasses. You can query LLMs about pretty basic questions about Polish language or literature and it's gonna either bullshit or say it doesn't know the answer.

Example:

    w której gwarze jest słowo ekspres i co znaczy?

    Słowo "ekspres" występuje w gwarze śląskiej i oznacza tam ekspres do kawy. Jest to skrót od nazwy "ekspres do kawy", czyli urządzenia służącego do szybkiego przygotowania kawy.
The correct answer is that "ekspres" is a zipper in Łódź dialect.

Re: Large Enough

#159

Post by layer8 »

Spivak wrote: post #95
layer8 wrote: post #62

Earlier quoted context omitted.

It’s not impressive that one has to go to that length though.

To me it's just a limitation based on the world as seen by these models. They know there's a letter called 'r', they even know that some words start with 'r' or have r's in them, and they know what the spelling of some words is. But they've never actually seen one in as their world is made up entirely of tokens. The word 'red' isn't r-e-d but is instead like a pictogram to them. But they know the spelling of strawber…

The great-parent demonstrates that they are nevertheless capable of doing so, but not without special instructions. Your elaboration doesn’t explain why the special instructions are needed.

Re: Large Enough

#160

Post by wesleyyue »

mpeg wrote: post #142
wesleyyue wrote: post #138

I'm building a ai coding assistant ( https://double.bot ) so I've tried pretty much all the frontier models. I added it this morning to play around with it and it's probably the worst model I've ever played with. Less coherent than 8B models. Worst case of benchmark hacking I've ever seen. example: https://x.com/WesleyYue/status/1816153964934750691

to be fair that's quite a weird request (the initial one) – I feel a human would struggle to understand what you mean

definitely not an articulate request, but the point of using these tools is to speed me up. The less the user has to articulate and the more it can infer correctly, the more helpful it is. Other frontier models don't have this problem.

Llama 405B response would be exactly what I expect

https://x.com/WesleyYue/status/1816157147413278811

Post reply on HN