Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

471–480 of 512 posts

Re: Large Enough

#471

Post by Zuiii »

rkwz wrote: post #67
ActorNightly wrote: post #25

Earlier quoted context omitted.

The thing I don't understand is why everyone is throwing money at LLMs for language, when there are much simpler use cases which are more useful? For example, has anyone ever attempted image -> html/css model? Seems like it be great if I can draw something on a piece of paper and have it generate a website view for me.

Perhaps if we think of LLMs as search engines (Google, Bing etc) then there's more money to be made by being the top generic search engine than the top specialized one (code search, papers search etc)

This is the real PVP of LLM for me. Compressing google search AND the internet into 8 GB and download is something unfathomable to me a two decades ago.

My hope now is that someone will figure out a way to separate intelligence from knowledge - i.e. train a model that knows how to interpret the wights of other models - so that training new intelligent models wouldn't require training them on a petabyte of data every run.

Re: Large Enough

#472

Post by lucubratory »

GenerWork wrote: post #186
Kuinox wrote: post #49

Earlier quoted context omitted.

Tokenization make it hard for it to count the letters, that's also why if you ask it to do maths, writing the number in letters will yield better results. for strawberry, it see it as [496, 675, 15717], which is str aw berry. If you insert characters to breaks the tokens down, it find the correct result: how many r's are in "s"t"r"a"w"b"e"r"r"y" ? > There are 3 'r's in "s"t"r"a"w"b"e"r"r"y".

>If you insert characters to breaks the tokens down, it find the correct result: how many r's are in "s"t"r"a"w"b"e"r"r"y" ? The issue is that humans don't talk like this. I don't ask someone how many r's there are in strawberry by spelling out strawberry, I just say the word.

>I don't ask someone how many r's there are in strawberry by spelling out strawberry, I just say the word.

You don't ask a human being how many r's there are in strawberry at all. The only reason you or anyone else asks that question is because it's an interesting quirk of how LLMs work that they struggle to answer it in that format. It's like an alien repeatedly showing humans an optical illusion that relies on the existence of our (literal) blind spot and using it as evidence of our supposed lack of intelligence.

Re: Large Enough

#473

Post by takumif »

ipython wrote: post #416
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

I really wanted to believe this could be simply fixed by more prompt engineering and chain of thought, so I tried it with Claude 3.5 and … well unfortunately still no luck. the reply is amusing - I’ll include it verbatim below. Of note, this happened because earlier I had asked it without CoT and it incorrectly stated two “r”s. Once I started a new conversation (so fresh context window), it gave the right answer (3)…

For these classes of problems that LLMs struggle with, a more reliable way to go about them seems to be to ask them to solve them using tools, e.g. writing and executing a Python script to count the "R"s.

Re: Large Enough

#474

Post by bugglebeetle »

bionhoward wrote: post #460
bugglebeetle wrote: post #442

Earlier quoted context omitted.

Switch to Claude. I haven’t used ChatGPT for coding at all since they release Sonnet 3.5.

yeah but you can’t use your code from either model to compete with either company, and they do everything. wtf is wrong with AI hype enjoyers they accept being intellectually dominated?

If you think this is enforceable, I’ve got a bridge to sell you.

Re: Large Enough

#475

Post by troupo »

sqeaky wrote: post #425
troupo wrote: post #115

Earlier quoted context omitted.

> this is peer reviewable It's not peer-reviewable in any shape or form.

Others can build models that try to have decent performance with a lower number of parameters. If they match what is in the paper that is the crudest form of review, but Mistral is releasing some models (this one?) so this can get more nuanced if needs. That said, doing that is slow and people will need to make decisions before that is done.

So, the best you can do is "the crudest form of review"?

Re: Large Enough

#476

Post by qwertox »

nickthesick wrote: post #365
qwertox wrote: post #318

Earlier quoted context omitted.

Claude needs to fix their text input box. It tries to be so advanced that code in backticks gets reformatted, and when you copy it, the formatting is lost (even the backticks).

They are using Tiptap for their input and just a couple of days ago we called them out on some perf improvements that could be had in their editor: https://news.ycombinator.com/item?id=41036078 I am curious what you mean by the formatting is lost though?

Odd, multiline backtick code works very good, I don't know why I thought that it was also broken.

When you type "test `foo` done" in the editor, it immediately changes `foo` into a wrapped element. When you then copy the text without submitting it, then the backticks are lost, losing the inline-code formatting. I thought that this could also happen to multiline code. Somehow it does.

Type the following:

   Test: ```
   def foo():
      return bar
   ```
Delete that and type

   Test:
   ```
      def foo():
         return bar
   ```
   done
In the first case, the ``` in the line "Test: ```" does not open the code block, this happens with the second backtics. Maybe that's the way markdown works.

In the second case, all behaves normally, until you try to copy what you just wrote into the clipboard. Then you end up with

   Test:

   def foo():
      return bar

   done
Ok, only the backticks are lost but the formatting is preserved.

I think I have been trained by OpenAI to always copy what I submit before submitting, because it sometimes loses the submitted content, forcing me to re-submit.

Re: Large Enough

#477

Post by rldjbpin »

the way these models are being pushed, it seems like more of one-upping each other through iterative improvements than actual breakthroughs.

these benchmarks are as good as random hardware ones apple or intel pushes to sell their stuff. in the real world, most people will end up with some modifications for their specific use case anyways. for those, i argue, we already have "capable enough" models for the job.

Re: Large Enough

#478

Post by diffeomorphism »

mattnewton wrote: post #130
layer8 wrote: post #62

Earlier quoted context omitted.

It’s not impressive that one has to go to that length though.

You can always find something to be unimpressed by I suppose, but the fact that this was fixable with plain english is impressive enough to me.

The problem is that the models hallucinate too confidently. In this case it is quite amusing (I had llama3.1:8b tell me confidently it is 1, then revise to 2, then apologize again and give the correct answer). However, while it is obvious here, having it confidently make up supposed software features from thin air when asking for "how do I ..." is more problematic. The answers sound plausible, so you actually waste time verifying whether they work or are nonsense.

Re: Large Enough

#479

Post by lostmsu »

Chilko wrote: post #438
lostmsu wrote: post #332

Earlier quoted context omitted.

Just don't listen to anecdata, and use objective metrics instead: https://chat.lmsys.org/?leaderboard

Anecdata seems quite valid for LLM comparison when trying to evaluate 'usefullness' for users. The lmsys chat leaderboard is literally just mass anecdata.

Yes, "mass anecdata" + blind collection is usually called "data".

Re: Large Enough

#480

Post by tcgv »

IncreasePosts wrote: post #303
tcgv wrote: post #201

Earlier quoted context omitted.

Chain-of-Thought (CoT) prompting to the rescue! We should always put some effort into prompt engineering before dismissing the potential of generative AI.

Why doesn't the model prompt engineer itself?

Because it is a challenging task, you would need to define a prompt (or a set of prompts) that can precisely generate chain-of-thought prompts for the various generic problems the model encounters.

And sometimes CoT may not be the best approach. Depending on the problem other prompt engineering techniques will perform better.

Post reply on HN