Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

threads.net

Post reply on HN
Print view View original

121–130 of 134 posts

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#121

Post by wakaru44 »

danielmarkbruce wrote: post #108
simonw wrote: post #105

Earlier quoted context omitted.

For "citation needed", read "please link me to a blog post showing how, don't just tell me to Google for one". The internet is full of blog posts about this. That doesn't mean they're actually good - I'd love to be pointed at one that has proven itself useful for someone (and definitely isn't just LLM blog-spam). I don't care if it's trivial to fine-tune and get crap results - I care about fine-tuning where the resul…

It's an internet forum, not an academic journal. Water tight arguments are not needed. If one wants to call bs, they can just do it, no need to dance around the topic by asking for a citation.

Exactly, it's a forum not Twitter/reddit. Without references and citations this is no better than a bunch of random words, and it's hard to make any argument of substance.

The person asked for citations, leave it be, stop dudexplaining how Internet works for you please.

I call bs on your 2 comments.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#122

Post by newswasboring »

hlfshell wrote: post #120
gradascent wrote: post #111

Earlier quoted context omitted.

Interesting! I would like to learn more about how AI is being applied to robotics. Do you have any suggestions for how to keep up with developments/ideas in this field?

In October I wrote a blogpost on this subject: https://hlfshell.ai/posts/llms-and-robotics-papers-2023/ ..and plan to do an updated version soon for much of what's been released since. I've also done work related to LLM and robotics integration, also on that site. Happy to chat about it.

Working my way through your blog post and it is so refreshing. Unfortunately my algorithm currently is showing me takes which are extreme on either end (like in your blog post).

> Technology’s largest leaps occur when new tools are provided to those that want to make things.

I love this sentence. And the general attitude of curiosity of your post.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#123

Post by rubymamis »

fragmede wrote: post #84
rubymamis wrote: post #10

Earlier quoted context omitted.

A programming task where Mistral-large beats both GPT-4 and Claude Opus: https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 (only Mistral got the current syntax) Although based on other tasks, overall, GPT-4 seems to be the best, but by a very small margin, so I cancelled my subscription. Although the native mobile app is really great.

Is there a way to use Mistral-large with TTS and STT engines so you can converse with it like you can ChatGPT in the mobile app? it's really great on long drives for learning/talking about stuff, like a customized personal podcast.

Exactly, I absolutely love this feature. And many times the conversation is quite natural and fluid (with good internet connection). I think I'll build something like that myself (:

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#124

Post by rubymamis »

cosmojg wrote: post #95
rubymamis wrote: post #10

Earlier quoted context omitted.

A programming task where Mistral-large beats both GPT-4 and Claude Opus: https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 (only Mistral got the current syntax) Although based on other tasks, overall, GPT-4 seems to be the best, but by a very small margin, so I cancelled my subscription. Although the native mobile app is really great.

Do you prefer Mistral-Large or Claude-Opus?

Not sure. Most of the time GPT-4 is better. Since I'm using Vercel AI playground[1], on almost every query I get a response from all models so it's easy to compare.

[1] https://sdk.vercel.ai/

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#125

Post by FrustratedMonky »

TMWNN wrote: post #96
FrustratedMonky wrote: post #79

Earlier quoted context omitted.

"bitter lesson that building in how we think we think does not work in the long run" Guess. Stop trying to shape the NN. And let it learn on its own.

>And let it learn on its own. The best single work of fiction ever created about LLMs' capabilities (and, perhaps, dangers) is Colossus by Jones. Although I think the film is even better than the book, only the latter mentions how, despite being created specifically for US national defense, Colossus is also fed unrelated data including Shakespeare's sonnets, because its creators do not know if it could be important.

I think the movie Colossus still holds up today. Saw it last year, It was pretty scary.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#126

Post by Ambix »

marviel wrote: post #61
minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

Several MSFT AI/ML friends actively dissuaded me and my team from fine-tuning. They said that it's pretty clear in all their internal tests that it "lobotomizes" the general reasoning capabilities of the model, unless you're really careful. "All work and no play makes GPT a very dull AI"

Yes, that's what I've seen from a lot of my experiments with fine-tuning. One should be really careful to not "lobotomize" already capable model and achieve better results at the end. It's trickier than seems from multiple of tutorials.

But I believe that most of the data stored in foundation models are just useless for some particular domain. So it's better to forget something, getting really useful info instead.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#127

Post by danielmarkbruce »

wakaru44 wrote: post #121
danielmarkbruce wrote: post #108

Earlier quoted context omitted.

It's an internet forum, not an academic journal. Water tight arguments are not needed. If one wants to call bs, they can just do it, no need to dance around the topic by asking for a citation.

Exactly, it's a forum not Twitter/reddit. Without references and citations this is no better than a bunch of random words, and it's hard to make any argument of substance. The person asked for citations, leave it be, stop dudexplaining how Internet works for you please. I call bs on your 2 comments.

They didn't ask for citations. They pointed out a citation was needed. It was a clever sounding way of calling bs. They admit as much.

Even when people sincerely ask for a citation on a debatable topic, on an internet forum, it's effectively saying "I won't be hear any opinion that doesn't match my own unless it's as water tight as a law of physics". Another form of this is "show me the data".

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#128

Post by danielmarkbruce »

simonw wrote: post #113
danielmarkbruce wrote: post #108

Earlier quoted context omitted.

It's an internet forum, not an academic journal. Water tight arguments are not needed. If one wants to call bs, they can just do it, no need to dance around the topic by asking for a citation.

OK, I call BS. Fine-tuning an LLM is not "trivial" - especially if you want to get useful results, as opposed to just being able to say "look, I fine-tuned an LLM".

Yup, largely agree.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#129

Post by hlfshell »

newswasboring wrote: post #122
hlfshell wrote: post #120

Earlier quoted context omitted.

In October I wrote a blogpost on this subject: https://hlfshell.ai/posts/llms-and-robotics-papers-2023/ ..and plan to do an updated version soon for much of what's been released since. I've also done work related to LLM and robotics integration, also on that site. Happy to chat about it.

Working my way through your blog post and it is so refreshing. Unfortunately my algorithm currently is showing me takes which are extreme on either end (like in your blog post). > Technology’s largest leaps occur when new tools are provided to those that want to make things. I love this sentence. And the general attitude of curiosity of your post.

Thanks! Appreciate the kind words. I should have in the next month or so (interviewing and finishing my Master's, so there's been delays) a follow up that follows more advancements in the router style VLA, sensoiromotor VLM, and advances in embedding enriched vision models in general.

If you want a great overview of what a modern robotics stack would look like with all this, https://ok-robot.github.io/ was really good and will likely make it into the article. It's a VLA combined with existing RL methods to demonstrate multi-tasking robots, and serves as a great glimpes into what a lot of researchers are working on. You won't see these techniques in robots in industrial or commercial settings - we're still too new at this to be reliable or capable enough to deploy these on real tasks.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#130

Post by MuffinFlavored »

cosmojg wrote: post #93
MuffinFlavored wrote: post #12

Earlier quoted context omitted.

The LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)

Right, like the other commenter suggested, that's an infrastructure-level thing, not a model-level thing. Given that you're talking about ChatGPT, I assume you aren't accessing GPT-3.5 or GPT-4 directly through the API but using the app or the interface provided at chat.openai.com. The magic that makes the kinds of interactions you're describing possible amounts to a bit of clever prompting sprinkled on top of some r…

> Right, like the other commenter suggested, that's an infrastructure-level thing, not a model-level thing.

By default, the ChatGPT "model" knows to not try to do math and instead write code to do the math then run it. I get that it's set up infrastructure wise to be able to run it, but why is Claude's main chat UI not trying to instead respond

"hey, do this calculation on your own since I can't" or something of this nature instead of responding to math incorrectly

Post reply on HN