Live data from Hacker News

Alpaca RLHF-ed to beat ChatGPT

crfm.stanford.edu

21–30 of 45 posts

Re: Alpaca RLHF-ed to beat ChatGPT

#21
Whenever I see a claim about GPT I get temporarily interested until I learn it’s GPT3.5 and not GPT4.

4 isn’t just marginally better at most tasks I use it for, it’s operating at an entirely different level to the point where I have little (no?) day-to-day use of 3.5 at this point.

Re: Alpaca RLHF-ed to beat ChatGPT

#22

Hm. Title: “beats Chat GPT” Reality: > With these evaluation instructions, we compare RLHF model responses to Davinci003 responses and measure the fraction of times the RLHF model is preferred; we call this statistic the win-rate. > Of the methods we studied, PPO proves the most effective, improving the win-rate against Davinci003 from 44% to 55% according to human evaluation, which even outperforms ChatGPT. …for the…

Yeah, and I think they are using an old version (3.0? 3.5?) of ChatGPT, not GPT4, which is way better. Can anyone verify? They confusingly list GPT4 as a separate LLM, even though ChatGPT supports GPT4.

Re: Alpaca RLHF-ed to beat ChatGPT

#23

Whenever I see a claim about GPT I get temporarily interested until I learn it’s GPT3.5 and not GPT4. 4 isn’t just marginally better at most tasks I use it for, it’s operating at an entirely different level to the point where I have little (no?) day-to-day use of 3.5 at this point.

From what I see practically everyone is making this comparison and it is bs. As you stay, 4 is an entirely different beast to 3.5.

Re: Alpaca RLHF-ed to beat ChatGPT

#24

Whenever I see a claim about GPT I get temporarily interested until I learn it’s GPT3.5 and not GPT4. 4 isn’t just marginally better at most tasks I use it for, it’s operating at an entirely different level to the point where I have little (no?) day-to-day use of 3.5 at this point.

I'm assuming you use gpt4 via ChatGPT plus. Does the message cap bother you? I heard it's something like 25 messages per 3 hours. That sounds so low I don't even bother subscribing.

I guess this doesn't apply if you use it via the api.

Re: Alpaca RLHF-ed to beat ChatGPT

#25

Whenever I see a claim about GPT I get temporarily interested until I learn it’s GPT3.5 and not GPT4. 4 isn’t just marginally better at most tasks I use it for, it’s operating at an entirely different level to the point where I have little (no?) day-to-day use of 3.5 at this point.

I'm assuming you use gpt4 via ChatGPT plus. Does the message cap bother you? I heard it's something like 25 messages per 3 hours. That sounds so low I don't even bother subscribing. I guess this doesn't apply if you use it via the api.

It sounds very low and somehow it very rarely bothers me. It sure is annoying when it bothers me, but it's a lot higher in practice than the number feels.

Re: Alpaca RLHF-ed to beat ChatGPT

#26

Whenever I see a claim about GPT I get temporarily interested until I learn it’s GPT3.5 and not GPT4. 4 isn’t just marginally better at most tasks I use it for, it’s operating at an entirely different level to the point where I have little (no?) day-to-day use of 3.5 at this point.

I'm assuming you use gpt4 via ChatGPT plus. Does the message cap bother you? I heard it's something like 25 messages per 3 hours. That sounds so low I don't even bother subscribing. I guess this doesn't apply if you use it via the api.

It does bother me, I’ve been hit by it 3 times now (I use it as a daily driver, for code you spend enough time between prompts working that it’s rare to go through that volume)

When I hit the limit, I work on the problem myself and wait until 4 resets instead of relying on 3.5. 4 is so much better that I don’t trust 3.5 with my work anymore.

Re: Alpaca RLHF-ed to beat ChatGPT

#27

Whenever I see a claim about GPT I get temporarily interested until I learn it’s GPT3.5 and not GPT4. 4 isn’t just marginally better at most tasks I use it for, it’s operating at an entirely different level to the point where I have little (no?) day-to-day use of 3.5 at this point.

I'm assuming you use gpt4 via ChatGPT plus. Does the message cap bother you? I heard it's something like 25 messages per 3 hours. That sounds so low I don't even bother subscribing. I guess this doesn't apply if you use it via the api.

I was initially deterred. But in practice, when using it for professional purposes, I never encounter it.

Your coding speed is unlikely to be that fast, requiring 25 code segments in 3 hours. GPT-4 outputs something, you need time to double check, test, additional googling etc. Its still a massive speed boost.

Using it recreationally (Especially chatting) will result in a lot more requests.

Re: Alpaca RLHF-ed to beat ChatGPT

#28

Whenever I see a claim about GPT I get temporarily interested until I learn it’s GPT3.5 and not GPT4. 4 isn’t just marginally better at most tasks I use it for, it’s operating at an entirely different level to the point where I have little (no?) day-to-day use of 3.5 at this point.

I'm assuming you use gpt4 via ChatGPT plus. Does the message cap bother you? I heard it's something like 25 messages per 3 hours. That sounds so low I don't even bother subscribing. I guess this doesn't apply if you use it via the api.

I've never hit the limit because GPT 4 is slow (like a dialup modem) and I don't like waiting for it. Usually I do something else while it's writing a response.

I haven't used it a whole lot.

Re: Alpaca RLHF-ed to beat ChatGPT

#29

Earlier quoted context omitted.

I'm assuming you use gpt4 via ChatGPT plus. Does the message cap bother you? I heard it's something like 25 messages per 3 hours. That sounds so low I don't even bother subscribing. I guess this doesn't apply if you use it via the api.

I was initially deterred. But in practice, when using it for professional purposes, I never encounter it. Your coding speed is unlikely to be that fast, requiring 25 code segments in 3 hours. GPT-4 outputs something, you need time to double check, test, additional googling etc. Its still a massive speed boost. Using it recreationally (Especially chatting) will result in a lot more requests.

The only time I hit it I usually realize I need a mental break anyway so usually it’s plenty. Suppose it depends on how your using it but for me I ask it for code of things I could write but would rather focus my energy on the bigger problem then a single function to merge two objects while keeping the order sorted of a joined list.. that kind of thing it’s great for

Re: Alpaca RLHF-ed to beat ChatGPT

#30

Earlier quoted context omitted.

I wonder how long "Using humans to rate the quality of other humans" thing can last. Surely academia has only so long before it collapses.

You're asserting that current LLMs are as capable as evaluating each other as are humans with advanced degrees?

Also humans aren't exact clones
Post reply on HN