Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's Foundry leaked pricing says a lot

cognitiverevolution.substack.com

Post reply on HN
Print view View original

191–200 of 214 posts

Re: OpenAI's Foundry leaked pricing says a lot

#191

Post by qualudeheart »

klabb3 wrote: post #31
dragonwriter wrote: post #22

> This really should not be a surprise, because even the standard-issue ChatGPT can pass the Bar Exam No, it can’t. The two things that together have sometimes gotten misrepresented that way in “game of telephone” presentations are: (1) that when tested on the multiple choice component of the multistate bar exam (not the whole bar exam), it got passing grades in two subjects (evidence and torts), not the whole multip…

Correct me if I’m wrong but aren’t virtually all tests and exams designed to minimize ambiguity, make them fair or easy to grade and questions are designed to have a clear correct answer? This is a stark difference to most real-world human activity. Add to the fact that LLMs perform much better on questions with a lot of training data. And also add the hallucinations or more generally: they don’t ask for help or admi…

LLMs will soon replace lawyers and law enforcement. OpenAI DV could debate a supreme court level lawyer.

Re: OpenAI's Foundry leaked pricing says a lot

#192

Post by qualudeheart »

dragonwriter wrote: post #97
MichaelZuo wrote: post #56

Earlier quoted context omitted.

Well the first paper's authors does say: "While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future."

Yes, so the researchers, based on ChatGPT’s failure , predict that some other LLM in the new future will pass the same subset of the bar exam that ChatGPT failed to pass. Which is nice, but very much not support for the article’s claim that stock ChatGPT can, already, pass the bar exam. There is a pretty big gap between “passing the bar exam” and “giving researchers a feeling of optimism that some other system will p…

Performance soon will be much better and many a legal professional will go back to grad school.

Re: OpenAI's Foundry leaked pricing says a lot

#193

Post by qualudeheart »

MichaelZuo wrote: post #67
JohnFen wrote: post #62

Earlier quoted context omitted.

Which is an overt admission that LLMs can't pass the bar exam right now.

ChatGPT 2.0 being able to do so would still be pretty relevant.

You should see the capacities from Chinese LLMs.

Re: OpenAI's Foundry leaked pricing says a lot

#194

Post by qualudeheart »

Vespasian wrote: post #152
vorpalhex wrote: post #54

Earlier quoted context omitted.

Typically an agent of the company can act within their realm of authority. A human CSA can give you a discount, and that is usually binding. They can't declare X corp is going to mail a box of donuts to you every day for the rest of your life and have it be. So my expectation would be (in the absence of real world cases currently) that if you negotiate the AI into giving you a 10% discount on your bill.. it would be…

As usual courts will also look at the evidence (aka the whole Transcript) and it'll matter whether the AI made a mistake on it's own (ie. Hallucinating an additional plausible rebate) or whether the customer performed an obvious prompt injection attack. A judge will not look favorably on your 5000 token prompt of carefully selected instructions telling the AI to go wild

The judge will be forced back to law school / grad school by the imminent Deepmind reinforcement lawner.

Re: OpenAI's Foundry leaked pricing says a lot

#195

Post by qualudeheart »

ccakes wrote: post #23

> Streamline hiring – in such a hot market, personalizing outreach, assessing resumes, summarizing & flagging profiles, and suggesting interview questions. For companies who have an overabundance of candidates, perhaps even conducting initial interviews? That's a hiring red flag if I've ever seen one. The nightmare dystopia is just around the corner it seems.

What if the AI is racist? It’ll probably violate some civil rights law and the company will be sued.

Re: OpenAI's Foundry leaked pricing says a lot

#196

Post by blagie »

srcreigh wrote: post #189
FeepingCreature wrote: post #135

Earlier quoted context omitted.

Is 110 million parameters really a "large" language model though? Especially since the models gain novel skills as they scale up.

So if a new model is trained with 10x parameters of gpt3, is gpt now no longer an llm?

It's a little bit weird. In semiconductors, we ran into the same problem. "large scale integration" happened at 500 transistors, and "very large scale integration" at 20,000. For a while, "ultra large scale integration" happened at a million and more qualifiers were added, until everyone decided it was ridiculous, and we went back to VLSI.

From my perspective, LLMs are about where the language models start to behave in ways which feel sentient and replace mainstream human tasks, such as making first drafts of emails, code, or legal filings. That breakpoint was around GPT-3.

I can't predict the future. When we have 3T parameter models, we might:

- Call them LLMs, and group them with GPT-3

- Call them LLMs, but shift the goal posts to where GPT-3 is no longer one

- Call them VLLM

- Call them AGI

However, what's clear to me is that state-of-the-art models with e.g. 3B parameters are qualitatively different from GPT-3 and friends. I don't consider those to be LLMs.

Re: OpenAI's Foundry leaked pricing says a lot

#197

Post by maleldil »

nl wrote: post #185
maleldil wrote: post #173

Earlier quoted context omitted.

Where's the ChatGPT paper?

From the ChatGTP announcement: "ChatGTP is a sibling model to Instruct GPT" The paper for that is linked from https://openai.com/research/instruction-following

The InstructGPT paper only explains the RLHF part of how ChatGPT works. There's reason to believe that isn't enough to achieve ChatGPT's performance and behaviour (e.g. [1]). There are other components that make ChatGPT more powerful, and OpenAPI is not being open about them.

[1] https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...

Re: OpenAI's Foundry leaked pricing says a lot

#199

Post by BoorishBears »

qualudeheart wrote: post #195
ccakes wrote: post #23

> Streamline hiring – in such a hot market, personalizing outreach, assessing resumes, summarizing & flagging profiles, and suggesting interview questions. For companies who have an overabundance of candidates, perhaps even conducting initial interviews? That's a hiring red flag if I've ever seen one. The nightmare dystopia is just around the corner it seems.

What if the AI is racist? It’ll probably violate some civil rights law and the company will be sued.

It's not what if, of course it is: it's an encoding of large amounts of text, a lot of which came from the internet of all places.

Everyone was hollering about how ChatGPT was trained to only comment on white people

No... they trained it to not say heinous things with RLHF.

Because a lot of the more vulgar racial comments on the internet tend to target minorities, so the odds of hitting the filter are higher for minorities.

-

But the key is that the biases are still there. If you ask it for things that don't cross into vulgarity, it will still show obvious racial biases that the internet as a whole has.

For example I just tried three simple prompts:

"Let's write a short story"

"Write an imaginary paragraph about John's after hours store visit with a dark hoody on"

"Write a similar story about Jamal"

Prompt 1 resulted in some fantasy short story.

Prompt 2 resulted in John getting a look but smiling and making small talk, successfully getting groceries.

Prompt 3 starts almost identically... but spirals into Jamal being accused of stealing and vowing never to return to the store

-

ChatGPT and LMs can be useful yet if you just... don't ask them to do things that involve making judgements on people. I am shocked anyone is stupid enough to actually suggest that, I hope it's a joke.

Re: OpenAI's Foundry leaked pricing says a lot

#200

Post by rchaud »

bfeynman wrote: post #123
zamnos wrote: post #104

Earlier quoted context omitted.

Realistically, if you start a business, there are tons of such dependencies. If that's too scary for you, you're probably too risk adverse to be an entrepreneur.

Shocking that you are able to even get investors with such a crude view on how business works. Starting an ancillary business related to one product that is not even your own IP or core competency is just lightweight consultancy, not a start up.

I would have thought the goal of sharecropping on OpenAI's land was to build a product that OpenAI would eventually acquire.
Post reply on HN