Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Codex

openai.com

Post reply on HN
Print view View original

91–100 of 181 posts

Re: OpenAI Codex

#91

Post by stavros »

karmasimida wrote: post #59
f0e4c2f7 wrote: post #11

They just finished a demo on twitch. Pretty crazy! https://www.twitch.tv/videos/1114111652 Starts at 15:45.

It is simultaneously impressive and underwhelming for me. I mean yes this is a super impressive demo, but it didn't go beyond my expectation. I really want to see whether this model can write a correct binary search method without seeing one before. Or even correctly using the binary search, does it understand concept like index boundaries?

> I really want to see whether this model can write a correct binary search method without seeing one before.

I don't believe the model was trained on Google interview answers, sadly.

Re: OpenAI Codex

#93

Post by bufferoverflow »

maxwells-daemon wrote: post #45

The "language models don't really understand anything" corner is getting smaller and smaller. In the last few months we've seen pretty definitive evidence that transformers can recombine concepts ([1], [2]) and do simple logical inference using contextual information ([3], "make the score font color visible"). I see no reason that this technology couldn't smoothly scale into human-level intelligence, yet lots of peop…

It probably can scale, but we're nowhere near the computational power we need to even recreate the brain. And don't forget, our brain took a billion years to evolve.

A typical brain has 80-90 billion neurons and 125 trillion synapses. That's a big freaking network to train.

Hopefully we can figure out how to train parts of it and then assemble something very smart.

Re: OpenAI Codex

#94

Post by dmurray »

They should have released this first instead of GitHub Copilot. The focus would then have been much more on "look at the cool stuff they can do" rather than "Microsoft is releasing a product that plagiarizes GPL code".

Once people had digested that and there had been a few other proof-of-concept business ideas around turning Codex into a SaaS (because some people will always queue to build their product on your API), announce the evil version. Not that I really think Copilot is evil, but the IP concerns are legitimate.

Re: OpenAI Codex

#95

Post by visarga »

nightski wrote: post #76
hackinthebochs wrote: post #60

Earlier quoted context omitted.

Considering the sum total of data and computation that goes in to creating an intelligent human mind, including the forces of natural selection in creating our innate structure and dispositions, it's not obvious that any conclusions can be drawn from the fact that so much data and compute goes into training these models.

Has this transfer of knowledge from one domain to another really been demonstrated by these models/learning processes? I know transfer learning is a thing (I have a couple books on my shelf on it). But it seems far from what you are describing.

DALL-E + CLIP models show a deep understanding of the relation between images and text.

Re: OpenAI Codex

#96

Post by polyanos »

karmasimida wrote: post #59
f0e4c2f7 wrote: post #11

They just finished a demo on twitch. Pretty crazy! https://www.twitch.tv/videos/1114111652 Starts at 15:45.

It is simultaneously impressive and underwhelming for me. I mean yes this is a super impressive demo, but it didn't go beyond my expectation. I really want to see whether this model can write a correct binary search method without seeing one before. Or even correctly using the binary search, does it understand concept like index boundaries?

I found the whole UI/sandbox they created the most interesting part. Now don't get me wrong, the tech is certainly great and all, but I really didn't had the feeling I watched/learned more than I already knew from what was shown with Github CoPilot, although I was kinda impressed, if it really is as simple as they stated, at how it is able to adapt to new apis.

It's a shame they only limited the demo to relatively simple instructions.

Re: OpenAI Codex

#97

Post by lstmemery »

maxwells-daemon wrote: post #45

The "language models don't really understand anything" corner is getting smaller and smaller. In the last few months we've seen pretty definitive evidence that transformers can recombine concepts ([1], [2]) and do simple logical inference using contextual information ([3], "make the score font color visible"). I see no reason that this technology couldn't smoothly scale into human-level intelligence, yet lots of peop…

I have to disagree with you here. In the Codex paper[1], they have two datasets that Codex got correct about 3% of the time. These are interview and code competition questions. From the paper:

"Indeed, a strong student who completes an introductory computer science course is expected to be able to solve a larger fraction of problems than Codex-12B."

This suggests to me that Codex really doesn't understand anything about the language beyond syntax. I have no doubt that future systems will improve on this benchmark, but they will likely take advantage of the AST and could use unit tests in a RL-like reward function.

[1] https://arxiv.org/abs/2107.03374

Re: OpenAI Codex

#98

Post by pchiusano »

maxwells-daemon wrote: post #63
maxwells-daemon wrote: post #45

The "language models don't really understand anything" corner is getting smaller and smaller. In the last few months we've seen pretty definitive evidence that transformers can recombine concepts ([1], [2]) and do simple logical inference using contextual information ([3], "make the score font color visible"). I see no reason that this technology couldn't smoothly scale into human-level intelligence, yet lots of peop…

As an add-on to this: I'd encourage anyone interested in this debate to read Rich Sutton's "The Bitter Lesson" ( http://www.incompleteideas.net/IncIdeas/BitterLesson.html ). At every point in time, the best systems we can build today will be ones leveraging lots of domain-specific information. But the systems that will continue to be useful in five years will always be the ones freely that scale with increased parall…

https://rodneybrooks.com/a-better-lesson/ is an interesting retort to the Sutton post.

Re: OpenAI Codex

#99

Post by sbierwagen »

nightski wrote: post #76
hackinthebochs wrote: post #60

Earlier quoted context omitted.

Considering the sum total of data and computation that goes in to creating an intelligent human mind, including the forces of natural selection in creating our innate structure and dispositions, it's not obvious that any conclusions can be drawn from the fact that so much data and compute goes into training these models.

Has this transfer of knowledge from one domain to another really been demonstrated by these models/learning processes? I know transfer learning is a thing (I have a couple books on my shelf on it). But it seems far from what you are describing.

The AlphaZero algorithm swapped between board games pretty easily. OpenAI could also have been gesturing at this when they named the GPT paper "Language Models are Few-Shot Learners".

Re: OpenAI Codex

#100

Post by visarga »

6gvONxR4sf7o wrote: post #69
maxwells-daemon wrote: post #45

The "language models don't really understand anything" corner is getting smaller and smaller. In the last few months we've seen pretty definitive evidence that transformers can recombine concepts ([1], [2]) and do simple logical inference using contextual information ([3], "make the score font color visible"). I see no reason that this technology couldn't smoothly scale into human-level intelligence, yet lots of peop…

> The "language models don't really understand anything" corner is getting smaller and smaller. In my mind, understanding a thing means you can justify an answer. Like a student showing their work and being able to defend it. An answer with a proof understands the answer with respect to the proof it provides. E.g. to understand an answer with regards to first order logic, it'll have to be able to defend a logical ded…

> Maybe the next step will require increasingly interesting justification systems.

You can just ask it to comment what it intends to do. It's surprising actually.

Post reply on HN