Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What it feels like to work in AI right now

robotic.substack.com

Post reply on HN
Print view View original

51–60 of 768 posts

Re: What it feels like to work in AI right now

#51

Post by mewpmewp2 »

nicpottier wrote: post #32
UncleOxidant wrote: post #25

Earlier quoted context omitted.

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

Until we see otherwise, don't we have to assume there's some secret sauce? Bard doesn't match GPT4 and it isn't for a lack of trying. (though perhaps that will change, so far that's the case)

Maybe it is just time to train and also good training data for prompts which openai has gathered for so long already? E.g. there is a bottleneck on how fast you can train and also gather good data.

Re: What it feels like to work in AI right now

#52

Post by dragonwriter »

diego wrote: post #49
UncleOxidant wrote: post #25

Earlier quoted context omitted.

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

If you try Bard or Claude or character.ai they are not far behind GPT4. They might even be on par in terms of raw LLM capabilities. ChatGPT has better marketing and in some cases better UX. A lot of this is self-fulfilling. We think it's far ahead, so it appears to be far ahead.

> If you try Bard or Claude or character.ai they are not far behind GPT4

Bard is way behind ChatGPT with GPT-3.5, much less GPT-4. Haven’t tried the others, though.

OTOH, that’s way behind qualitatively, not in terms of time-of-progress. So I don’t think it is at all an insurmountable lead, as much as it is a big utility gap.

Re: What it feels like to work in AI right now

#53

Post by JohnFen »

davesque wrote: post #28

I feel like the whole ChatGPT bubble has really highlighted what feel like some fundamental shortcomings in the worldview that is represented in tech. That is, there seems to be a winner take all dynamic baked into the tech world. Maybe this arises from the simple fact that tech places a lot of power in the hands of individuals. But there's an emergent downside to this which is that it makes those who were already mu…

> there seems to be a winner take all dynamic baked into the tech world.

This has been a pathology in the computer/software industry for a very long time. It's never been actually true except in a couple of special cases, but the industry acts as if it is. That has led to all sorts of bizarre and undesirable things.

> Guys, what the heck are we doing??

I think we're playing with fire and, without extreme caution and careful consideration (which I'm not seeing much of), this could end very, very badly for both the industry and society.

I have always been optimistic about technology and society, but (perhaps like you), my optimism has largely evaporated over the last several weeks. I wish the future didn't look so dark. Perhaps, though, things will look less gloomy with time.

Re: What it feels like to work in AI right now

#54

Post by JumpCrisscross »

UncleOxidant wrote: post #25
fdgsdfogijq wrote: post #12

I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that underperform ChatGPT by a huge margin. Twiddling my thumbs and keeping quiet while no one wants to recognize the elephant in the room. Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API.

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

> Why do you think OpenAPI is so far out in front?

There is a network effect forming around its models. The strengths of its kit speak for themselves. (It also cannot be understated how making ChatGPT public, something its competitors were too feeble, incompetent and behind the curve to do, dealt OpenAI a massive first-mover advantage.)

But as others note, other models are in the ballpark. Where OpenAI is different is in the ecosystem of marketing literature, contracts, code and e.g. prompt engineers being written and trained with GPT in mind. That introduces a subtle switching cost, and not-so-subtle platform advantage, that–barring a Google-scale bout of incompetence–OpenAI is set to retain for some time.

Re: What it feels like to work in AI right now

#55

Post by karmakurtisaani »

lacker wrote: post #17

I think this depends a lot on where you are working. I've talked to academics who are getting discouraged that they don't see how their approach to AI is going to be possible any more, with so much funding going toward the largest models from industry. On the other hand I've talked to startup founders building AI products whose business is booming because ChatGPT brought so much attention to the entire space. We are…

A PhD student I recently talked to was complaining how in academia they will never have the resources it takes to train models like GPT. And how now some academics are only researching the input/outputs to these LLMs. Seems like it's pretty dire or at least uninspiring to all but very few at the moment.

Re: What it feels like to work in AI right now

#57

Post by comeonbro »

pgwhalen wrote: post #4
version_five wrote: post #2

Huge crypto vibes Every single person I know working in AI these days ... has been sparked by the ChatGPT moment.

I'm curious why you draw the parallel - did crypto ever have a ChatGPT/iPhone moment? As an observant outsider, my impression is that every "moment" of note in crypto has been about some price going way up or way down - there has never been a "this is the use case" moment.

[flagged]

Re: What it feels like to work in AI right now

#58

Post by black3r »

UncleOxidant wrote: post #25
fdgsdfogijq wrote: post #12

I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that underperform ChatGPT by a huge margin. Twiddling my thumbs and keeping quiet while no one wants to recognize the elephant in the room. Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API.

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

For a little more than a year I worked in an AI startup doing basically everything other than AI (APIs, webapps, devops...), but from what I've seen there the "secret sauce" to AI success is the training process (dataset, parameters, fine-tuning steps, ...). And OpenAI isn't open about theirs since their beginnings.

Re: What it feels like to work in AI right now

#59

Post by logicallee »

UncleOxidant wrote: post #25
fdgsdfogijq wrote: post #12

I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that underperform ChatGPT by a huge margin. Twiddling my thumbs and keeping quiet while no one wants to recognize the elephant in the room. Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API.

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

All of the neural network architecture for human level thinking and processing, including vision, speech, emotion, abstract thought, balance and fine motor skills, everything was publicly released in April 2003, twenty years ago this month. It's a 700 megabyte tarball and sets up an 80b parameter neural network.

What? Huh? Yes the human genome encodes all human level thought.[1] Clearly it does because the only difference between humans that have abstract thought as well as language capabilities and primates that don't is slightly different DNA.

In other words: those slight differences matter.

To anyone who has used GPT since ChatGPT's public release in November and who pays to use GPT 4 now, it is clear that GPT 4 is a lot smarter than 3 was.

However, to the select few who see an ocean in a drop of water, the November release already showed glimmers of abstract thought, many other people dismiss it as an illusion.

To a select few, it is apparent that OpenAI have found the magic parameters. Everything after that is just fine tuning.

Is it any surprise that without OpenAI releasing their weights, models, or training data, Google can't just come up with its own? Why should they when without turning it into weights and models, the human neural network architecture itself is still unmatched (even by OpenAI) despite being digitized twenty years ago?

No, it's no surprise. OpenAI performed what amounts to a miracle, ten years ahead of schedule, and didn't tell anyone how they did it.

If you work for another company, such as Google, don't be surprised that you are ten years behind. After all, the magic formula had been gathering dust on a CD-ROM for 20 years (human DNA which encodes the human neural network architecture), and nobody made the slightest tangible progress toward it until OpenAI brute forced a solution using $1 billion of Azure GPU's that Microsoft poured into OpenAI in 2019.

Is your team using $1 billion of GPU's for 3 years? If not, don't expect to catch up with OpenAI's November miracle.

p.s. two months after the November miracle, Microsoft closed a $10 billion follow-on investment in OpenAI.

[1] https://en.m.wikipedia.org/wiki/Human_Genome_Project

Re: What it feels like to work in AI right now

#60

Post by jasmer »

version_five wrote: post #2

Huge crypto vibes Every single person I know working in AI these days ... has been sparked by the ChatGPT moment.

Crypto had the disadvantage of being a natural conduit for MLM-ish pyramid schemes and a lot of nefarious activities. That gave it some wind, but it was bad wind. Notably, it turned out to be not very useful. AI will be useful, and it's just regular tech, so there's good reason to be optimistic this time.
Post reply on HN