Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What it feels like to work in AI right now

robotic.substack.com

Post reply on HN
Print view View original

101–110 of 768 posts

Re: What it feels like to work in AI right now

#101

Post by flangola7 »

redox99 wrote: post #47
UncleOxidant wrote: post #25

Earlier quoted context omitted.

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

I don't know about GPT4, but GPT3.5 I'd bet is pretty traditional and boring. It's power comes from a really good, properly curated dataset (including the RLHF). GPT3.5 turbo is much more interesting probably, because they seem to have found out how to make it much more efficient (some kind of distillation?). GPT4 if I had to make a very rough guess, probably flash attention, 100% of the (useful) internet/books for i…

According to financial reports they are building a $225 million supercomputer for AI. What we can probably expect is the same dataset with even more compute ran on it.

Re: What it feels like to work in AI right now

#103

Post by Animats »

dragonwriter wrote: post #50
saynay wrote: post #44

Earlier quoted context omitted.

Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did. I just think Google has far more to lose than Bing, so they are being more cautious.

> Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did Bard is overtly a reduced-resources model compared to the best version of the same technology (which, if true, is probably a boneheadedly bad choice for a public demo when everyone is already wowed by the people who got theirs out first, but easily explains that disparity. Though so does “guy who wanted public attentio…

> Bard is overtly a reduced-resources model compared to the best version of the same technology

There's a scaling problem. ChatGPT/LLM systems cost far more to run per query than the Google search engine. Google can't afford to make those the first line query reply.

A big business model question is whether Google will insist you be logged in to get to the large language model.

At Google scale, these things are going to have to be a hierarchy. Not everything needs to go to a full LLM system. Most Google queries by volume can be answered from a a cache.

Re: What it feels like to work in AI right now

#104

Post by theturtletalks »

redox99 wrote: post #47
UncleOxidant wrote: post #25

Earlier quoted context omitted.

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

I don't know about GPT4, but GPT3.5 I'd bet is pretty traditional and boring. It's power comes from a really good, properly curated dataset (including the RLHF). GPT3.5 turbo is much more interesting probably, because they seem to have found out how to make it much more efficient (some kind of distillation?). GPT4 if I had to make a very rough guess, probably flash attention, 100% of the (useful) internet/books for i…

Is there a limit on how big the context size can be?

Re: What it feels like to work in AI right now

#105

Post by ImaCake »

the_lonely_road wrote: post #43
hvs wrote: post #26

Earlier quoted context omitted.

It has helped me learn things more interactively because I can ask questions and get answers without having to read 10 blog posts or a book before understanding things. For my personality, it would be hard to recreate without having a living mentor available 24/7 that had knowledge of everything I wanted to ask of it. Does it get things wrong? Sure. But generally the subtle ways it might get stuff wrong would be the…

You must have missed the article posted this morning where ChatGPT complexity invented a sexual allegation against a law professor including citing a fictional news article that never existed. I wouldn’t call that a subtle misunderstanding.

Yes, chatGPT hallucinates facts and is not great at causal inference chains. The interactivity is the real boon as what I can do is query the chat bot, and then explore the results either with traditional research or with more queries and exploration. It's a really fun and productive way to learn.

Re: What it feels like to work in AI right now

#106

Post by SamvitJ »

"Every single person I know working in AI these days (in both the academy and industry) has been sparked by the ChatGPT moment."

This does not ring true to me at all. Anecdotally, it feels the rush to get into AI (in both industry and academia, for both individuals and organizations) peaked around 2016-2020, post-AlexNet/ResNet, around the time Transformers became very popular. Hiring for ML research roles in particular definitely slowed down in 2021, and 2022 of course saw a broader course correction across all of tech.

That said, I do agree that ChatGPT may be the "first iPhone moment of AI", in that is the first mainstream, end-user application of deep learning that millions of people have really engaged with.

Re: What it feels like to work in AI right now

#107

Post by iamleppert »

xyzzy123 wrote: post #92
pixl97 wrote: post #75

Earlier quoted context omitted.

I am still wondering if Google is experiencing it Kodak moment. They have the dominant product that makes them billions and billions of dollars at 'relatively' low cost. The new dominant product is on its way, but it costs far more to operate and will net them far less money, so... um no one wants to kill the goose that is still laying golden eggs, even though its days are numbered already.

I think it's not just run cost - there's also a risk of ad revenue destruction, that has to be worrying.

100% this -- a Google search requires multiple input queries, refinement, and scrolling a list of possible answers that are really just links to other web sites. The ChatGPT experience is far superior to this, for the average consumer and getting close for the power user. It's a better way to ask the Internet what it knows with a more natural interface that everyone already knows how to use -- real natural language. Less cognitive overhead, no busy search results that require clicking back and forth and (for now) no ads. That last part is key -- ChatGPT is doing Google's job right now, and not even having to run ads. Google doesn't even offer a premium no ads option for search and if they did I doubt enough people would buy it anyway to matter.

If I was Google I'd be worried. Very worried indeed. They either need to dramatically change their entire company within 18 months, or accept they are going to loose substantial amount of market -- and once its gone, it's gone in a first mover, winner takes all environment like what we have right now. Just ask Google themselves what it felt like back in the early 2000's when they completely destroyed the other search engines.

Re: What it feels like to work in AI right now

#108

Post by lexandstuff »

redox99 wrote: post #47
UncleOxidant wrote: post #25

Earlier quoted context omitted.

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

I don't know about GPT4, but GPT3.5 I'd bet is pretty traditional and boring. It's power comes from a really good, properly curated dataset (including the RLHF). GPT3.5 turbo is much more interesting probably, because they seem to have found out how to make it much more efficient (some kind of distillation?). GPT4 if I had to make a very rough guess, probably flash attention, 100% of the (useful) internet/books for i…

In this interview [1] with Ilya Sutskever, he indicates that they aren't even close to tapping out of data.

[1] https://www.youtube.com/watch?v=Yf1o0TQzry8&t=656s

Re: What it feels like to work in AI right now

#109

Post by alain94040 »

xyzzy123 wrote: post #92
pixl97 wrote: post #75

Earlier quoted context omitted.

I am still wondering if Google is experiencing it Kodak moment. They have the dominant product that makes them billions and billions of dollars at 'relatively' low cost. The new dominant product is on its way, but it costs far more to operate and will net them far less money, so... um no one wants to kill the goose that is still laying golden eggs, even though its days are numbered already.

I think it's not just run cost - there's also a risk of ad revenue destruction, that has to be worrying.

I don't see ads as a big problem with ChatGPT. You could put a side-bar on the right and on the fly recommend products relevant to the on-going conversation.

The cost of computing these ads would be a lot more than today's keyword-based approach, that's certainly a problem. But think of hyper-relevant ads, based on the chat itself. There's a lot of information there, that beats tracking people's behavior online all day.

Re: What it feels like to work in AI right now

#110

Post by Robotbeat »

SamvitJ wrote: post #106

"Every single person I know working in AI these days (in both the academy and industry) has been sparked by the ChatGPT moment." This does not ring true to me at all. Anecdotally, it feels the rush to get into AI (in both industry and academia, for both individuals and organizations) peaked around 2016-2020, post-AlexNet/ResNet, around the time Transformers became very popular. Hiring for ML research roles in particu…

DallE->StableDiffusion->Midjourney has occurred simultaneously and although it’s not as profound as ChatGPT, is definitely compounding the cultural impact of end-user AI. But probably social media AI filters are a yet bigger cultural impact on most people.
Post reply on HN