Bard and new AI features in Search • Hacker News Board

Re: Bard and new AI features in Search

#811

Post by cromwellian » Tue, Feb 07, 2023, 4:46 AM UTC

somenameforme wrote: post #779

eklitzke wrote: post #418

Earlier quoted context omitted.

The point about the economics of running these models is an important one that slides under the radar a lot of times. The training costs for large language models like GPT are enormous, and the inference costs are substantial too. Right now things like ChatGPT are very cool parlor tricks, but there's absolutely no way to justify them in terms of the economics of running the service today. Obviously this is all going…

GLM-130B[1] is something comparable to GPT-3. It's a 130billion parameter model vs GPT-3's 175 billion, and it can comfortably run on current-gen high end consumer level hardware. A system with 4 RTX 3090s ( The proverbial 'some guy on Twitter'[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of…

Google currently handles 100,000 queries per second. The costs to run GLM-130B or GPT-3 at this rate would be astonishingly high.

Re: Bard and new AI features in Search

#812

Post by senttoschool » Tue, Feb 07, 2023, 4:49 AM UTC

brookst wrote: post #798

somenameforme wrote: post #779

Earlier quoted context omitted.

GLM-130B[1] is something comparable to GPT-3. It's a 130billion parameter model vs GPT-3's 175 billion, and it can comfortably run on current-gen high end consumer level hardware. A system with 4 RTX 3090s ( The proverbial 'some guy on Twitter'[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of…

Agree, but much less than 10 years. Now that the transformer is establishing itself as the model, we’ll see dedicated HW acceleration for transformers, encoders, decoders, etc. I will eat my hat* if we don’t see local inference for 200B+ parameters within 5 years. * I don’t own a hat

Which chip company do you think will benefit from the move to transformers?

Re: Bard and new AI features in Search

#813

Post by DannyBee » Tue, Feb 07, 2023, 4:55 AM UTC

alfor wrote: post #808

eklitzke wrote: post #418

Earlier quoted context omitted.

The point about the economics of running these models is an important one that slides under the radar a lot of times. The training costs for large language models like GPT are enormous, and the inference costs are substantial too. Right now things like ChatGPT are very cool parlor tricks, but there's absolutely no way to justify them in terms of the economics of running the service today. Obviously this is all going…

1- the chips are not efficient currently (graphic card reused as neural net) 10X-100X gain 2- Moore’s Laws 3- Algorithm/architecture improvement

1. Im not sure who you believe will produce these chips, or who will use them. You are correct that specialized inference chips will get you a 10x gain. So what? If you want several million of them in a datacenter, that's a tall order.

That's on top of

A. the hundreds of millions it will take to get a design to production.

B. The complete and total lack of allocation at anybody who could make you chips, except at very very very high cost, if at all - have you forgotten that automakers still can't get cheap chips made on older processes? Most allocation of newer processes is bought out for years.

While there is some ability to get things made at newer process, building a chip for 7nm is 10-100x as expensive as say 45nm.

C. The fact that someone has to be willing to build, plan, and execute putting them in datacenters.

This will all happen, but like, everyone just assumes the hard part is the chip inefficiency.

We already can make designs that are ~10x more efficient at inference (though it depends on if you mean power or speed or what). The fact that there are not millions in datacenters should tell you something about the difficulty and economics of accomplishing this.

People aren't sitting around twiddling their thumbs. If Microsoft or Google or anyone could make themselves a "100x better cloud for AI", they would do it.

2. Dead. Dennard scaling went out the window years ago. Other scaling has mostly followed. The move to specialization you see and push to higher frequencies is because its all dead.

3. Will take a while.

The economics of this kind of thing sucks. It will not change instantly, there isn't the capability to make it happen.

Re: Bard and new AI features in Search

#814

Post by cavisne » Tue, Feb 07, 2023, 4:59 AM UTC

mrkramer wrote: post #523

I don't expect much from Bard but we shall see. For few weeks now I had a thought experiment of creating a LLM search engine trained on books. Such LLM search engine would be most reliable if you seek knowledge but as others mentioned if you want up to date information, search engine is probably your fastest and easiest way to go. But actually I wonder and somewhat doubt that data, information and knowledge in books…

Google Books does seem to still be digitizing books (albeit with the various copyright lawsuits they arent displaying many results) [1]

So they would be in a unique situation to build this, as they have 40 million books many of which no one else would be able to scan.

[1] https://www.blog.google/products/search/15-years-google-book...

Re: Bard and new AI features in Search

#815

Post by danychok » Tue, Feb 07, 2023, 5:01 AM UTC

elorant wrote: post #329

alexb_ wrote: post #9

>AI can be helpful in these moments, synthesizing insights for questions where there’s no one right answer. Soon, you’ll see AI-powered features in Search that distill complex information and multiple perspectives into easy-to-digest formats, so you can quickly understand the big picture and learn more from the web: whether that’s seeking out additional perspectives, like blogs from people who play both piano and gui…

I like that, and that's the one thing I enjoy most about ChatGPT, but the problem with this scenario is that it breaks the web. If a search engine can give me all the answers I want, then there’s no point in visiting sites anymore, aside for e-commerce. So if you have a site you wouldn’t want the next AI search engine to crawl your content because you get no traffic back. So either they find a way to give traffic bac…

Also, if no one's site is getting human traffic would the quantity of human generated content degrade over time? And AI content will increase? And then what will the AI's scrape/learn from? Themselves? How do you change their minds when the information changes in reality, but a bunch of AIs are stuck in a consensus loop? =)

Re: Bard and new AI features in Search

#816

Post by DannyBee » Tue, Feb 07, 2023, 5:01 AM UTC

eklitzke wrote: post #418

alain94040 wrote: post #388

Earlier quoted context omitted.

Those new language models are "Google killers" because they reset all the assumptions that people have made about search for several decades. Imagine that people start using those chat bots massively as a replacement for Google search. Then the notion of keyword disappears. Google AdSense becomes mostly irrelevant. Of course, Google is a giant today with a history of machine learning innovation. So they have a good c…

The point about the economics of running these models is an important one that slides under the radar a lot of times. The training costs for large language models like GPT are enormous, and the inference costs are substantial too. Right now things like ChatGPT are very cool parlor tricks, but there's absolutely no way to justify them in terms of the economics of running the service today. Obviously this is all going…

+1

For a group of people on a site frequented by startup people, did nobody read the terms of MS's investment into OpenAI?

"Microsoft would reportedly put down $10 billion for a 75% share of the profits that OpenAI earns until the money on the investment is paid back. Then when Microsoft breaks even on the $10 billion investment, they would get a 49% stake in OpenAI.

These are not the terms you would take if, tomorrow, or even two years from now, you were about to be wildly profitable because everything was about to be so easy.

These are the terms you would take if Microsoft was the only hope you had of getting the resources you need, or if getting somewhere was going to be very expensive and you needed to defray costs.

Honestly, with the level of optimism in the rest of this thread about how easy this will all be, they would probably be profitable enough to just buy MS in like 3 years , and wouldn't have needed investment at all!

Re: Bard and new AI features in Search

#817

Post by Illniyar » Tue, Feb 07, 2023, 5:02 AM UTC

KETpXDDzR wrote: post #805

AI generated summary: Sundar Pichai, CEO of Google and Alphabet, has announced the release of Bard, an experimental conversational AI service powered by Google's Language Model for Dialogue Applications (LaMDA). Bard seeks to combine the breadth of the world's knowledge with the power, intelligence and creativity of Google's large language models. It draws on information from the web to provide fresh, high-quality re…

Missed an important detail: "And today, we’re taking another step forward by *opening it up to trusted testers ahead* of making it more widely available to the public in the coming weeks."

Re: Bard and new AI features in Search

#818

Post by ss108 » Tue, Feb 07, 2023, 5:07 AM UTC

...where is the link to it

Re: Bard and new AI features in Search

#819

Post by xs83 » Tue, Feb 07, 2023, 5:09 AM UTC

wiremine wrote: post #232

I agree this is bland corporate speak. But it reminded me of a question that's been floating around: A number of pundits, here on HN and elsewhere, keep referring to these large language models are "google killers." This just doesn't make sense to me. It feels like Google can easily pivot its ad engine to work with the AI-driven chat systems. It can augment answers with links to additional sources of information, be…

I think simply there are 2 killer features in LLM's vs Google search for me:

1 - Natural Language with prepositions and easy ways to include, exclude and filter 2 - Refinement - "No that wasn't quite right because X - please factor this in and try again" is a lot more intuitive than multiple rounds of operator uses and "memory exclusion" of pages you have already seen.

I find that chatGPT will give me what I need within a few iterations, Google search sometimes takes a lot of searching and reading to get an idea of what I need.

I feel like ChatGPT + Github Code search could be a killer combination for programmers

Re: Bard and new AI features in Search

#820

Post by granzymes » Tue, Feb 07, 2023, 5:13 AM UTC

somsak2 wrote: post #785

funstuff007 wrote: post #736

Earlier quoted context omitted.

> These AI queries are resource intensive. But perhaps only for now. When youtube first started taking off, Google was bleeding cash on the resources needed to support it. Perhaps the same will be true for LLMs.

It's still unclear if YouTube is actually profitable.

It’s been clear since Google started breaking out YouTube in 2020 that YouTube is profitable.