Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advancements in machine learning for machine learning

blog.research.google

Post reply on HN
Print view View original

71–80 of 162 posts

Re: Advancements in machine learning for machine learning

#71

Post by ben_w »

nodogoto wrote: post #52
sbierwagen wrote: post #44

Earlier quoted context omitted.

In order for an AI to evaluate the effect of a small molecule on the brain, it would have to... simulate the operation of a human brain in a simulated environment. Similarly, to avoid Thalidomide-style disasters, it would have to simulate the conception, development and growth to adulthood of a human. These things are... physically possible, but have WBE and uploads as a hard requirement. Those are going to affect a…

>In order for an AI to evaluate the effect of a small molecule on the brain, it would have to... simulate the operation of a human brain in a simulated environment. Humans aren't capable of doing this, but still make useful drug discoveries. AI can be empowered to conduct research in the real world, it doesn't need to simulate everything.

We start by doing them on mice (well, in vitro first, mice as the first in vivo), who have no say in the matter; and as mice are only rough analogues of humans, the human trials are still cautious once the animal trials are over.

Re: Advancements in machine learning for machine learning

#72

Post by rvnx »

xbmcuser wrote: post #63
DeathArrow wrote: post #33

Earlier quoted context omitted.

For me it's just another gold rush after dotcom, mobile, cloud, VR.

The first 3 have and did result as of today in trillions in dollars of economic activity. And have changed societies, politics, political participation, access to knowledge etc worldwide for good and bad. So I don't get why you are so dismissive of them.

AI is definitively here to stay forever. It's not a hype, it's 100% here for the long-term.

The hype may be specific to some companies for now, but AI is deeply going to change many industries, especially due to open-source, specialized chips to allow running in local, and new hardware (I strongly hope a clone of H100 A80G comes quickly).

The next step is to add limbs to the LLMs.

Then we get Tesla bot who is going to help you with daily chores, and to execute tasks in a factory.

The bot can ask its own internal knowledge base to know what action to execute next, and because the model can output JSON, the action commands can be sent to motors for in-real-life execution.

Re: Advancements in machine learning for machine learning

#73

Post by rvnx »

kccqzy wrote: post #67
vitorgrs wrote: post #29

Earlier quoted context omitted.

GPT4 was created before most feedback cycle. They had GPT4 ready before ChatGPT launch. If I recall right, GPT4 got done in October. After that, it was RLHF and safety work (Bing starts using GPT4 publicly in February, a month earlier than official launch)

If I recall right, before ChatGPT launched Google already had LaMDA which an employee believed to be sentient and was subsequently fired. The foundation model was definitely done, but to launch Bard, Google needed a kick in the ass in additional RLHF, safety and groundedness work. Ultimately though, it's futile to argue which model got done first, as long as the models were behind closed doors. But ChatGPT launched b…

LaMDA is really far from being sentient.

It's outputs non-sensical (aka highly hallucinating) or relatively useless but coherent text.

It really needs further refinement.

This is one big reason why GPT-4 is still the most popular.

Re: Advancements in machine learning for machine learning

#74

Post by ben_w »

SirMaster wrote: post #11
greatpostman wrote: post #6

Earlier quoted context omitted.

I really don’t people will be programming like we do today in five years

I don’t see why not. I like programming how I do now. I don’t plan to stop. People do lots of things manually that machines have been able to do for a long time.

We can't all run a YouTube channel for the programming equivalent of Primitive Technology, fun though that would be. 99.99% of us will have to adapt to AI being a coworker, who will probably eventually replace us.

Right now we're still OK because the AI isn't good enough; when it gets good enough, doing things manually is as economically sensible as making your own iron by gathering a few times your mass in wood, burning some of in a sealed clay dome to turn it into charcoal, digging up some more clay to make a porous pot and a kiln to fire it in, filling it with iron rich bacterial scum from a creek, letting the water drain, building a furnace, preheating it with the rest of the wood, then smelting the bacterial ore with the charcoal, to yield about 7 grams or iron.

Re: Advancements in machine learning for machine learning

#75

Post by d3m0t3p »

dalbasal wrote: post #45

Can anyone bring this down to earth for me? What's the actual state of these "ML compilers" currently, and what is rhe near term promise?

One of the easiest approache is torch.compile, it's the latest iteration of pytorch compiler (previous methods were : TorchScript and FX Tracing.)

You simply write model = torch.compile(model)

"Across these 163 open-source models torch.compile works 93% of time, and the model runs 43% faster in training on an NVIDIA A100 GPU. At Float32 precision, it runs 21% faster on average and at AMP Precision it runs 51% faster on average."[1]

What google is trying to do, is to involve more people in the R&D of these kind of methods.

[1]https://pytorch.org/get-started/pytorch-2.0/

Re: Advancements in machine learning for machine learning

#76

Post by dindobre »

kccqzy wrote: post #67
vitorgrs wrote: post #29

Earlier quoted context omitted.

GPT4 was created before most feedback cycle. They had GPT4 ready before ChatGPT launch. If I recall right, GPT4 got done in October. After that, it was RLHF and safety work (Bing starts using GPT4 publicly in February, a month earlier than official launch)

If I recall right, before ChatGPT launched Google already had LaMDA which an employee believed to be sentient and was subsequently fired. The foundation model was definitely done, but to launch Bard, Google needed a kick in the ass in additional RLHF, safety and groundedness work. Ultimately though, it's futile to argue which model got done first, as long as the models were behind closed doors. But ChatGPT launched b…

The LaMDA is sentient guy gave me the impression of being a bit nuts. I'm sure google would show their weight and out-compete openai if they could. We all know all this "AI safety" is for show, right?

Re: Advancements in machine learning for machine learning

#77

Post by code51 »

sbierwagen wrote: post #20
GreedClarifies wrote: post #9

How’s Gemini looking?

It is interesting how persistently dominant GPT-4 is: https://twitter.com/lmsysorg/status/1735729398672716114 Off the top of my head, I can think for at least five foundation models (Llama, Claude, Gemini, Falcon, Mistral) that are all trading blows, but GPT is still a head above them and has been for a year now. Transformer LLMs are simple enough that, demonstrably, anyone with a million bucks of GPU time can make o…

Their special sauce is most probably the quality of data and the amount of data cleaning effort they put in.

I’m speculating here but I think Google always refrains from getting into the manual side of things. With LLMs, it became obvious so fast that data is what matters. Seeing Microsoft’s phi-2 play, I’m convinced more about this.

DeepMind understood the properties, came up with Chinchilla but DeepMind couldn’t integrate well with Google, in terms of understanding what kind of data Google should supply to increase model quality.

OpenAI put annotation/cleaning work almost right from the start. Not too familiar with this but human labor was heavily utilized to increase training data quality after ChatGPT started.

Re: Advancements in machine learning for machine learning

#78

Post by moffkalast »

wait_a_minute wrote: post #8
RyanShook wrote: post #4

The pace that ML seems to be advancing right now is amazing. I don’t believe in the singularity but it’s changing software and then society in ways no one can predict.

This + FunSearch make it seem like Singularity is imminent. https://deepmind.google/discover/blog/funsearch-making-new-d...

Some speculate that this is what OpenAI's Q* model is about and what caused the Altman/Sutskever split.

Re: Advancements in machine learning for machine learning

#79

Post by ben_w »

flatline wrote: post #27
Der_Einzige wrote: post #17

Earlier quoted context omitted.

They’re making fun of your typo, but you’re right. Pretty much every software job in 5 years will be an AI job. This rustles a lot of feathers, but ignoring the truth will only hurt your career. I think the era of big tech paying fat stacks to a rather larger number of technical staff will start to wane as well. Better hope you have top AI paper publications and deep experience with all parts of using LLMs/whatever f…

LLMs are cool and will continue to change society in ways we cannot readily predict, but they are not quite that cool. GPT3 has been around for a little bit now and the world has not ended or encountered a singularity. The models are expensive to run both in compute and expertise. They produce a lot of garbage. I see the threat right now to low-paid writing gigs. I’m sure there’s a whole stratum of those they have wi…

> GPT3 has been around for a little bit now and the world has not ended or encountered a singularity.

And they won't right up until they do. Reason why is that…

> The models are expensive to run both in compute and expertise.

…doesn't extend to the one cost that matters: money.

Imagine a future AI that beats graduates and not just students. If it costs as much per line of code as 1000 gpt-4-1106-preview[0] tokens, the cost of rewriting all of Red Hat Linux 7.1 from scratch[1] is less than 1 million USD.

[0] $0.03 / 1K tokens

[1] https://dwheeler.com/sloc/

Re: Advancements in machine learning for machine learning

#80

Post by moffkalast »

xbmcuser wrote: post #64
greatpostman wrote: post #6

Earlier quoted context omitted.

I really don’t people will be programming like we do today in five years

I think the biggest blind spot for many programers/coders is that yes it might not change much for them but it will allow many more people to code and do stuff that they were not able to before. As the the models get better and people use them more and learn how to use them more efficiently they will start changing things. I am hoping we get to the point where the models are good enough that classes in schools are in…

It's not like schools have any other option on the table, students will find a way to use all the help they can get like they always have. Embracing it is the only way they can stay relevant in the coming age of one-on-one AI tutors.

It reminds me of the middle ages where only the priest was allowed to read and interpret the bible, mostly through the virtue of knowing latin. Then suddenly the printing press comes around and everyone can get their own cheap bible in their language. You just can't fight and enforce this kind of thing in the face of such insane progress. In 100 years (if we're not extinct then) people will probably look back on mass education where one overworked teacher tries to explain something in a standard way to 30 people (over half of who are bored or can't keep up) as some kind of old age savagery.

Post reply on HN