Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advancements in machine learning for machine learning

blog.research.google

Post reply on HN
Print view View original

111–120 of 162 posts

Re: Advancements in machine learning for machine learning

#111

Post by ben_w »

ChatGTP wrote: post #95
ben_w wrote: post #74

Earlier quoted context omitted.

We can't all run a YouTube channel for the programming equivalent of Primitive Technology, fun though that would be. 99.99% of us will have to adapt to AI being a coworker, who will probably eventually replace us. Right now we're still OK because the AI isn't good enough; when it gets good enough, doing things manually is as economically sensible as making your own iron by gathering a few times your mass in wood, bur…

who will probably eventually replace us. no one is going to be using AI and then just have it 'replace them', they're going to use it to augment their abilities and avoid replacement.

The people using AI to write code aren't necessarily former professional programmers, for the same reasons people using AI to make pictures aren't necessarily former professional artists, and those using aim bots aren't necessarily former professional snipers or olympic shooters.

A manager can prompt a chatbot to write a thing instead of prompting me to write the same thing — for the moment, what (I hope) keeps me employable is that the the chatbot is "only" at the level of getting good grades rather than n-years professional experience.

I have no expectation for any specific timeline for that to change. Perhaps there are enough incentives it will never get trained to that level, but also perhaps it was already trained 4 months back and the improvement to capabilities are what caused the OpenAI drama.

Re: Advancements in machine learning for machine learning

#112

Post by stabbles »

owlbite wrote: post #107

These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude. These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster…

Take a look at the chess engine Stockfish: they tossed out years and years of human written heuristics in board evaluation, to a small neural net that does the same but better.

Now consider all the heuristics for inlining, loop unrolling, vectorization etc in compilers, certainly a neural net can be beneficial and possibly easier to maintain than tons of human written heuristics.

Re: Advancements in machine learning for machine learning

#113

Post by Jensson »

IanCal wrote: post #109
Jensson wrote: post #87

Earlier quoted context omitted.

> Otherwise the description is the same as "select randomly, keep the best, iterate". That is what they did though. The LLM didn't know what problem it was "solving".

That's not really true unless you're ignoring the rest of my points. The process did not just uniformly randomly create programs. They also don't just keep the best and search from that point but feed the resulting programs and their scores into an LLM.

They did remove the worst results from the group over time, the others was just uses a seed to generate new examples from instead of starting each function from scratch.

Re: Advancements in machine learning for machine learning

#114

Post by sfn42 »

rvnx wrote: post #72
xbmcuser wrote: post #63

Earlier quoted context omitted.

The first 3 have and did result as of today in trillions in dollars of economic activity. And have changed societies, politics, political participation, access to knowledge etc worldwide for good and bad. So I don't get why you are so dismissive of them.

AI is definitively here to stay forever. It's not a hype, it's 100% here for the long-term. The hype may be specific to some companies for now, but AI is deeply going to change many industries, especially due to open-source, specialized chips to allow running in local, and new hardware (I strongly hope a clone of H100 A80G comes quickly). The next step is to add limbs to the LLMs. Then we get Tesla bot who is going t…

Pretty sure we're quite a ways off that stuff yet, especially for consumers. But sure, maybe something like that will be reality in the coming decades.

Re: Advancements in machine learning for machine learning

#115

Post by bart_spoon »

porompompero wrote: post #102
dmarchand90 wrote: post #101

Earlier quoted context omitted.

I kinda wonder if maybe it's at least partially due to openai hitting a kind of hyperparameter lottery. When each experiment costs millions it might be that (aside from good/ unique data) they just have a good set of hyperparameters used in training and it's too expensive for a competitor to find equal or better settings

Sorry for my ignorance: why does each experiment cost millions?

It’s the cost of compute hardware required to train a model of that size

Re: Advancements in machine learning for machine learning

#116

Post by bart_spoon »

m3kw9 wrote: post #19
RyanShook wrote: post #4

The pace that ML seems to be advancing right now is amazing. I don’t believe in the singularity but it’s changing software and then society in ways no one can predict.

I want to see it come out with a cure for a disease that is tough to cure first. Singularity itself is pointless unless it benefits humans which is mainly in health/lower suffering

“Cure” is a tough bar, but I believe Paxlovid, the anti-viral used to reduce Covid severity, was identified using ML. There’s many companies like Recursion Pharma which are entirely focused on using ML for drug discovery, and from what I can tell seem to have promising results, but drug development is slow enough that nothing will come of it for a while.

Also, while not medicine focused, Google’s GNOME project results announced a few weeks ago was pretty remarkable. They discovered more theoretical new materials using their ML approach than the rest of human history combined, and they are already confirming many of the results in laboratory settings. That has the potential to be a revolution in limitless scientific and engineering applications.

Re: Advancements in machine learning for machine learning

#117

Post by euos »

valine wrote: post #35
euos wrote: post #26

Earlier quoted context omitted.

I’ve been programming since middle school. That would be 30 years. Nothing really changed much. C++ is incrementally more convenient but fundamentally the same. Code editors are same. Debugger are same. Shell is same. I am certain in 30 years everything will still be the same.

The way I write code was fundamentally altered in the last year by GPT4 and copilot. Try having GPT4 write your code, you won’t be so certain about the future of programming afterward I guarantee it.

I have free Copilot due to my OSS work. This week I disabled it for C++ because it is chronically incapable to match brackets. I was wasting too much time fixing the messes.

I use it for TypeScript/React. But it’s just a more comprehensive code complete. Incremental.

Re: Advancements in machine learning for machine learning

#118

Post by jodrellblank »

educaysean wrote: post #22
m3kw9 wrote: post #19

Earlier quoted context omitted.

I want to see it come out with a cure for a disease that is tough to cure first. Singularity itself is pointless unless it benefits humans which is mainly in health/lower suffering

I'd say advancement in mathematics, computer science, and heck, even art is far from "pointless". Why does it feel like goalposts get moved everytime there is a significant progress in AI?

> "Why does it feel like goalposts get moved everytime there is a significant progress in AI?"

Why does it feel like people think this is a useful or interesting whine? OK you brute-force solved tic-tac-toe, you built an AI. Congratulations, everyone recognises the dawn of Artificial Intelligence - and truly, enumerating all states of tic-tac-toe is all we ever dreamed of, all we could want, it's really all there is to intelligence. The term "AI" will never ever be used to mean anything else.

The term "AI" means (solving the first problem that was ever suggested to be AI decades ago) - how is that a better state of the world? What has anyone gained from "not moving the goalpoasts"?

Re: Advancements in machine learning for machine learning

#119

Post by jwuphysics »

dmarchand90 wrote: post #101
sbierwagen wrote: post #20

Earlier quoted context omitted.

It is interesting how persistently dominant GPT-4 is: https://twitter.com/lmsysorg/status/1735729398672716114 Off the top of my head, I can think for at least five foundation models (Llama, Claude, Gemini, Falcon, Mistral) that are all trading blows, but GPT is still a head above them and has been for a year now. Transformer LLMs are simple enough that, demonstrably, anyone with a million bucks of GPU time can make o…

I kinda wonder if maybe it's at least partially due to openai hitting a kind of hyperparameter lottery. When each experiment costs millions it might be that (aside from good/ unique data) they just have a good set of hyperparameters used in training and it's too expensive for a competitor to find equal or better settings

I would be surprised if this is the case. Neural scaling laws are well known and are used by all big industry players to extrapolate experiments.

Re: Advancements in machine learning for machine learning

#120

Post by Palmik »

staunton wrote: post #82
code51 wrote: post #77

Earlier quoted context omitted.

Their special sauce is most probably the quality of data and the amount of data cleaning effort they put in. I’m speculating here but I think Google always refrains from getting into the manual side of things. With LLMs, it became obvious so fast that data is what matters. Seeing Microsoft’s phi-2 play, I’m convinced more about this. DeepMind understood the properties, came up with Chinchilla but DeepMind couldn’t in…

Indeed, making poor people in 3rd world countries rate the worst sludge of the internet for 8+h a day might backfire on your marketing... OpenAI could risk it, Google maybe doesn't want to...

This is a naive take. How do you think Google collects or collected data for their safe-search classifiers? Now that's a sludge.

Or how do you think Google evaluates search-ranking changes (or gather data for training various ad-ranking & search-ranking models).

Post reply on HN