Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advancements in machine learning for machine learning

blog.research.google

Post reply on HN
Print view View original

131–140 of 162 posts

Re: Advancements in machine learning for machine learning

#131

Post by YawningAngel »

stabbles wrote: post #112
owlbite wrote: post #107

These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude. These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster…

Take a look at the chess engine Stockfish: they tossed out years and years of human written heuristics in board evaluation, to a small neural net that does the same but better. Now consider all the heuristics for inlining, loop unrolling, vectorization etc in compilers, certainly a neural net can be beneficial and possibly easier to maintain than tons of human written heuristics.

My understanding is that stockfish retains and uses its classical evaluation model in addition to the NNUE model

Re: Advancements in machine learning for machine learning

#132

Post by owlbite »

dbecker wrote: post #126
owlbite wrote: post #107

These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude. These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster…

> These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler Funny you should say that. Because traditional compilers have been incredibly useful.

Right, but we still tend to sidestep the compiler and/or spend hours of human time tuning the input to get the right output for core kernels.

Re: Advancements in machine learning for machine learning

#133

Post by jhardy54 »

owlbite wrote: post #107

These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude. These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster…

Exactly! Why would anyone use gcc/clang when you can just hire someone to hand-write assembly instead?

Re: Advancements in machine learning for machine learning

#134

Post by y04nn »

dindobre wrote: post #76
kccqzy wrote: post #67

Earlier quoted context omitted.

If I recall right, before ChatGPT launched Google already had LaMDA which an employee believed to be sentient and was subsequently fired. The foundation model was definitely done, but to launch Bard, Google needed a kick in the ass in additional RLHF, safety and groundedness work. Ultimately though, it's futile to argue which model got done first, as long as the models were behind closed doors. But ChatGPT launched b…

The LaMDA is sentient guy gave me the impression of being a bit nuts. I'm sure google would show their weight and out-compete openai if they could. We all know all this "AI safety" is for show, right?

No, it's for brand safety and reputation. In 2016 Microsoft released Tay [1] without or lacking guards and it ended up being a failure and hurter the Microsoft brand.

[1] https://en.wikipedia.org/wiki/Tay_(chatbot)

Re: Advancements in machine learning for machine learning

#135

Post by owlbite »

stabbles wrote: post #112
owlbite wrote: post #107

These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude. These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster…

Take a look at the chess engine Stockfish: they tossed out years and years of human written heuristics in board evaluation, to a small neural net that does the same but better. Now consider all the heuristics for inlining, loop unrolling, vectorization etc in compilers, certainly a neural net can be beneficial and possibly easier to maintain than tons of human written heuristics.

We'll have to see. I could definitely see someone spending a lot of time training for a specific algorithmic kernel and microarchitecture and beating the best human results (by a few percent).

I'd be very surprised if that can be extended to a large complex algorithmic system that is amenable to mathematical reformulations (at least within the next 10 years).

Re: Advancements in machine learning for machine learning

#136

Post by flatline »

ben_w wrote: post #79
flatline wrote: post #27

Earlier quoted context omitted.

LLMs are cool and will continue to change society in ways we cannot readily predict, but they are not quite that cool. GPT3 has been around for a little bit now and the world has not ended or encountered a singularity. The models are expensive to run both in compute and expertise. They produce a lot of garbage. I see the threat right now to low-paid writing gigs. I’m sure there’s a whole stratum of those they have wi…

> GPT3 has been around for a little bit now and the world has not ended or encountered a singularity. And they won't right up until they do. Reason why is that… > The models are expensive to run both in compute and expertise. …doesn't extend to the one cost that matters: money. Imagine a future AI that beats graduates and not just students. If it costs as much per line of code as 1000 gpt-4-1106-preview[0] tokens, th…

I like financial breakdowns like this. The thing an LLM cannot do is all the decision making that went into that. Framing the problem is harder to quantify, and is almost certainly an order of magnitude more work than writing and debugging the code. But a sufficiently good LLM should be able to produce code cheaper than humans. Maybe with time and outside sources of truth, better.

Re: Advancements in machine learning for machine learning

#137

Post by summerlight »

owlbite wrote: post #107

These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude. These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster…

> It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude.

That throughput is the point though? You cannot have performance specialists on every single ML workload. It's still significantly better than not having these kinds of optimization.

Re: Advancements in machine learning for machine learning

#138

Post by pixl97 »

staunton wrote: post #123
Palmik wrote: post #120

Earlier quoted context omitted.

This is a naive take. How do you think Google collects or collected data for their safe-search classifiers? Now that's a sludge. Or how do you think Google evaluates search-ranking changes (or gather data for training various ad-ranking & search-ranking models).

I don't know. How do they?

I was going to make a joke about all those CAPTCHAs we've solved, but I don't have an answer here.

Re: Advancements in machine learning for machine learning

#139

Post by tommychillfiger »

soundarana wrote: post #40
anu7df wrote: post #18

Earlier quoted context omitted.

Good god. Anyone else who is curious about the human centipede, please don't Google it. Trust me on this one.

You might want to google the South Park version of the human centipede, now that you know.

LOL, agree. I can't think of one without the other. South Park is absolutely iconic - incredible that they've done what they do for so long with as much consistency in quality.

Re: Advancements in machine learning for machine learning

#140

Post by summerlight »

sbierwagen wrote: post #20
GreedClarifies wrote: post #9

How’s Gemini looking?

It is interesting how persistently dominant GPT-4 is: https://twitter.com/lmsysorg/status/1735729398672716114 Off the top of my head, I can think for at least five foundation models (Llama, Claude, Gemini, Falcon, Mistral) that are all trading blows, but GPT is still a head above them and has been for a year now. Transformer LLMs are simple enough that, demonstrably, anyone with a million bucks of GPU time can make o…

Beside the fact that Gemini pro is more comparable to GPT-3.5, one more interesting observation is that even OpenAI themselves was not able (or didn't intend) to deliver a significantly better model than GPT-4 almost over a year. And OpenAI does not seem to hide their own magical "AGI" behind the scene as they've been more focused on efficiency and engineering works reportedly, primarily driven by Sam, rather than developing a new model. I'm reasonably sure that the current transformer itself as an architecture is at its peak and most improvements will be mostly incremental.
Post reply on HN