Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Democratization in the Era of GPT-3

thegradient.pub

Post reply on HN
Print view View original

71–80 of 90 posts

Re: AI Democratization in the Era of GPT-3

#71

Post by ben_w »

partingshots wrote: post #66
ben_w wrote: post #58

Earlier quoted context omitted.

How else would you define “AI democratization”? The fact I can even program puts me in an elite of less than 0.5% of the world population [0], and that includes all specialties not just AI. I do iPhone apps these days; even though I can follow the various tutorials for how to train an AI to recognise handwritten digits [1], I don’t actually grok the maths behind the back-propagation algorithm and why it’s better than…

What’s so difficult to understand about back propagation? All you’re doing is computing a derivative.

If I could explain why it feels like a magic trick, it would immediately stop feeling like a magic trick.

I can follow each of the steps in online lectures, tutorials, etc., but something about it never clicks.

Re: AI Democratization in the Era of GPT-3

#72

Post by tetris11 »

darawk wrote: post #62
jackcosgrove wrote: post #39

The PC was supposed to democratize computing. It did, and then Microsoft found a choke point. The internet was supposed to democratize communication. It did, and then Google found a choke point. It seems like this two steps forward, one step back pattern might be the rule rather than the exception. Even the article defines AI democratization in terms of using models rather than training models, as the costs of traini…

Neither Google nor Microsoft were or are actually choke points. The internet is plenty democratized. People simply choose to use a few big centralized services. They have the ability to use others. A democracy where everyone votes for the same candidate because they're the best candidate is still a democracy.

You could argue the losing net neutrality debate influences people to use the more centralised services, and the dominance of chrome pushes people towards more google-centric services

Re: AI Democratization in the Era of GPT-3

#74

Post by chosen1111 »

jackcosgrove wrote: post #39

The PC was supposed to democratize computing. It did, and then Microsoft found a choke point. The internet was supposed to democratize communication. It did, and then Google found a choke point. It seems like this two steps forward, one step back pattern might be the rule rather than the exception. Even the article defines AI democratization in terms of using models rather than training models, as the costs of traini…

Wonder what the chokepoint is for blockchain. And who will find it?

Re: AI Democratization in the Era of GPT-3

#75

Post by PeterStuer »

mensetmanusman wrote: post #24

I learned some things. GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run. Assumption: Bleeding edge AI will require tens of millions of dollars of computation before new network architectures fall out of state space. After this, the models can be pruned to be ran by mere mortals. If this is true, OpenAI will not be able to move to the next level with…

Sure there might be challenges and opportunities on the bleeding edge. However, in this case, this opportunity was so orthogonal to their original mission statement it should not come as surprise t does more than raise eyebrows.

Re: AI Democratization in the Era of GPT-3

#76

Post by m-p-3 »

chosen1111 wrote: post #74
jackcosgrove wrote: post #39

The PC was supposed to democratize computing. It did, and then Microsoft found a choke point. The internet was supposed to democratize communication. It did, and then Google found a choke point. It seems like this two steps forward, one step back pattern might be the rule rather than the exception. Even the article defines AI democratization in terms of using models rather than training models, as the costs of traini…

Wonder what the chokepoint is for blockchain. And who will find it?

Proof-of-work and the massive computing farms that belongs to a handful of miners would be a choke point, at least for Bitcoin?

Re: AI Democratization in the Era of GPT-3

#77

Post by david-gpu »

ben_w wrote: post #71
partingshots wrote: post #66

Earlier quoted context omitted.

What’s so difficult to understand about back propagation? All you’re doing is computing a derivative.

If I could explain why it feels like a magic trick, it would immediately stop feeling like a magic trick. I can follow each of the steps in online lectures, tutorials, etc., but something about it never clicks.

Backpropagation is literally the chain rule of derivatives.

Imagine you have a black box, such that when it produces an output, you can compare it against a target. The black box has a ton of little dials (weights) you can turn up or down, which affect what the output will be.

Say the output of the box was too small relative to the target. Now you want to know how to tweak each of the dials (weights) to increase the output a little, so that next time the output will be closer to the target.

How do you do that? You could do it by trial and error, changing one dial at a time to see how it affects the output (find the derivative of the output relative to the weight). That works but it is very inefficient.

So, what if it was not a black box? What if you could peek into some of the circuitry right before the output is produced? There are fewer knobs there to tweak. More efficient!

You could then figure out how sensitive the output is to each of those end-knobs (weight gradients), and even to the inputs to those end-knobs (activation gradients). Your life is getting easier.

But wait, why stop there? Now that you know how the circuitry close to the end output works, you can repeat the same process iteratively working backwards towards the inputs of the black box (chain rule). In the process, you will know exactly how much each dial affects the final output.

Instead of doing all this numerically by playing with the dials, you can do the same analytically if you know the (derivable) functions that compose this big box of dials.

That's how it's done, basically.

Re: AI Democratization in the Era of GPT-3

#78

Post by darepublic »

ben_w wrote: post #71
partingshots wrote: post #66

Earlier quoted context omitted.

What’s so difficult to understand about back propagation? All you’re doing is computing a derivative.

If I could explain why it feels like a magic trick, it would immediately stop feeling like a magic trick. I can follow each of the steps in online lectures, tutorials, etc., but something about it never clicks.

I feel the same way but my favourite explanation to date is a mini series by 3 brown 1 blue on youtube

Re: AI Democratization in the Era of GPT-3

#80

Post by ralfn »

mensetmanusman wrote: post #24

I learned some things. GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run. Assumption: Bleeding edge AI will require tens of millions of dollars of computation before new network architectures fall out of state space. After this, the models can be pruned to be ran by mere mortals. If this is true, OpenAI will not be able to move to the next level with…

>Ergo, it is also very expensive to run

Wether thats too expensive depends on what you are doing, and how much what you are doing creates value.More disturbingly is that the whole approach seems to scale linearly in quality based on the amount of training. This implies that the NLP market at least, will not be conquored by start-ups from somebody's garage, but will be owned by whoever already had a shit load of money. Whoever gets more funding will have the best model. Not arguing that people involved don't have a lot of skill, but within this space, its funding (not skill) that will determine who wins the market.

So, unless you can get more funding than the other guys, don't even try the NLP space.

>If this is true, those calling for OpenAI to not monetize intermittent progress are essentially preventing next generation discovery, unless they have alternative monetization ideas to generate 8 figures for research

I suspect they will share the models directly with select customers (with too much money), because the negotation position of these counterparties will be different. The results are easy to replicate with a lot of money. So if you have enough money to do so, your price negotation with them would be more like 'ill pay you 1/3 of that price to liscense your model and save me the time'.

If you dont have enough money to pay the electricity bill to train this much data, however, you can be forced to comply with this bussiness model. Obviously, there will be competitors. Obviously the big boys will likely try to replicate these results (and be succesfull at it). The hope is one of them just open-sources a 'good-enough' model.

Its a bit like Colombus 'discovering America'. Once you know its there the risk/reward of trying to go there drastically changes.

Post reply on HN