The Google employees who created transformers • Hacker News Board

Re: The Google employees who created transformers

#251

Post by shiroiushi » Fri, Mar 22, 2024, 3:27 AM UTC

cmrdporcupine wrote: post #239

shiroiushi wrote: post #228

Earlier quoted context omitted.

>They got outcompeted because they were a photography company, not a digital hardware manufacturer. Companies like Sony or Canon did better because they were in the business of consumer electronics / hardware already. Huh? This makes no sense. Sony was indeed a consumer electronics company at that time, but Canon was not: Canon was a camera manufacturer. They didn't get into electronics until later as cameras became…

What are you talking about? Canon was making copiers, printers, scanners, video cameras from the late 80s on. So they were already making digital hardware and so presumably had the internal expertise on how to product manage that. https://global.canon/en/corporate/history/04.html

Ok, but they were a camera company before they moved into copiers and printers, right?

Obviously, they were well-positioned since they had already moved into electronics and such before the digital revolution happened.

Re: The Google employees who created transformers

#252

Post by felixleungsc » Fri, Mar 22, 2024, 3:36 AM UTC

hn_throwaway_99 wrote: post #229

a_wild_dandan wrote: post #225

Earlier quoted context omitted.

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose). Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no comp…

Question for you, as someone relatively new to the world of AI (well, not exactly new - I took many courses in AI, including neural networks, but in the late 90s... the world is just a tad different now!) Is there any good summary of the history of AI/deep learning from, say, late 00s/2010 to the present? I think learning some of this history would really help be better understand how we ended up at the current state…

Maybe this? https://arxiv.org/abs/2212.11279

Re: The Google employees who created transformers

#253

Post by shiroiushi » Fri, Mar 22, 2024, 9:03 AM UTC

dragonwriter wrote: post #230

shiroiushi wrote: post #228

Earlier quoted context omitted.

>They got outcompeted because they were a photography company, not a digital hardware manufacturer. Companies like Sony or Canon did better because they were in the business of consumer electronics / hardware already. Huh? This makes no sense. Sony was indeed a consumer electronics company at that time, but Canon was not: Canon was a camera manufacturer. They didn't get into electronics until later as cameras became…

> Kodak could have done the same. Kodak, while they incidentally made some cameras, were a film and film processing company that wasn’t great at cameras and wasn’t anything in electronics. They were much worse positioned than either a camera company, or a consumer electronics company, for a pivot to the post-film photography world.

Canon was only a camera company when it started, but by the 1960s was moving into other markets like lenses, magnetic heads, photocopiers, fax machines, etc.

Kodak could have diversified like that too, but they didn't. They were positioned badly because they concentrated almost all their efforts on film and nothing else. Of course, part of this is probably due to American business culture compared to Japanese; Japanese businesses tend to be much more diverse and long-term in thinking, but regardless, Kodak did this to themselves.

Re: The Google employees who created transformers

#254

Post by cmrdporcupine » Fri, Mar 22, 2024, 3:45 PM UTC

shiroiushi wrote: post #253

dragonwriter wrote: post #230

Earlier quoted context omitted.

> Kodak could have done the same. Kodak, while they incidentally made some cameras, were a film and film processing company that wasn’t great at cameras and wasn’t anything in electronics. They were much worse positioned than either a camera company, or a consumer electronics company, for a pivot to the post-film photography world.

Canon was only a camera company when it started, but by the 1960s was moving into other markets like lenses, magnetic heads, photocopiers, fax machines, etc. Kodak could have diversified like that too, but they didn't. They were positioned badly because they concentrated almost all their efforts on film and nothing else. Of course, part of this is probably due to American business culture compared to Japanese; Japane…

Agreed but my additional point is that Kodak actually did branch out beyond film, and did so pretty well from a technical POV.

The problem was competing in hardware manufacturing, which is a whole different ballgame from concentrating on just the imaging aspects of it. So they were reduced, near the end, to just being really a (decent) component supplier to other companies. But that's the wrong part of the food chain to be in.

Re: The Google employees who created transformers

#255

Post by crowcroft » Fri, Mar 22, 2024, 5:43 PM UTC

empath-nirvana wrote: post #245

crowcroft wrote: post #242

Earlier quoted context omitted.

'A while now' being long enough that I might expect some notable impact on Google's business by now. I do agree it could just be too early. My iPhone/iPod timeline is referring to the time it took for the iPod's sales to be severely impacted [1]. I'm not disputing that the iPod continued to exist for a long time, the iPod Touch wasn't discontinued until 2022, but it was a pretty meaningless part of Apple's business b…

Blackberry chugged along with record profits for a couple of years after the iphone had already made them irrelevant.

Yea, this is a good example that it might just be too soon still to say.

There was a period where it was not obvious that the iPhone or Blackberry would become the dominant player in the field and it could be true with search and AI chat etc too.

Re: The Google employees who created transformers

#256

Post by mptest » Sat, Mar 23, 2024, 4:42 AM UTC

hn_throwaway_99 wrote: post #229

a_wild_dandan wrote: post #225

Earlier quoted context omitted.

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose). Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no comp…

Question for you, as someone relatively new to the world of AI (well, not exactly new - I took many courses in AI, including neural networks, but in the late 90s... the world is just a tad different now!) Is there any good summary of the history of AI/deep learning from, say, late 00s/2010 to the present? I think learning some of this history would really help be better understand how we ended up at the current state…

The worlds I see by fei fei li is exactly what you're looking for, intertwined with a beautiful story of a brilliant scientist's perseverance

Re: The Google employees who created transformers

#257

Post by ant6n » Sat, Mar 23, 2024, 12:59 PM UTC

marcosdumay wrote: post #49

ant6n wrote: post #38

Earlier quoted context omitted.

Can't all expenses be deducted when filing taxes? I mean, taxes are paid on profit, which is revenue minus expenses.

Not to mention the thing about tax credits, but when people say "R&D expenses" they often include quite a lot of investments on their definition. In fact, it's arguable if anything in R&D qualifies as an expense at all.

Only money spent that turns into an asset is investment. I’m not sure how it is in US, but in Germany, for example, not even registering a patent is an investment and does not imply the creation of an asset — only buying the patent creates the asset.

Re: The Google employees who created transformers

#258

Post by TimPC » Thu, Mar 28, 2024, 2:53 PM UTC

lelag wrote: post #143

TimPC wrote: post #130

Earlier quoted context omitted.

I think we've found repeatedly in self-driving that it's not enough to solve the problem in the normal case. You need an AI model that has good behaviors in the edge cases. For the most part it won't matter how good the vision models get, you're going to need similar models that can make the same predictions from LIDAR signals because the technology needs to work when the vision model goes crazy because of the reflec…

I don't quite agree on this one. While I think that Musk choices to go full vision when he did was foulish because he made his product worse, his main point is not wrong: human do drive well while using mostly vision. Assuming you can replicate the thought process of human driver using AI, I don't see why you could not create a self-driving car using only vision. That's also where I would see transformers or another…

Sure but humans do a lot more with vision than just convolutions. So maybe we need to wait for AI to invent new techniques equally revolutionary and equally impactful to convolutions to the point where it's believable that AI models can handle the range of exceptions humans handle. Humans are very good at learning from small data where AI tends to be pretty terrible at one-shot learning by comparison. That's going to continue being hugely relevant for edge cases. We've seen many examples now where a self-driving car crashes due to too much sunlight distorting its perception of where objects are. We can either bury our heads in the sand and pretend AI models work like humans and need the exact same inputs humans do or we can admit there are limitations to the technology and act accordingly.

I also think dumb sensors is unfair, there are Neural Network solutions for processing LIDAR data so we are talking about a similar level of intelligence applied over both sensors.