Why did Google Brain exist? • Hacker News Board

Re: Why did Google Brain exist?

#311

Post by erenyeager » Thu, Apr 27, 2023, 4:03 PM UTC

hintymad wrote: post #111

erenyeager wrote: post #60

Earlier quoted context omitted.

Ok but the lack of underrepresented minorities in the field and the important role people like Gebru played in extending the political and status of minorities is ok to extinguish? We need more than just white male / Chinese male / Indian male monoculture “STEM lords”. This is already recognized in fields like medicine, where minorities treating minorities results in better outcomes and the greater push to open posit…

> important role people like Gebru played in extending the political and status of minorities is ok to extinguish? No, she didn't. Attacking everyone for baseless motives and identities is the worst kind of activism. She alienated people by attacking them without basis. She disdained those who truly fought for the fairness and justice of every race. She left a bad taste in people who truly cared about progress. Yes,…

Take a look at this lecture about the history of slavery and racism in the US then [1]. This will provide an answer many of your questions. Dismissing a prominent PoC’s position for being outspoken and critical of existing unfairness really demonstrates who is maintaining political power here. Do you really think that centuries of unfairness and trauma that still is perpetuated today will be swept under the rug by your fantasy of “equivalent” treatment? Equity is about Justice which is fundamentally about ethics. So tell me why are you so rooted and uncomfortable with confronting historical injustice?

1: https://youtu.be/BGjSday7f_8

Re: Why did Google Brain exist?

#312

Post by vl » Thu, Apr 27, 2023, 4:41 PM UTC

rerx wrote: post #306

vl wrote: post #265

Earlier quoted context omitted.

I assume/suspect internally Google has v5 already. The thing about TPU clusters that they have hyper-torus optical interconnect between TPUs. This allows for extremely efficient weight updates. To replicate this with A100s you need very custom hardware/software deployment. But to be fair, I don’t know what is latest and greatest available from NVidia or other clouds in this area right now. EDIT: Looks like NVidia has…

External NVSwitch (for more than 8 GPUs) isn't available for purchase yet. But even now you typically buy your A100 or H100 training GPUs in servers where each GPU has an individual 400 Gbps Mellanox networking adapter, then connect them in a sophisticated switched network that provides full throughput between all GPUs of the cluster. The deployment is not that custom, typically you would let Nvidia and your vendors…

How is network adapter connected to GPU? They just seat on the same bus?

It looks like in TPU v4 cluster each pod with 2 or 4 (?) TPUs has 6 optical interfaces, which directly connect to next pods. I have no idea how they route though this configuration, but my guess most messages are weight updates, which are essentially broadcasts, so it should work out fine with some basic forwarding.

Re: Why did Google Brain exist?

#313

Post by [deleted] » Thu, Apr 27, 2023, 4:55 PM UTC

simonster wrote: post #125

I work for Google Brain. I remember meeting Brian at a conference and I have nothing but good things to say about him. That said, I think Brian is underestimating the extent to which the Brain/DeepMind merger is happening because it's what researchers want. Many of us have a strong sense that the future of ML involves models built by large teams in industry environments. My impression is that the goal of the merger i…

[deleted]

Re: Why did Google Brain exist?

#314

Post by rerx » Thu, Apr 27, 2023, 5:35 PM UTC

vl wrote: post #312

rerx wrote: post #306

Earlier quoted context omitted.

External NVSwitch (for more than 8 GPUs) isn't available for purchase yet. But even now you typically buy your A100 or H100 training GPUs in servers where each GPU has an individual 400 Gbps Mellanox networking adapter, then connect them in a sophisticated switched network that provides full throughput between all GPUs of the cluster. The deployment is not that custom, typically you would let Nvidia and your vendors…

How is network adapter connected to GPU? They just seat on the same bus? It looks like in TPU v4 cluster each pod with 2 or 4 (?) TPUs has 6 optical interfaces, which directly connect to next pods. I have no idea how they route though this configuration, but my guess most messages are weight updates, which are essentially broadcasts, so it should work out fine with some basic forwarding.

Can't find documentation online yet for this year's H100 systems, but here's a schematic for an A100 server: https://docs.nvidia.com/dgx/dgxa100-user-guide/introduction-... There 2 GPUs, 2 NICs and 1 NVME are connected via one PCIe switch.

The torus topology makes sense for ring algorithms: Allreduce, Allgather, Reducescatter. For purely data parallel training you could put all model replicas into the same ring (although Nvidia also uses hierarchical algorithms that benefit from lower lately). With added model parallelism one will need smaller rings running concurrently. I guess the TPU cluster layout will then put constraints on the most efficient model architectures (as does the network topology of a GPU cluster).

Re: Why did Google Brain exist?

#315

Post by oofbey » Thu, Apr 27, 2023, 5:44 PM UTC

tadeegan wrote: post #228

oofbey wrote: post #208

Earlier quoted context omitted.

TF1 was pretty rough to use, but beat the pants off Theano for usability, which was really the best thing going before it. Sure it was slow as dirt ("tensorslow") even though the awkward design was justified on being able to make it fast. But it was by far the best thing going for a long time. Google really killed TF with the transition to TF2. Backwards incompatible everything? This only makes sense if you live in a…

Ehhh, no. Even inside google TF2 migration was a total disaster and is still ongoing :)

Sure, my point is that only a googler would even consider this kind of breaking change as a sensible option. People in the real world with regular code tooling would reject the proposal before it got started.

The analogy to Angular that others have made is spot on. It's not just first-mover disadvantage. Google has particular blind spots for certain pain points, like deprecating APIs. Also q.v. Google Cloud.

Re: Why did Google Brain exist?

#316

Post by nylon_panda » Thu, Apr 27, 2023, 7:07 PM UTC

petesergeant wrote: post #272

quickthrower2 wrote: post #255

Earlier quoted context omitted.

Sport incentives work out nicely for getting a supply of top talent. Big money through sponsorship/ads/tickets, fairly formulaic system to get better at a sport (top coaches/teams, get fitter, practice a lot), an easy way to identify progress (wins games, scores goals). But for sciences, someone could do great work for 20 years and produce obscure papers that might stay obscure, or may unlock the mystery of the unive…

Would employees of non-sports industries accept the weird-ass ways that sports contracts work? Major League Pharma Research is certainly intriguing

To my understanding it'd need to be set-up so the proposition of R&D is granted by unbiased parties. I believe its been linked on HN before, where it describes what system would be best to incentivize research? It stated the dangers of funding the research of only the popular (and contemporary) topics, which tilts R&D away from discovery and more towards 'lets try to yield xyz, and if it fails, lets not release it'. Someone who is motivated to grant something must be completely separate from the publication, most likely those who researched it and many other scientists, & unlike peer review try not to trim any of it out

Re: Why did Google Brain exist?

#317

Post by aflag » Thu, Apr 27, 2023, 7:35 PM UTC

potatolicious wrote: post #142

asdfman123 wrote: post #56

Earlier quoted context omitted.

Well, Google could have easily not have shared its technology. However, the bloat problem you’ve described are difficult problems to solve, and are to some degree endemic to large businesses with established products.

> "Well, Google could have easily not have shared its technology." Sure, but the idea is that if they didn't share their technology, they'd still be in the same spot: they would have invented transformers and still not shipped major products around it. Sure maybe OpenAI won't exist, but competitors will find other ways to compete. They always do. So at best they are very very slightly better off than the alternative,…

Also, these things are not secret for a very long time and eventually other people figure out (completely independently or maybe even influenced by some google's ex-employee). Then, they wouldn't even be able to say they invented it.

Re: Why did Google Brain exist?

#318

Post by nighthawk454 » Thu, Apr 27, 2023, 7:59 PM UTC

sundarurfriend wrote: post #245

seatsniffer wrote: post #163

Earlier quoted context omitted.

Thanks for taking the time to explain. I'll check out the link also.

Just to make it clear, you're hearing one side of the argument. There are also a lot of people who firmly believe that Schmidhuber was wrongfully deprived of recognition he deserved. I don't have a strong opinion one way or another, but the issue isn't as clear cut as your parent comment might have made it sound like.

Yes, thanks. Didn't intend to be one-sided. Not super versed in this subject so please everyone do your own research!

Re: Why did Google Brain exist?

#319

Post by kergonath » Thu, Apr 27, 2023, 8:39 PM UTC

selimthegrim wrote: post #308

kergonath wrote: post #293

Earlier quoted context omitted.

Lots of things, but mostly things like designing software architectures for massively parallel codes implementing fancy physics or coming up with ML models to predict complicated properties (things like wear, radiation, or corrosion resistance for which we just do not have any comprehensive model) or explore humongous problem spaces (things like predicting simple properties of very complex materials with 5 to 8 eleme…

Ha well if you’re looking for a solid state physics PhD I’m on the market ;)

Not right now, unfortunately. Where are you located? We can discuss if you need any advice (username at me.com).

Re: Why did Google Brain exist?

#320

Post by jltsiren » Thu, Apr 27, 2023, 9:05 PM UTC

ramraj07 wrote: post #304

jltsiren wrote: post #299

Earlier quoted context omitted.

Genomics, for example. The entire field could not exist without extensive algorithmic research done over several decades. Even today, many of the key people in the field have a background in CS and mathematics. Theoretical ecology is a bit more old-school answer. Many mathematicians have been involved in that field.

I am not sure I have seen any meaningfully brilliant breakthroughs in those fields though. I take that back. One can never forget the brilliance of HiC by Erez. But other than that TBH all other “mathematical” or computational breakthroughs I’ve seen are just at best meticulous application of obvious math and computational algorithms to biological problems (and may I say poorly? Thinking back to the microarray nightm…

I guess that depends on what you count as a breakthrough. From my perspective, HiC is a small detail, and even AlphaFold is just the latest improvement to an existing process. A true breakthrough would be something like ancient DNA, which gave the humanity a new tool for studying the past.

Pretty much everything in genomics depends on shotgun sequencing, which in turn depends on CS. The algorithms used for assembling something useful out of the sequence reads are highly non-obvious. In fact, most developments in string algorithms since the 80s have been driven by the needs of DNA sequencing. (Information retrieval used to be another contender, but word tokens turned out to be a better tool for that field.)