Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every developer should know about GPU computing

codeconfessions.substack.com

Post reply on HN
Print view View original

21–30 of 186 posts

Re: What every developer should know about GPU computing

#21

Post by bobwaycott »

pjmlp wrote: post #9
Jeff_Brown wrote: post #3

Not every developer. I'm not trying to be snarky. I think there's an unhelpful compunction to want to know everything about everything among STEM types like programmers (of which I am one). Specialization is fundamental to the success, not just of whole economies, but of the individuals in them. It can feel like a paintful sacrifice to admit that you'll never (have time to) learn, say, the entire Python language spec…

Indeed, one thing that most seniors learn is humility and being able to say I don't know, without caring about consequences.

An invaluable life skill, as well.

Re: What every developer should know about GPU computing

#22

Post by bachmeier »

loeg wrote: post #11
Jeff_Brown wrote: post #3

Not every developer. I'm not trying to be snarky. I think there's an unhelpful compunction to want to know everything about everything among STEM types like programmers (of which I am one). Specialization is fundamental to the success, not just of whole economies, but of the individuals in them. It can feel like a paintful sacrifice to admit that you'll never (have time to) learn, say, the entire Python language spec…

At this point, I think the running shtick / inside joke of "Every Developer Should Know ..." headlines is that of course every developer doesn't need to know the contents of the article that follows.

Not true. Anyone experienced in programming discussions knows that "software development" overlaps exactly with the programming done by the commenter, and everything else is solving toy problems with tools that don't scale.

Re: What every developer should know about GPU computing

#23

Post by wongarsu »

loeg wrote: post #11
Jeff_Brown wrote: post #3

Not every developer. I'm not trying to be snarky. I think there's an unhelpful compunction to want to know everything about everything among STEM types like programmers (of which I am one). Specialization is fundamental to the success, not just of whole economies, but of the individuals in them. It can feel like a paintful sacrifice to admit that you'll never (have time to) learn, say, the entire Python language spec…

At this point, I think the running shtick / inside joke of "Every Developer Should Know ..." headlines is that of course every developer doesn't need to know the contents of the article that follows.

> of course every developer doesn't need to know the contents of the article that follows.

Things every developer should know about English, part 1: why word order matters in negation /s

Obviously you meant "not every developer needs to know" instead of "every developer doesn't need to know", but I see this switch so often lately that I'm beginning to wonder if it's a dialect thing (similar to double negative implying a positive to some and an emphasis of the negative to others)

Re: What every developer should know about GPU computing

#24

Post by cratermoon »

abhi9u wrote: post #19
cratermoon wrote: post #6

Earlier quoted context omitted.

I dunno. The section "Latency Tolerance, High Throughput and Little’s Law" has many applications in programming. Ever need to scale your cache, or size a connection pool? Little's Law.

Thank you for reading! :)

Thanks for writing this. I note you mention that GPUs are massively parallel, and I think it would strengthen the article to add a paragraph or two discussing what kinds of loads lend themselves to GPU computing. The sort of "embarrassingly parallel" things like linear algebra vs inherently serial example of numerical analysis.

Re: What every developer should know about GPU computing

#25

Post by owl57 »

yodsanklai wrote: post #8
Jeff_Brown wrote: post #3

Not every developer. I'm not trying to be snarky. I think there's an unhelpful compunction to want to know everything about everything among STEM types like programmers (of which I am one). Specialization is fundamental to the success, not just of whole economies, but of the individuals in them. It can feel like a paintful sacrifice to admit that you'll never (have time to) learn, say, the entire Python language spec…

> It can feel like a paintful sacrifice to admit that you'll never (have time to) learn And even if you do have time, you'll probably forget most of it if you don't make use of it in your daily job. Also if you do want to learn about a new topic, it'll take commitment. I remember going through an MIT OS project, it took me something like 50 hours of hours to complete the project. Pretty much impossible when you have…

> I remember going through an MIT OS project, it took me something like 50 hours of hours to complete the project. Pretty much impossible when you have a full time job (I didn't at the time). And despite that, I still consider myself a newbie in OS development.

I took an OS course loosely based on MIT one, and of course wouldn't consider myself an experienced OS developer as well, but I think the residual knowledge is sometimes really helpful when debugging bad application performance at work.

Re: What every developer should know about GPU computing

#26

Post by United857 »

Rizu wrote: post #20
cebert wrote: post #12

> Most programmers have an intimate understanding of CPUs and sequential programming because they grow up writing code for the CPU Maybe it’s just where I work, but I feel like even this isn’t true. A lot of the newer/young employees don’t even seem to have a lot of OS/system understanding. A lot of the higher-level languages abstract the immediate need of having any “intimate understanding” of CPUs.

Do you have any recommendations to learn CPUs and Systems well ?, thanks

The Patterson Hennessy computer architecture textbook is the classic and still the best IMO.

Re: What every developer should know about GPU computing

#27

Post by mavelikara »

Rizu wrote: post #20
cebert wrote: post #12

> Most programmers have an intimate understanding of CPUs and sequential programming because they grow up writing code for the CPU Maybe it’s just where I work, but I feel like even this isn’t true. A lot of the newer/young employees don’t even seem to have a lot of OS/system understanding. A lot of the higher-level languages abstract the immediate need of having any “intimate understanding” of CPUs.

Do you have any recommendations to learn CPUs and Systems well ?, thanks

Not the GP, but I recommend reading (or watching lectures based on) the book “Computer Systems: A Programer’s Perspective.”

Re: What every developer should know about GPU computing

#28

Post by mannyv »

Now I understand why ML uses floats for precision. It wasn't a choice, it was because graphics code uses them.

Another piece in the "why is ML so inefficient" puzzle!

I wonder what that memory copying overhead is IRL. If it's like normal stuff it'll be brutal. I mean, they offload tcp processing into hardware to avoid that. This is way more data, though it is done in bigger chunks.

Re: What every developer should know about GPU computing

#29

Post by i_am_a_peasant »

cebert wrote: post #12

> Most programmers have an intimate understanding of CPUs and sequential programming because they grow up writing code for the CPU Maybe it’s just where I work, but I feel like even this isn’t true. A lot of the newer/young employees don’t even seem to have a lot of OS/system understanding. A lot of the higher-level languages abstract the immediate need of having any “intimate understanding” of CPUs.

Meh, I was lucky enough to start my career in embedded and stay in it for a decade before I started delving into higher level stuff. Not everyone is that lucky to be at the right place right time receive guidance from the right people.

Re: What every developer should know about GPU computing

#30

Post by abhi9u »

cratermoon wrote: post #24
abhi9u wrote: post #19

Earlier quoted context omitted.

Thank you for reading! :)

Thanks for writing this. I note you mention that GPUs are massively parallel, and I think it would strengthen the article to add a paragraph or two discussing what kinds of loads lend themselves to GPU computing. The sort of "embarrassingly parallel" things like linear algebra vs inherently serial example of numerical analysis.

Yes, that's a good feedback. Few other people also mentioned this. I was trying to save space, to spend time more on the "how things work" part. But I see your point that some motivation behind parallelization would make it a more interesting read.
Post reply on HN