Building your own deep learning computer is 10x cheaper than AWS • Hacker News Board

Re: Building your own deep learning computer is 10x cheaper than AWS

#181

Post by jbgreer » Tue, Sep 25, 2018, 5:47 PM UTC

grogenaut wrote: post #100

drewg123 wrote: post #85

Earlier quoted context omitted.

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

When I started at Amazon they gave us dual 22" monitors. I bought dual 27" dells and a gfx card to back them and plugged it in. I also explained how much I was swapping with 8gb and a virt (45 minutes of lost productivity a day), and ram was $86. Manager happily expensed ram for me and the rest of the team . 2 years later that was the standard setup. Now I have 7 monitors and 3 PC's from multiple projects and oses an…

A colleague at my office recently brought in a 43" 4K TV to use as a monitor. He was reimbursed immediately.

After a few days I bought one, too, took it home, and took the 2x 23" monitors it replaced to the office.

In a different department another friend systematically acquired new large monitors for her dev team. Her management chain complained about the expense.

Bias for action not only gets you far, it attracts folks to your team inside a company. ;-)

Re: Building your own deep learning computer is 10x cheaper than AWS

#182

Post by ghostbrainalpha » Tue, Sep 25, 2018, 5:49 PM UTC

spullara wrote: post #174

unbearded wrote: post #165

Earlier quoted context omitted.

Unfortunately, the fight goes even further than that when you go against the cloud. Last week I was in an event with the CTOs of many of the hottest startups in America. It was shocking how much money is wasted on the cloud because inefficiencies and they simply don't care how much it costs. I guess since they are not wasting their own money, they can always come up with the same excuse: developers are more expensive…

Only a Rails developer would think 6k requests per minute with 40ms latency is reasonable with all that hardware. If you rewrote it you probably only need 1 server but you will probably make an argument about how developer time is more valuable :)

I get Rails people are crazy biased, but I still don't understand your argument.

What type of business that someone would basically run out of an office closet like this would need to service more than 6k requests per minute?

Re: Building your own deep learning computer is 10x cheaper than AWS

#183

Post by [deleted] » Tue, Sep 25, 2018, 5:49 PM UTC

nabla9 wrote: post #122

dagw wrote: post #29

You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware Much easier to spend huge amounts of money of Azure/AWS and politely tell them it's their own fucking fault when they complain about the costs. (what me? no I'm not bitter, why do you ask?)

That's very passive aggressive way to deal with it. If that's your only option you are really a cubicle slave in corporate hell. In my opinion it's better to escalate upwards with proposals and not back down easily. You just have to frame it correctly and use right names and terms. * Usually big companies understand the concept of "a lab" that has infrastructure managed outside the corporate IT. Once you fight the ha…

[deleted]

Re: Building your own deep learning computer is 10x cheaper than AWS

#184

Post by Thrymr » Tue, Sep 25, 2018, 5:56 PM UTC

TeMPOraL wrote: post #39

raverbashing wrote: post #28

Earlier quoted context omitted.

> Last time I checked baking my own cookies is more expensive than just buying a bag at Walmart. Yes, but most likely they aren't of comparable quality

Quite likely, the Walmart ones are better, at least if it comes to taste and looks. Unless you're an expert baker. And then your cookies probably still aren't better, you will just claim they are, and it's polite to agree with that assessment. In fact, it's polite in our society to agree that any half-good DIY attempt has better quality than commercial equivalent, even though it doesn't stand up to scrutiny.

If you had picked another example of pre-made food, I might agree, but cookies are really easy. Just follow the recipe on the back of the chocolate chip bag, you'll get something that is far better than anything out of a bag that's engineered to sit around in a store for months.

Re: Building your own deep learning computer is 10x cheaper than AWS

#185

Post by mrguyorama » Tue, Sep 25, 2018, 5:58 PM UTC

titanomachy wrote: post #161

oneshot908 wrote: post #145

Earlier quoted context omitted.

Indeed, off-topic, but I find it fascinating that I can buy a year of Kosher, Vegan, Gluten Free Survival Food on Amazon from ~$1469. https://www.amazon.com/NorthWest-Fork-Gluten-Free-Emergency-... That drops to $1149 if I don't require Vegan, non-GMO, Kosher: https://www.amazon.com/Healthy-Survival-Food-Emergency-Prepa... SNAP benefits at anywhere from $134 to $192 a month are both more than sufficient for this. So…

Normal food can be even cheaper. You could probably indefinitely avoid starvation with dried rice and beans, oil, and a vitamin supplement for $2/day (in NA), without a big upfront cost. Poverty is complicated.

You've ignored the cost of the home required to cook those things in, which is really the whole point.

(not directing at you, more explaining to those who didn't already know)

Re: Building your own deep learning computer is 10x cheaper than AWS

#186

Post by gebeeson » Tue, Sep 25, 2018, 6:10 PM UTC

xfitm3 wrote: post #107

grogenaut wrote: post #100

Earlier quoted context omitted.

When I started at Amazon they gave us dual 22" monitors. I bought dual 27" dells and a gfx card to back them and plugged it in. I also explained how much I was swapping with 8gb and a virt (45 minutes of lost productivity a day), and ram was $86. Manager happily expensed ram for me and the rest of the team . 2 years later that was the standard setup. Now I have 7 monitors and 3 PC's from multiple projects and oses an…

TIL Bias for action.

As did I and the excellent rabbit hole that it led to.

Re: Building your own deep learning computer is 10x cheaper than AWS

#187

Post by vl » Tue, Sep 25, 2018, 6:11 PM UTC

grogenaut wrote: post #100

drewg123 wrote: post #85

Earlier quoted context omitted.

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

When I started at Amazon they gave us dual 22" monitors. I bought dual 27" dells and a gfx card to back them and plugged it in. I also explained how much I was swapping with 8gb and a virt (45 minutes of lost productivity a day), and ram was $86. Manager happily expensed ram for me and the rest of the team . 2 years later that was the standard setup. Now I have 7 monitors and 3 PC's from multiple projects and oses an…

When I worked at Amazon in 2009 I brought my own monitor, chair and keyboard, because standard one were, well... sub-standard. :)

Re: Building your own deep learning computer is 10x cheaper than AWS

#188

Post by tetha » Tue, Sep 25, 2018, 6:16 PM UTC

mlthoughts2018 wrote: post #172

tetha wrote: post #164

Earlier quoted context omitted.

I dunno, that sounds like the a good way to do it. I kinda shake my head if people call for devs to handle all the infrastructure and everything in prod. Why should a developer concern himself with details of scaling and tuning postgres, elasticsearch or loadbalancers? If you don't outsource that, that's the job for the ops team. However, if that's the responsibility of the ops team, there's no reason for dev to have…

I’m a machine learning engineer. I’d love it if I only needed to worry about machine learning and application interfaces. Instead, because ops / infra arbitrarily block me from what I need, I have to be an expert on database internals, network bottlenecks, app security topics, deployment, containers, CI tools, etc. etc., both so that I can “do it myself” when ops e.g. refuses to acknowledge some assumption-breaking G…

This is so frustrating to read as a service provider.

Especially because a well-done infrastructure scales so much harder than the manual style. We're currently dealing with a bit of fallout from a bureaucratic system at one point. People are so confused because to get a custom container build, all they do is create a repository with a specific name and a templated Jenkinsfile and 10 - 15 minutes later they get a mail with deployment instructions. It's so easy, and no one has to do anything else.

Re: Building your own deep learning computer is 10x cheaper than AWS

#189

Post by dekhn » Tue, Sep 25, 2018, 6:17 PM UTC

drewg123 wrote: post #85

VHRanger wrote: post #70

Earlier quoted context omitted.

They seriously can't buy a graphics card and slap it in the PCIe slot?

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

I always waited until my interns left and then "collected" their monitors. Eventually I had too many monitors (Just managing the asset tags and reconciling things was a huge pain.), and replaced everything with a small 24" 4K.

Re: Building your own deep learning computer is 10x cheaper than AWS

#190

Post by vl » Tue, Sep 25, 2018, 6:23 PM UTC

drewg123 wrote: post #85

VHRanger wrote: post #70

Earlier quoted context omitted.

They seriously can't buy a graphics card and slap it in the PCIe slot?

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

I have a hilarious story about this from Google:

I wanted second 30" monitor, so I filed a ticket. They sent me long email listing reasons why I shouldn't get a second monitor, including (numbers are approximate, employee count from 2013 or so) "If every googler gets an extra monitor, in a year it would be equivalent to driving Toyota Camry for 18000 miles."

I'm thinking "this can't possible be right", so I spend some time calculating, and it turned out to be approximately correct. So I'm thinking, "we should hire one less person and give everyone an extra monitor!". I replied "yes, I understand, go ahead and give me an extra monitor anyway". They replied "we'll require triple management approval!". Me: "please proceed". First two managers approved the request, director emails me "Why I'm bother with this?". Me: "they want your approval to give me second monitor", him: "whatever".

And finally few days later I got a second monitor...