Live data from Hacker News

Rubygems.org AWS bill for Feb 2014 [pdf]

dropbox.com

111–120 of 153 posts

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#111
post #53

Earlier quoted context omitted.

Do you know who are the biggest consumers of bandwidth? I would guess the CI servers (Travis, Circle)

A very good question. I'll see about crunching some of the logs to break it down by subnet.

Hey, as I mentioned in another part of this thread, my startup crunches those logs for a living (and they're sadly not really designed for crunching by anything that comes off the shelf). Ping me if you'd like a hand doing the crunching.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#112
post #16
post #10

The thing is there are many providers who can do the same and most of them will do it for less than half of this. Some less than 1/5th. I think they should move this to Digital Ocean and save $5000. The bias towards AWS for this type of application is ridiculous and a big waste of money.

> The bias towards AWS for this type of application is ridiculous and a big waste of money. They could get an even better deal by just going through a dedicated server provider (or even better, colocating). There's little advantage with choosing DO versus going with a dedicated server provider (and again, colocating). I guess the advantage would be the control panel that they wouldn't use, having a few one-click stac…

at $600/month you've only saved them $1500/month (the hosting portion is only $2.1k), and now they also have physical hardware to manage, requiring a broader skillset from the volunteers, plus someone having to be in physical proximity for 'on-call' issues.

I don't know what datacentres tend to charge for data transfer, but as that's the largest item on the bill, it's the more salient point.

Also, just because it's not on the bill doesn't mean they're not using other AWS services; there are several free ones.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#113
post #41
post #37

Earlier quoted context omitted.

You miss the point; bandwidth from a single source != CDN. Hosting without a CDN would be extremely slow for people that were far (hops) away from DO. CDN's solve this, and other issues.

How much does that really matter? Even going to the otherside of the world is only 200ms or so, and the time taken to run rubygems is hardly a factor in just about any workflow I can imagine. Think of it another way - what would be more valuble - RubyGems hosted on CDN, or RubyGems on DO and give a couple of grants for talent hackers to work on their gems fulltime for a few months. (ala GSoC) Even if you ARE concerne…

That's been my impression so far with repositories where you configure a mirror explicitly, like Debian or CPAN. I used to be diligent about doing "the right thing" and switching out the default (usually something in the U.S.) for a Danish or nearby mirror. But I've stopped caring much because it doesn't really seem to make any perceptible difference. If I remember to, I still will switch it just so I don't unnecessarily waste intercontinental infrastructure, but it doesn't make much difference to my own experience.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#114
post #88
post #85

Earlier quoted context omitted.

I still don't understand how AWS was preferable to a dedicated server host. Could you elaborate on that?

Virtualization allows us to spin up new instances and migrate traffic to them. This means we can work entirely from chef and keep things clean. This is important for our volunteers to have a complete picture of an instance and to be able to make new ones.

You can easily do that on dedicated hardware too. We run all our stuff in vm's and containers even on the office dev servers 3 meters behind my desk.

And pretty much "all" dedicated server providers these days also have cloud offerings if you need to spin up some instances quickly to handle traffic spikes etc., or for dev/testing purposes.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#115
post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

No question. Though I'll use the occasion to thank you for all the dedication, financial commitment and awesome software you've provided us with in the Ruby community.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#116
post #8

Earlier quoted context omitted.

> which costs less than the fully-loaded cost of a single billing clerk in your local municipal water department. To be fair, a lot of maintenance value goes into the software that is never quantified. Broken software breaks hard, not partially, so maintenance is even more crucial.

When a levy breaks people die. Software maintenance and damage is nothing compared to real engineering.

That may be true of web development, but certainly isn't of software as a whole: http://en.m.wikipedia.org/wiki/Therac-25

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#117
post #8

Earlier quoted context omitted.

> which costs less than the fully-loaded cost of a single billing clerk in your local municipal water department. To be fair, a lot of maintenance value goes into the software that is never quantified. Broken software breaks hard, not partially, so maintenance is even more crucial.

When a levy breaks people die. Software maintenance and damage is nothing compared to real engineering.

What about the defense industry? People die if you screw up. I mean, people die if you don't, too, but you know what I mean.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#118
post #8

Earlier quoted context omitted.

> which costs less than the fully-loaded cost of a single billing clerk in your local municipal water department. To be fair, a lot of maintenance value goes into the software that is never quantified. Broken software breaks hard, not partially, so maintenance is even more crucial.

When a levy breaks people die. Software maintenance and damage is nothing compared to real engineering.

It was never definitively proven but poorly designed software was considered to be at the heart of a helicopter crash that killed 25 people including almost all of the UK's top North Ireland intelligence experts: http://en.wikipedia.org/wiki/1994_Scotland_RAF_Chinook_crash...

Software controls everything from nuclear power stations to missles to dams to radiation therapy machines (where, again, software killed 3 people - http://en.wikipedia.org/wiki/Therac-25)

Proper software engineering is increasingly more important and, I'd posit, likely to become even more important than civil engineering for public safety as time goes on.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#119

Earlier quoted context omitted.

This bill is 2/3 bandwidth, and 1/3 compute. Some games require massive amounts of compute, but the bandwidth to deliver the assets is generally paid by Apple. I can guarantee you, your company is paying a metric fuck-ton more. It is called Apple's 30% cut. Your company is paying AWS $200k to pass json messages around for analytics and social aspects of the game. You are paying Apple something like $1 million per wee…

We aren't paying that much in cut just yet. We're a small team (6 engineers in total). You don't have to be pulling in millions per week to get high on the grossing charts. We're probably around 1/4 of what you estimated. Though you mention delivering the assets. Actually (like a lot of games) we make a big effort in getting under 50MB over-the-air limit on the App Store. The total content for retina iPhone is ~300MB…

FYI, the OTA limit was increased to 100MB back in September.

We're managing to squeeze our apps into this at the moment, but will likely need a similar solution using S3/CloudFront in the near future.

[1] http://www.macrumors.com/2013/09/18/apple-increases-over-the...

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#120
post #75

While by no means insignificant, this bill is no where near what I'd imagine would warrant a HN post. I wouldn't be surprised if most startups beat this regularly. The startup whose backend I co-created racks up an AWS bill that hovers around a half million dollars a month. We make use of all of the ways to save with Amazon: pre-paid reserved instances, negotiated deals, etc. And we're not even that big; imagine what…

> this bill is no where near what I'd imagine would warrant a HN post. Can only guess at why folks like any post, but it's not necessarily how large the bill is. Maybe it's how low it is for a service that's widely relied on, or maybe it's the level of transparency, which turned out to include evanphx above showing up to answer questions about the project.

Absolutely, this is a transparency thing.

Compared to npm asking for $300,000 in donations to keep the thing running. I'm glad RubyGems can run for relatively so little, and be transparent in doing so.

Post reply on HN