Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rubygems.org AWS bill for Feb 2014 [pdf]

dropbox.com

Post reply on HN
Print view View original

51–60 of 153 posts

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#51

Post by TylerE »

evanphx wrote: post #48
TylerE wrote: post #41

Earlier quoted context omitted.

How much does that really matter? Even going to the otherside of the world is only 200ms or so, and the time taken to run rubygems is hardly a factor in just about any workflow I can imagine. Think of it another way - what would be more valuble - RubyGems hosted on CDN, or RubyGems on DO and give a couple of grants for talent hackers to work on their gems fulltime for a few months. (ala GSoC) Even if you ARE concerne…

Person who pays the bill here. Latency does matter and we're paying an additional $1.5k new in Feb to improve european latency. RubyCentral can afford to spend the money to improve latency issues, so we do!

Fair enough!

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#52

Post by teeparham »

evanphx wrote: post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

Do you know who are the biggest consumers of bandwidth? I would guess the CI servers (Travis, Circle)

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#53

Post by evanphx »

teeparham wrote: post #52
evanphx wrote: post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

Do you know who are the biggest consumers of bandwidth? I would guess the CI servers (Travis, Circle)

A very good question. I'll see about crunching some of the logs to break it down by subnet.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#54

Post by jacquesct »

evanphx wrote: post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

Need help with the VCL with Fastly? Drop me a line to my name minus ct @npmjs.com.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#55

Post by knyt »

evanphx wrote: post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

Did you consider using a mirror network, with servers run by external organizations, instead of going with AWS bandwidth for rubygems? Seems like that would be a good approach for the static/bulk part of your dataset, and there are lots of companies and universities who are set up to serve software. (The mirror I manage serves about 50 TB/month for several Linux distros, and many sites are larger.) Do the work and infrastructure required to manage these networks make them not worthwhile?

Edit: Found a post [0] calling for a rubygems mirror network. Otherwise there is lots of information about setting up local mirrors of the repository.

[0] http://binarymentalist.com/post/1314642927/proposal-we-have-...

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#56

Post by ksec »

vertis wrote: post #24
saurik wrote: post #11

Earlier quoted context omitted.

A lot of the price is bandwidth. They are effectively being reamed by using CloudFront instead of negotiating a better rate with a "real" CDN (which will also give then much better performance, as CloudFront doesn't have many edge locations). (Although, actually, while I verified their total dollars spent is greater than what would be required to get a fundamentally better deal on bandwidth, I didn't take into consid…

Could probably fix some of this just by talking to Amazon about it. It's not like this is a 'for profit' setup.

Exactly, and if you include a little Powered by AWS CloudFront i am pretty sure they could drive down the price a lot. Or, they could start talking to Fastly, I am pretty sure they can work out a much better deal while being faster.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#57

Post by briancurtin »

evanphx wrote: post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

> In the past, RubyGems was hosted on dedicated hardware within Rackspace. While this was certainly cheaper, it created administrative issues. Granted those can be solved without using AWS, but we get back to again desiring to have as low of friction on the administration as possible.

If Rackspace can be of assistance in the future, feel free to reach out (brian.curtin@rackspace.com). We currently donate hosting to many open source projects, including ones in a similar space, like the Python Package Index.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#58

Post by evanphx »

knyt wrote: post #55
evanphx wrote: post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

Did you consider using a mirror network, with servers run by external organizations, instead of going with AWS bandwidth for rubygems? Seems like that would be a good approach for the static/bulk part of your dataset, and there are lots of companies and universities who are set up to serve software. (The mirror I manage serves about 50 TB/month for several Linux distros, and many sites are larger.) Do the work and in…

It's been discussed many times before, yes. Rubygems usage pattern by our users make any kind of mirror delay unacceptable. We currently run a number of mirrors, configured as caching proxies. I want to get us going on a CDN like Fastly soon because they provide effectively the same functionality but distributed to many, many more POPs that I will ever setup.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#59

Post by evanphx »

briancurtin wrote: post #57
evanphx wrote: post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

> In the past, RubyGems was hosted on dedicated hardware within Rackspace. While this was certainly cheaper, it created administrative issues. Granted those can be solved without using AWS, but we get back to again desiring to have as low of friction on the administration as possible. If Rackspace can be of assistance in the future, feel free to reach out (brian.curtin@rackspace.com). We currently donate hosting to m…

Thanks! I'll bring it up with the team.

Re: Rubygems.org AWS bill for Feb 2014 [pdf]

#60

Post by evanphx »

jacquesct wrote: post #54
evanphx wrote: post #50

I thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays…

Need help with the VCL with Fastly? Drop me a line to my name minus ct @npmjs.com.

Thanks! I'll definitely keep you in mind as we're (finally) getting around to setting up correctly.
Post reply on HN