Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to colocate your first server at a datacenter

blog.definedcode.com

Post reply on HN
Print view View original

41–50 of 67 posts

Re: How to colocate your first server at a datacenter

#41

Post by hosay123 »

detournement wrote: post #19

I am so glad I live in a world where I will never have to do this again for anything but large infrastructure projects. I've colocated servers at major POPs like 111 8th Avenue in Manhattan, Equinix in Seacaucus, One Wilshire in LA. For smaller projects, the convenience of the cloud is absolutely worth the price. For a larger build - like over 10k a month in infrastructure cost the cloud starts to make less sense eco…

> redundancy, backup, remote hands, environmental monitoring, hardware maintenance are just not worth figuring out until there is substantial cost savings to realize.

So you don't need to monitor temperature sensors any more with a VM, but most of the above are still costs with cloud - flaky RAM, redundancy, backups, monitoring, etc. There are also the things you previously didn't have to worry about - crappy resource isolation turning your scratch disks into 2kb/sec joys, total ineffectiveness of the CPU cache, managing a now-essential network fabric to tie pieces of your app together where previously it all fit on 2 master/slave machines, etc.

Of course if your application isn't simply some stock PHP/MySQL application, and you want to really "embrace cloud", then the time you saved fighting a subset of hardware problems is replaced by a fixed development cost integrating with someone else's higher tier APIs (S3, Dynamo, etc) you can then never escape even if you wanted to.

I've never seen any realistic numbers comparing the use of traditional hosting facilities, say, providing managed servers, to the new generation VM stuff. Any material I've seen has been sponsored crap involving some multinational.

My own experience is similar to yours - hosting your own hardware is a pain in the ass. However there is middle ground, there are many colos that will happily provide managed hardware, and perf/pence, this still tends to be far cheaper than the equivalent in VMs, and increasingly they're coming with similar APIs to order/replace machines

Re: How to colocate your first server at a datacenter

#42

Post by IgorPartola »

hosay123 wrote: post #41
detournement wrote: post #19

I am so glad I live in a world where I will never have to do this again for anything but large infrastructure projects. I've colocated servers at major POPs like 111 8th Avenue in Manhattan, Equinix in Seacaucus, One Wilshire in LA. For smaller projects, the convenience of the cloud is absolutely worth the price. For a larger build - like over 10k a month in infrastructure cost the cloud starts to make less sense eco…

> redundancy, backup, remote hands, environmental monitoring, hardware maintenance are just not worth figuring out until there is substantial cost savings to realize. So you don't need to monitor temperature sensors any more with a VM, but most of the above are still costs with cloud - flaky RAM, redundancy, backups, monitoring, etc. There are also the things you previously didn't have to worry about - crappy resourc…

From my limited experience, the cloud is always more expensive if you know your exact usage requirements. If, for example, you know that six octo-core 16 GB RAM, 512 GB SSD-in-RAID1 servers would fit your needs from now until 10 years from now, you will do better to just rent them from SoftLayer, Hetzner, etc.

However, if you anticipate growth, or need to be able to spin up a test server, then shut it down a day later, etc. then you are better off paying premium for the cloud. Sure, there are economies of scale at play here: AWS has so many servers, they are not paying a person to log into every one of them every so often to run updates, etc. However, make no mistake: everything you would have to do with a server, Amazon has to do too. In fact, they have to do much more to keep all of them running at once. That cost will be passed onto you.

Even with all of that, it's cheaper if you want to be able to spin something up, then shut it down. Another great example is the additional services provided by the likes of AWS: you can get things like load balancers, cache servers, database servers, orchestration services, etc. You can do all of this yourself, but at some point it's cheaper to just pay for something like ELB than to learn how to do it yourself and spend the hours to set it up. Human time is more expensive than that.

Lastly, if you just need a really small machine, there is no beating the cloud. You simply cannot get a dedicated machine for $5/month, and you likely never will.

Re: How to colocate your first server at a datacenter

#43

Post by techsupporter »

turnip1979 wrote: post #35
techsupporter wrote: post #26

Earlier quoted context omitted.

I'd like to know more about how you do that. I just looked at Opus' site and the 4U they advertise in Hillsboro is pretty good when compared to what I pay for 1U of hobbyist colo in Seattle. Do you put multiple machines inside the 4U with a switch in the same space? Can I be so bold as to ask for pictures? What kind of physical access do you get? I have a 1U server-grade machine but moving to a more flexible space wo…

I've been interested in hobbyist colo for a while but the cheapest I've found is over $100 a month. Is this in the ballpark of what you are paying or are there better deals to be had?

The rate I'm paying no longer exists and the company I'm with got merged into another one so my point of reference isn't so good. :)

That said, there are a couple of companies on Webhostingtalk that have good prices for hobbyist colo if you look in their colocation forums and search for "Seattle." Opus is nice ($129 for 4U and 3A of power with 400GB of transit) and there is a company in Seattle--their name escapes me but I've seen them on WHT--that is $35 for 1U and 1A with 500GB of transit.

So, yes, I think you can do better especially if you just want space for a medium-usage 1U.

Re: How to colocate your first server at a datacenter

#44

Post by waivej »

csbrooks wrote: post #6

What are the advantages of doing this over just paying for computing power on AWS or some other cloud service? Feels like it would be a very small niche at this point.

I priced and ran benchmarks for my last server upgrade. AWS was 2x the cost of collocating and some outages around the same time. I think it would have been only marginally more expensive since I could scale my server down to current need rather than project 3-4 years into the future.

AWS had the fastest network and but colocation allowed for cheaper CPU and RAM upgrades. As it turns out, my RAM/CPU needs plateaued at 4x my previous server so I have twice as much as I need. AWS got steadily cheaper over that same time and the servers got faster.

Upgrading to a new server is pretty disruptive. It would probably be easier to do slowly over time rather than big jumps.

Re: How to colocate your first server at a datacenter

#45

Post by vacri »

abc123xyz wrote: post #24
csbrooks wrote: post #6

What are the advantages of doing this over just paying for computing power on AWS or some other cloud service? Feels like it would be a very small niche at this point.

cost! Cost!! COST!!! I collocate 5x 4u servers with 24 and 36 drive bays, 128GB ram, SSD drives squished in for OS, for a total (multiple raid 5 volumes for each 6 disks) usable space for my project of 375 TB Power and 2-3gbit of bandwidth is included, as well as remote hands to check up on server (for example IPMI sometimes does not work) or replace drives (pay extra for enterprise grade drives would save you alot o…

400TB s3 storage in US-East is ~US$11800, so it's about 6 times the cost. You also get power and remote hands. Bandwidth isn't free, though, and can get pricey. On the other hand, for data you can shift to Glacier, you only pay a third of that cost. On the other, other hand, if you're using this amount of gear, you may be able to talk to an account manager and get something else sorted out. You'll probably want to throw some VMs in there with S3, of course, so that adds to the cost... 122GB ram reserved instance VMs run to about $700/mo, so with five of those you're looking at about US$15k + bandwidth, though of course you're not paying for the hardware or hardware setup time.

Re: How to colocate your first server at a datacenter

#46

Post by maaarghk »

IgorPartola wrote: post #42
hosay123 wrote: post #41

Earlier quoted context omitted.

> redundancy, backup, remote hands, environmental monitoring, hardware maintenance are just not worth figuring out until there is substantial cost savings to realize. So you don't need to monitor temperature sensors any more with a VM, but most of the above are still costs with cloud - flaky RAM, redundancy, backups, monitoring, etc. There are also the things you previously didn't have to worry about - crappy resourc…

From my limited experience, the cloud is always more expensive if you know your exact usage requirements. If, for example, you know that six octo-core 16 GB RAM, 512 GB SSD-in-RAID1 servers would fit your needs from now until 10 years from now, you will do better to just rent them from SoftLayer, Hetzner, etc. However, if you anticipate growth, or need to be able to spin up a test server, then shut it down a day late…

You can get damn close!

http://www.kimsufi.com/uk/

I found OVH's dedicated server offerings to be so cheap that there was no point in using shared hardware for the flexibility. Then again, I'm not running my entire business on these boxes... but I don't think I'd have a major issue if I wanted to!

Re: How to colocate your first server at a datacenter

#47

Post by abc123xyz »

davb wrote: post #36
abc123xyz wrote: post #24

Earlier quoted context omitted.

cost! Cost!! COST!!! I collocate 5x 4u servers with 24 and 36 drive bays, 128GB ram, SSD drives squished in for OS, for a total (multiple raid 5 volumes for each 6 disks) usable space for my project of 375 TB Power and 2-3gbit of bandwidth is included, as well as remote hands to check up on server (for example IPMI sometimes does not work) or replace drives (pay extra for enterprise grade drives would save you alot o…

I absolutely agree in the long term , but can you say a little about how much those servers cost initially? I tend to favour dedicated servers which I own over VPS but for small businesses with extremely constrained budgets ("prove we can make money before we invest in hardware") and startups, flexible virtual servers can be a great way to ramp up in the beginning.

About $10,000 per server, equipment (very heavy) was shipped from US so there were import duties but it worked out a little bit cheaper than buying in EU in end, and had other reasons to shop in US.

Anyway its a once off cost, that accountant can do all sorts of magic with this.

Ive been in business for 7 years and hope to remain around for as long.

Have already saved money as to compared to renting dedicated servers before. AWS etc were never an option the sums simply never work out.

AWS might be great at first when you are starting out, but the costs can cripple a business, especially if you dont have others peoples venture capital to burn.

edit: when bandwidth costs are factored in the difference is and order of magnitude between my current setup and amazon, at time of this post using 2200 mbit outgoing, 800 incoming

edit2: my only regrets is not collocating earlier, have spent well into upper 6 figures over all the years :( on renting, AWS etc werent around when was starting off either.

Re: How to colocate your first server at a datacenter

#48

Post by Alupis »

detournement wrote: post #19

I am so glad I live in a world where I will never have to do this again for anything but large infrastructure projects. I've colocated servers at major POPs like 111 8th Avenue in Manhattan, Equinix in Seacaucus, One Wilshire in LA. For smaller projects, the convenience of the cloud is absolutely worth the price. For a larger build - like over 10k a month in infrastructure cost the cloud starts to make less sense eco…

I actually find colo'ing a new server quite fun and enjoyable... even after doing it a half dozen times... as a techy... something is just so cool about putting a server in the same racks as some monster equipment. ;-P

However, I disagree about colo'ing not being economical. If you setup a physical box in a colo with a hypervisor (kvm, xen), then the density you can get out of 1U is amazing, and makes the entire thing much more affordable.

Take this site for example: https://www.ubiquityhosting.com/cloud

16GB ram, 8 coers for $128 a month. Considering most 2U colo spots I've seen hover around $150 a month, seems like the colo is not a good deal. However, the VPS for $128 a month is a single server. Your $150 a month colo, with xen, you can pack 3-7 vm's in the same space, making the cost-per-(virtual)server much lower.

Re: How to colocate your first server at a datacenter

#49

Post by neurotixz »

turnip1979 wrote: post #38
neurotixz wrote: post #33

Earlier quoted context omitted.

Yes, always compare with rented dedicated, and only go with colo if it makes sense. I did this analysis last year and rented won big time. Never looked back since. Main points in favor of rented dedicated servers: - Much cheaper in my business case (5-10 TB/month by server, only a few high CPU requirements) most dedicated come with bandwidth included - No need to take care of the hardware in case of failure (opening…

Maybe it is a Canada thing. I looked at some colos in the Toronto area a few back and found rental prices were about the same as bringing your own hardware. Didn't make sense. I was just looking for a home for some server hardware I have in my garage. Oh well.

Almost all the independant colo centers in Montreal have been bought out by big players with the corporate world in sight, not the small 1-2 servers market. So the prices skyrocketed.

The opposite is true for dedicated, since they have to compete with us-based players, and OVH got in the East Coast market big time. Prices are lowering.

Post reply on HN