A million-dollar engineering problem • Hacker News Board

Re: A million-dollar engineering problem

#241

Post by fpp » Fri, Mar 17, 2017, 3:44 PM UTC

QuinnyPig wrote: post #51

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

> At this point simply not using AWS is a competitive advantage. Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction / understanding. If you blindly run things on AWS without an understanding of the costing model, that'll work for a time. As you scale, you start to realize "oh my god it runs on money." There are myriad ways around that, but a blanket "never use AWS" isn't goi…

Running / monthly cloud cost and comparison to other options (e.g. run on a rented dedicated machine / co-location) should already been looked at when you do even the most basic business plan.

Generally most of the time the best approach is to create solutions that can scale out independent from the HW / infrastructure they run on. With frameworks like K8s / Docker / e.a. this is now relatively easy.

With that for example you can start of with a few small cloud machines / docker containers. When you get continuous load - migrate some of them to 1-2 dedicated machines. Same with peak load - just spin up more images.

All that of course depends on what kind of application you are providing. If you are running a data intensive apps you might start directly with a dedicated machine.

Easy in, expensive out. One cost with cloud services many forget is the cost to get out again. Have a look at the AWS / Azure price-lists.

Re: A million-dollar engineering problem

#242

Post by patrickg_zill » Fri, Mar 17, 2017, 3:49 PM UTC

rbranson wrote: post #239

patrickg_zill wrote: post #235

Earlier quoted context omitted.

I would bet you 1000 USD that it would be eye-opening to buy a 64gb RAM, 8 core CPU server, put 4 SSDs in it in RAID10 config, and test your entire stack on just that box using Docker, KVM, whatever. Just running it beside your desk and having your other devs beat on it. You might have to stub out some things using e.g. Cassandra or another open source alternative; just use a Docker or VM image without the production…

I've operated on-prem and colo physical data centers. I've also been part of larger companies where someone else deploys baremetal on my behalf. I've been running infrastructure on top of AWS off-and-on since 2008. I've done a live migration of one of the largest Internet sites from AWS to baremetal in data centers. I feel pretty good about keeping our infrastructure in AWS for the foreseeable future.

I apologize; I didn't read that part of your bio on the site.

Your post seemed to indicate that you hadn't done any calculations for the bare metal server with your own vms scenario. But clearly you have, if only mentally.

Re: A million-dollar engineering problem

#243

Post by biot » Fri, Mar 17, 2017, 4:54 PM UTC

abalone wrote: post #199

eastbayjake wrote: post #166

Earlier quoted context omitted.

> Why would you pack duplicate containers like that instead of using one larger container? Microservices that are scaling horizontally to handle concurrent requests, etc. Depending on how seriously your architects take the "micro" in microservice, you may easily get to 100+ unique microservices/processes each running in their own Docker container.

That's not what's going on here (100-200 unique microservices). Aside from the fact that that's insane , they specifically state they want to "pack" multiple containers of the same service on one machine: "You cannot run two containers of the same service on a single host because they will collide and attempt to listen on the same port. (no bin packing)" So, what's the sense in doing that instead of one bigger contai…

They're reimplementing the actor model using Docker containers. Let it crash.

Re: A million-dollar engineering problem

#244

Post by jackgolding » Fri, Mar 17, 2017, 5:14 PM UTC

aresant wrote: post #128

omarchowdhury wrote: post #111

Earlier quoted context omitted.

What would you say is the most important, specific skill required in conversion rate optimization? Is it copywriting? Is it UI/UX? Is it analytics? Or would it be fairer to say CRO is holistic, requiring a multitude of skills?

Great question, by the end of the process we were experts @ holistic level. But started with very basic web design skills at first. From the start key "skill" for the pitch / and our clients was we had the BANDWIDTH to focus on implementing testing as a regular / rigorous process for them. EG most - even big cos - wind up with their good web dev resources completely tied up with 100 other things. And 10 departments c…

I have an analytics background and have been applying for multiple CRO jobs. It's very frustrating the breadth of skills that people want the inhouse CRO guy to have (i.e I've had interview questions on JS frameworks, very advanced stats, non-trivial SQL, CMS systems, photoshop, SEO, paid search, Salesforce ...) especially when I know a lot of whole agencies couldn't tick all those boxes.

Would love to start freelancing this stuff but it seems like it needs so much personal branding

Re: A million-dollar engineering problem

#245

Post by eip » Fri, Mar 17, 2017, 5:15 PM UTC

pm90 wrote: post #53

GFischer wrote: post #41

Earlier quoted context omitted.

I disagree. Depends on the service's business model obviously, but there are services where the convenience will more than offset the extra costs. Once the company grows big enough, yes, then it might make sense to go off AWS (or not - see: Netflix).

I've always found it interesting that Netflix has not at least tried to go off AWS; if only because Amazon has a competing service with Amazon video. I'm sure its not an easy problem to solve, but Dropbox has gone off AWS as well; it seems like they would be much better served with their own "Video Cloud" with specialized hardware for streaming/processing videos.

Netflix doesn't pay retail prices for AWS.

Re: A million-dollar engineering problem

#246

Post by eip » Fri, Mar 17, 2017, 5:23 PM UTC

patrickg_zill wrote: post #231

walrus01 wrote: post #87

Earlier quoted context omitted.

A million dollars buys a shitload of fast 1U dual socket xeon servers each with 512GB of RAM.

There are companies that will lease you a dedicated server with 40 cores and 256gb RAM for $499/month. Figure a 10% quantity discount and you are looking at at least 200 of those for about $1 million per year.

Hetzner will rent you 24 cores (48 'cloud' cores), 1024GB RAM, and 4TB SSD for less than $499/month.

Re: A million-dollar engineering problem

#247

Post by cagataygurturk » Fri, Mar 17, 2017, 5:57 PM UTC

hueving wrote: post #202

ig1 wrote: post #157

However they miss the easiest fix: Calling their account rep at AWS and cutting a deal. AWS loves startups that could end up being huge customers so they're willing to slash bills upfront to help you get to growth stage; not only will they assign you an account rep but they'll have a rep whose job it is to build a good relationship with your VC. Speak to your account rep. Have your VC speak to their Amazon rep. Push…

Gross. AWS is becoming the Cisco of cloud computing. Get people hooked when they are low revenue and twist the screws in later when they get big.

It's customer obsession, one of the leadership principles of Amazon.

Re: A million-dollar engineering problem

#248

Post by sdflkd » Fri, Mar 17, 2017, 6:38 PM UTC

sillysaurus3 wrote: post #134

sdflkd wrote: post #38

Earlier quoted context omitted.

How does one begin to learn about these things? Minimizing cost of running services sounds super interesting but as a student I've never had to deal with it and am basically starting with 0 knowledge.

I worked through the labs at https://pdos.csail.mit.edu/6.824/ for fun. It's more along the lines of "How can we write a distributed fault-tolerant database?" but you might like it anyway. Lab 4 is a beast with more lives than . The moment you think you've finally written your distributed system correctly, the unit tests will prove your service fails during XYZ network partition topologies. It's very worthwhile to be…

Wrt the first link, I'm going to be taking a distributed systems course next semester that (hopefully) covers the same material. Nice to know what I learn at school is somewhat applicable.

My question is learning about design ideas like having Redis keeping a list of "work to be done" and such. Using modern tools to combat 'modern' problems. Is it just something you figure out after knowing the fundamentals (i.e. when you can identify a problem you know what the solution needs to be)?

Re: A million-dollar engineering problem

#249

Post by rbranson » Fri, Mar 17, 2017, 6:55 PM UTC

patrickg_zill wrote: post #242

rbranson wrote: post #239

Earlier quoted context omitted.

I've operated on-prem and colo physical data centers. I've also been part of larger companies where someone else deploys baremetal on my behalf. I've been running infrastructure on top of AWS off-and-on since 2008. I've done a live migration of one of the largest Internet sites from AWS to baremetal in data centers. I feel pretty good about keeping our infrastructure in AWS for the foreseeable future.

I apologize; I didn't read that part of your bio on the site. Your post seemed to indicate that you hadn't done any calculations for the bare metal server with your own vms scenario. But clearly you have, if only mentally.

No worries! I'm not super into self-promotion, but I do think my background is relevant in this case. Either way, the decisions that go into a strategy like this involve a mountain of variables.

Re: A million-dollar engineering problem

#250

Post by hueving » Fri, Mar 17, 2017, 7:14 PM UTC

bearjaws wrote: post #106

I work in a start up, we own all our own hardware, and it is HELL. We are forced to pay extremely large sums of money to upgrade our infrastructure as any purchase requires a redundant piece as well. For example we have used 90% of our SANs storage, our IO is suffering and now were looking at purchasing two $10k SANs to upgrade. In the meantime, we have probably spent over 10k worth of development time to compress, c…

>For example we have used 90% of our SANs storage, our IO is suffering and now were looking at purchasing two $10k SANs to upgrade. In the meantime, we have probably spent over 10k worth of development time to compress, clean up, and remove any data we don't need.

I suspect you haven't priced out AWS storage (and the bandwidth to actually access the data with frequency). If the cost of a SAN was hurting your company and you had to look to reduce data usage, AWS will absolutely wipe you out.