Live data from Hacker News

A million-dollar engineering problem

segment.com

121–130 of 265 posts

Re: A million-dollar engineering problem

#121

Addendum missing from the article: Segment charges per request, these users who run performance testing with analytics enabled should be very profitable. Business rule => You gotta optimize for your cash cows. ;)

Actually, our pricing is no longer based on API calls-- https://segment.com/pricing .

$10 / 1000 MTUs

"Monthly Tracked User, or the sum of logged in plus anonymous users that access your product in a given month. "

P.S. I understand that you have volume discounts.

Re: A million-dollar engineering problem

#123

Earlier quoted context omitted.

Reliability wise, do you actually use these services, as in my experience that's a complete myth. One of my clients on a dedicated server has never gone down. The site is blazingly fast, barely touches 5% CPU and pages have sub 50ms response times. Deploys take 10 seconds or so, I could make it faster but it's not really worth the cost/benefit. My client on Azure, with a significantly lower visitor count, pays 3-10 t…

> on top of that there's a 3-5ms delay between the database and website which causes all sorts of performance problems Yea that's something I don't get too. With all of that IPC going over amazon's network for all of their services how much time are programs wasting sending and waiting for messages from other amazon services? I never really thought about that. If someone could measure this it would be interesting.

I can't comment on Azure, but most of the complaints about networking issues in AWS went away when they redesigned their networks around an SDN model with VPC. We see consistent <1ms RTTs within region, and <300µs within availability zone (datacenter).

Re: A million-dollar engineering problem

#124

Earlier quoted context omitted.

There are companies "in the trough" who are contently spending tens of millions a year on AWS. It's hard to give generic advice for all situations-- as with so much else, context matters heavily. Those marginal gains could potentially pay for a dozen engineers...

My point was based on how dynamic the architectures are (or aren't) of those in the trough, not their level of spending.

Ah, my apologies!

It depends. I have a toy service that exists entirely as an autoscaling group. That's pretty dynamic!

Counterpoint: Mainframes.

Re: A million-dollar engineering problem

#125

I work in a start up, we own all our own hardware, and it is HELL. We are forced to pay extremely large sums of money to upgrade our infrastructure as any purchase requires a redundant piece as well. For example we have used 90% of our SANs storage, our IO is suffering and now were looking at purchasing two $10k SANs to upgrade. In the meantime, we have probably spent over 10k worth of development time to compress, c…

Exactly. Neither path is easy to walk; it always comes down to trade-offs. What do you value?

Re: A million-dollar engineering problem

#126
post #119

Earlier quoted context omitted.

> At this point simply not using AWS is a competitive advantage. Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction / understanding. If you blindly run things on AWS without an understanding of the costing model, that'll work for a time. As you scale, you start to realize "oh my god it runs on money." There are myriad ways around that, but a blanket "never use AWS" isn't goi…

I would guess the network egress costs, specifically, might justify a blanket statement like... "If you have content where you can't put a cache in front of it, and it drives significant bandwidth use, then AWS is not a great option"

This. I really feel like egress traffic costs is the most overlooked issue with popular cloud services like AWS or Google Cloud. Charging 9 or 12 cents for a mere GB of regular traffic is just insane, keeping in mind that even most CDNs are cheaper.

Data driven applications on AWS are already expensive because of the high traffic costs, but storing/serving media on S3 single handedly lets your bill explode. For everyone who feels like the statements above are exaggerations, just check out their actual prices (https://aws.amazon.com/ec2/pricing/on-demand/) in comparison with a regular CDN (https://www.stackpath.com/pricing/).

Re: A million-dollar engineering problem

#127

Earlier quoted context omitted.

> At this point simply not using AWS is a competitive advantage. Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction / understanding. If you blindly run things on AWS without an understanding of the costing model, that'll work for a time. As you scale, you start to realize "oh my god it runs on money." There are myriad ways around that, but a blanket "never use AWS" isn't goi…

How do you approach a horrifying AWS bill? Can you recommend any reading? I’m going to need to optimize a tiny fleet on GCE pretty soon.

It gets into the specifics of "what makes it horrifying" pretty quickly. It usually boils down to "nobody's responsible for cleaning things up," a misunderstanding of the billing model, and by the time anyone cares to fix it the billing report is multiple gigabytes.

Step 1 is always "figure out what you've got, and why it is this way."

Re: A million-dollar engineering problem

#128
post #97

Earlier quoted context omitted.

This was several years ago before we had the "growth hackers" lexicon. One of our customers bought the entire company to get a hold of the core team + essentially continued the "30%" deal as a long term incentive as convertible equity. Was a good run, and reinforces my post of if you can bring measurable / substantial value to large enterprise companies amazing things can happen. Large enterprise have fun accounting…

What would you say is the most important, specific skill required in conversion rate optimization? Is it copywriting? Is it UI/UX? Is it analytics? Or would it be fairer to say CRO is holistic, requiring a multitude of skills?

Great question, by the end of the process we were experts @ holistic level.

But started with very basic web design skills at first.

From the start key "skill" for the pitch / and our clients was we had the BANDWIDTH to focus on implementing testing as a regular / rigorous process for them.

EG most - even big cos - wind up with their good web dev resources completely tied up with 100 other things.

And 10 departments competing for their attention.

I came in and said we can do this, full service and you just have to approve the tests and give us a dev environment.

But started with just basic web design skills / willingness to read the multitude of good resources / books / blogs / thought leaders on CRO an apply their ideas.

EG - Common CRO theme - "try different, high-contrast button colors on your call to action"

Ok, I can do that.

- Take existing page

- set up optimizely or VWO (at the time we used home brew or GoogleWO!)

- make some really great buttons (or outsource on elance for $10)

- get client buy in

- set up the test

- ensure production readiness / testing

- go live

- provide nice reporting format for client weekly that lets them stay involved and see results, and have confidence in your ability to execute.

- prepare next test while first one is running, and remember that a huge % of your best ideas will fail, be agnostic to results but be statistically honest and educate clients under same process / instruction

Simple example, but you can ramp the complexity up from there.

Like "better to have this "sign up now" button go to another page w/form or pop a modal window w/form?"

and on down the list, the CRO blogs / experts / etc have 1000 ideas.

And over time got better about understanding what did / didn't work across clients.

So my win rate crept up from say 20% of tests significant win to 50%+ out of gate.

And every person on my team I hired because we wanted to run a test that I / we couldn't implement ourselves :)

What's your specific interest / skill set that you're trying to adapt over?

Re: A million-dollar engineering problem

#129
post #42

Earlier quoted context omitted.

Also as a SaaS founder running on AWS, I would totally do this once our AWS bill is in the 4-5 figures.

The savings are too miserable to pay a consultant. Remember that he has to charge at least $1000 a day. He's more expensive than your entire bill.

Right. Surprisingly, the comment you're responding to resembles some of my clients. This is a business problem.

In this case it's not "save us a few grand" that they're asking for-- the real ask is "help us analyze and forecast our spend as we continue to grow." Identify the knobs and dials that impact infrastructure costs, devise a costing model that states "each additional user costs $X to service," identify what makes sense to arbitrage between AWS and other providers / on-prem...

If I'm being honest, "lower the bill" is where it starts, but not nearly where it ends. :-)

Post reply on HN