Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

561–570 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#561

Post by kevinslin »

iaresee wrote: post #450
viraptor wrote: post #229

Earlier quoted context omitted.

Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs. Then they have things that I wanted to try for a long time, but... support doesn't care? Repeated "would you like to use this? / very likely, can we try it out? / (silence)". I love their product, but they are so annoying to d…

> Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs. I, quite literally, was griping to my Datadog CSM about this exact thing last week. They'll email me and be, "Oh, you know you're logging volume this month put you into on-demand indexing rates, right?" and my answer is alwa…

> You can't reference your contracted volume rates when building monitors out and the units for the metrics you need to watch don't match the units you contract with them on the SKU.

Are you referring to the `datadog.estimated_usage.logs.ingested_events` metric? It includes excluded events by default but you can get to your indexed volume by excluding excluded logs. `sum:datadog.estimated_usage.logs.ingested_events{datadog_index:*,datadog_is_excluded:false}.as_count()`

Re: Almost every infrastructure decision I endorse or regret

#562

Post by ZoomerCretin »

hhsectech wrote: post #317
jumploops wrote: post #280

Earlier quoted context omitted.

Funny story time. I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud. Our calculations said it would cost 3x as much to run our infra on the cloud. We pushed back, and were greenlit on creating a hybrid architecture that allowed us…

There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck! I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere. I personally dont care if the docs are crap because usually the first thing I do is u…

Documentation? What for? It's self-documenting (to me, because I wrote it)!

Re: Almost every infrastructure decision I endorse or regret

#563

Post by yolo3000 »

yellow_lead wrote: post #546
maccard wrote: post #402

Earlier quoted context omitted.

I've just done it a dozen times at this point. Hello world from gin-gonic [0], terraform file with a DO K8s cluster [1] and load balancer, and CI/CD [2] on deploy. There's even time to make a cuppa when you run terraform. We use this for our internal services at work, and the last time I touched the infra was in 2022 according to git [0] https://github.com/gin-gonic/gin [1] https://gist.github.com/donalmacc/0efbb0b37…

Thanks! Does DO K8s come with sufficient monitoring / logging or do you add anything?

You can just deploy other applications to Kubernetes, for example you can deploy this operator https://prometheus-operator.dev/ and you get Prometheus and Grafana running with a bunch of dashboards already created. Then you annotate your pods to tell Prometheus what to scrape, and you got monitoring. It also comes with AlertManager for alerting. Same for logging, you deploy Elasticsearch and Kibana and you're good to go.

Re: Almost every infrastructure decision I endorse or regret

#564

Post by alecthomas »

iandanforth wrote: post #449

For people who enjoyed this post but want to see the other side of the spectrum where self hosted is the norm I'll point to the now classic series of posts on how Stack Overflow runs its infra: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... If anyone has newer posts like the above, please reply with links as I would love to read them.

https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47... is another good one. There are a few different posts on it scattered around:

https://world.hey.com/dhh/we-stand-to-save-7m-over-five-year...

https://world.hey.com/dhh/our-cloud-exit-has-already-yielded...

Related, looks like X is doing similar: https://twitter.com/XEng/status/1717754398410240018

Re: Almost every infrastructure decision I endorse or regret

#565

Post by pclmulqdq »

nemothekid wrote: post #399
pclmulqdq wrote: post #186

Earlier quoted context omitted.

I assume that you do that math on all your new features too, right? The calculation of how much extra money they will bring in? On some level, AWS/GCP/California relies on you doing this calculation for the things that you can do it on easily (the savings of moving away), while not doing this calculation on things where it's hard to do (new development). That way, you can pretend that your new features are a lot more…

> The calculation of how much extra money they will bring in? Yes, I've done the math. The piece you are missing is, saving money on infra will bring in $0 new dollars. There is a floor to how much money I can save. There is no ceiling to how much money the right feature can bring in. Penny pinching on infra, especially when the amount of money is saved is less than the cost of an engineer is almost always a waste of…

I am well aware of the math on that. Also, switching to faster infra can be a surprising benefit to your revenue, by the way, if it makes your app feel nicer.

The thing is, most features, particularly later in the life of a company, don't have an easy-to-measure revenue impact, and I suspect that many features are actually worth $0 of revenue. However, they cost money to implement (both in engineering time and infra), making them very much net negative value propositions. This is why Facebook and Google can cut tons of staff and lose nothing off their revenue number.

Also, there's a bit of a gambling mentality here which is that a feature could be worth effectively infinite revenue (ie it could be the thing that gives you PMF), so it's always worth doing over things with known, bounded impact on your bottom line. However, improving your efficiency gives you more cracks at finding good features before you run out of money.

Re: Almost every infrastructure decision I endorse or regret

#566

Post by maccard »

yellow_lead wrote: post #546
maccard wrote: post #402

Earlier quoted context omitted.

I've just done it a dozen times at this point. Hello world from gin-gonic [0], terraform file with a DO K8s cluster [1] and load balancer, and CI/CD [2] on deploy. There's even time to make a cuppa when you run terraform. We use this for our internal services at work, and the last time I touched the infra was in 2022 according to git [0] https://github.com/gin-gonic/gin [1] https://gist.github.com/donalmacc/0efbb0b37…

Thanks! Does DO K8s come with sufficient monitoring / logging or do you add anything?

As the other commentor said, you can deploy Prometheus/grafana into the k8s cluster and it pretty much does what you want it to do.

Re: Almost every infrastructure decision I endorse or regret

#567

Post by oooyay »

dvfjsdhgfv wrote: post #506
oooyay wrote: post #222

Earlier quoted context omitted.

Context: I build internal tools and platforms. Traffic on them varies, but some of them are quite active. My nasty little secret is for single server databases I have zero fear of over provisioning disk iops and running it on SQLite or making a single RDBMS server in a container. I've never actually run into an issue with this. It surprises me the number of internal tools I see that depend on large RDS installations…

The problem with single instance is that while performance-wise it's best (at least on bare metal), there comes a moment when you simply have too much data and one machine can't handle. Your your scenario, it may never come up, but many organizations face this problem sooner or later.

I agree, my point is that clusters are overused. Most applications simply don't need them and it results in a lot of waste. Much of this has to do with engineers being tasked with an assortment of roles these days, so they obviously opt for the solution where a database and upgrades are managed for them. I've just found that managing a single containers upgrades aren't that big of an issue.

Re: Almost every infrastructure decision I endorse or regret

#568

Post by eptcyka »

isbvhodnvemrwvn wrote: post #292
davedx wrote: post #267

Utter insanity. So much cost and complexity, and for what? Startups don’t think about costs or runway anymore, all they care about is “modern infrastructure”. The argument for RDS seems to be “we can’t automate backups”. What on earth?

Is spending time to make it reliable worth it vs working on your actual product? Databases are THE most critical things your company has.

So investing in a critical part of my business is the bad thing to do?

Re: Almost every infrastructure decision I endorse or regret

#569

Post by mixmastamyk »

Cacti wrote: post #503
mixmastamyk wrote: post #274

Earlier quoted context omitted.

Found the person who never used Lotus Notes haha.

I was blown away when I found out a couple years ago that there were major corporations still using that as their primary communication platform.

Surely has improved in the last 20+ years? :hope:

Re: Almost every infrastructure decision I endorse or regret

#570

Post by Sankozi »

layer8 wrote: post #502
Sankozi wrote: post #401

Earlier quoted context omitted.

> How about "database is used by everyone, it becomes cared for by everyone". So every one needs to know every use case of that database? Seems very unlikely if there are multiple teams using same DB. FKs? Unique constraints? Not null colums? If not added at the creation of the table they will never be added - the moment DB is part of a public API you cannot do a lot of things safely. The only moment when you want to…

The database should never ever become part of a public API. You don’t share a DB for performance reasons (rather the opposite), you do it to ensure data integrity and consistency. And no, not everyone needs to know every use case. But every team needs to have someone who coordinates any overlapping schema concerns with the other teams. This needs to be managed, but it’s also not rocket science.

If database is shared it is a part of an API. If it is shared between teams then it is a public API.

If DB is shared then data from different users is entered/updated through multiple transactions. So you cannot get anything better regarding consistency and integrity compared to multiple DBs and distributed TXs.

By introducing schema change coordination you will introduce enormous delays to almost any DB change. This is more realistic than everyone knowing each use case but less practical. Shared DB is an antipattern either way.

Post reply on HN