Live data from Hacker News

My manager spent $1M on a backup server that I never used

blog.dijit.sh

241–244 of 244 posts

Re: My manager spent $1M on a backup server that I never used

#241
post #205

Earlier quoted context omitted.

> I see what you mean, but with ACID you can tell the user right away if the order was successfully placed. With eventual consistency you don't know how long it'll take so you either show them a loading spinner that takes an indeterminate amount of time, or what? It's doing the same thing either way though; in either world you can even wait for the order to be processed, or not. In an event sourcing system you'll com…

There is a difference between theoretical worst case and practical experience. Whether the direct-update experience or the event sourcing experience on this point is better depends on a lot of factors (many of which are dimensions of scale.) Neither is categorically ideal, you've got to understand the particular application. Of course, when using an ACID RDBMS, you can also very trivially use a CQRS/event-sourcing ap…

> when using an ACID RDBMS, you can also very trivially use a CQRS/event-sourcing approach for some flows with append-only event log tables and a separate process applying them to query tables that other clients only read and direct update for other flows.

Sort of. You can't opt out of transactionality (transaction isolation settings are often global) and you can't defer index updates except by making your indices "manual".

The first job I worked for essentially built an event sourcing system on top of an RDBMS. We still had the production system brought down by an analyst leaving a transaction open for weeks (we weren't big enough to have a separate analytics datastore at that point). They should've known to enable autocommit (even though their queries were read only!), but it's still a footgun that was only there because of the ACID RDBMS.

Re: My manager spent $1M on a backup server that I never used

#242
post #238

Earlier quoted context omitted.

Your boss uses Excel and doesn't like it when the reports\ read\ like\ this because you ignored his requirements. Your coworker is also angry at you now, because when he scheduled your script to run regularly, the output occasionally turns into gibberish because of inconsistent handling of special characters and error conditions, breaking automations randomly. I spotted several errors in your provided "solution", whi…

saying your thousands of servers wont have ssh keys installed when they have arbitrary software installed is disingenuous at best. thousands of servers are managable only with orchestration solutions, this is a solved problem. the unix world also has evolved from a bash for loop to ssh into servers and read outputs. i can solve your task in comparable time with pyinfra or ansible executing arbitrary python code on ev…

> thousands of servers are managable only with orchestration solutions

I've done "thousands of servers" with PowerShell, with no expensive and complex orchestration solutions.

Re: My manager spent $1M on a backup server that I never used

#243
post #219
post #205

Earlier quoted context omitted.

> I see what you mean, but with ACID you can tell the user right away if the order was successfully placed. With eventual consistency you don't know how long it'll take so you either show them a loading spinner that takes an indeterminate amount of time, or what? It's doing the same thing either way though; in either world you can even wait for the order to be processed, or not. In an event sourcing system you'll com…

> in an ACID RDBMS you'll wait potentially arbitrarily long for your commit to execute (and maybe if you're lucky your database has got a deadlock detector, but what are you going to if it tells you you hit a deadlock? 100% of the time I've seen the answer is "backoff and retry"). I don't agree with the conclusions from this. re: deadlocks, these can be prevented as they are only possible in certain situations, and h…

> re: deadlocks, these can be prevented as they are only possible in certain situations, and have mitigations (keep transactions short, never acquire the same locks in a different order, ...).

In principle yes, but this relies on human vigilance; as far as I know there's no automatic checker that can reliably tell you whether your queries have the possibility of deadlocking. Do you review every query before it gets run? And when you miss a case like acquiring locks in the wrong order, it can be weeks or months before it actually bites you.

> But I would still maintain that, even if you have a commit that times out, it does so in an atomic way. It takes at most your "statement timeout" (should be a few seconds probably), and then you can (in deterministic time) show the user an error message. This is still an improvement for the user experience over showing a "your order has been placed" message and then later cancelling it due to overconsumption of inventory.

One of the most fun ways I've seen an SQL system break: user navigates to a page, gets a timeout in their browser; 23 days later the database falls over.

(the page initiated a query for 2 years' worth of data, the database server chugged away through its indices for 23 days and then started trying to stream all the data back).

I agree that it's good to have that kind of fallback behaviour - in the system I currently work on we have something like that, where if a complex process doesn't get an event for over 1 second (most likely because the thing computing it broke, but it could also just be slow) then we have a simple consumer downstream that just emits a cancel event in that case (and passes everything else through otherwise) and we take that stream as canonical. And having something like that by default is a good thing, and one of the things that SQL databases do right is that they're a lot more request-response, whereas event sourcing things can be a bit "shouting into the void". I'd like a system with better support for that kind of case. That said, I think in a lot of cases the SQL defaults aren't great - I don't think I've ever used a database that had a good default timeout setting, and the way SQL databases treat any validation failure as "drop the data on the floor" is rarely what you want in practice.

Re: My manager spent $1M on a backup server that I never used

#244
post #71
post #35

How does one force exponential backoff of clients without taking in any load on themselves? Is it some sort of TCP window congestion control futzing? But even that wouldn't work since the client can do whatever the heck it wants.

Hi, I’m the author of the article and I’m horrified by how many spelling mistakes I made. I must have written this in a rage. To answer your question, since we had control of the clients (since its a game) we used a proof of work challenge on TLS handshake which increased in complexity the more failed attempts you gave us. Very cheap on the server, very expensive on the client, which effectively rate limited connecti…

I see. But still after the initial TLS handshake. So clients can still connect over and over and force your server to do TLS handshakes all day long.
Post reply on HN