Live data from Hacker News

Is something bugging you?

antithesis.com

131–140 of 438 posts

Re: Is something bugging you?

#131

interesting, this kind of responsive environment is dear but rare i can't recall the last time i went to a place and people even considered investing in such setups i assume that except for hard problems and teams seeking challenges, most people will revert to the mean and refuse any kind of infrastructure work because it's mentally more comfortable piling features and fixing bugs later ps: i wish there was a meetup…

We'll be starting some meetups, attending conferences, etc. this year. Also hop into our Discord if you want to chat, lots of us are in there regularly. discord.gg/antithesis

oh, that's cool, thanks

Re: Is something bugging you?

#132
post #92

> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size. I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out. But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to…

I'm tired of hearing about 10x engineers. I just want to be a good 1x engineer. Or good at anything in life realy.

The “10x engineer” comes from the observation that there is a 10x difference in productivity between the best and the worst engineers. By saying that you want to be a 1x engineer, you’re saying you want to be the least productive engineer possible. 1x is not the average, 1x is the worst.

Re: Is something bugging you?

#133
post #92

> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size. I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out. But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to…

I'm tired of hearing about 10x engineers. I just want to be a good 1x engineer. Or good at anything in life realy.

It depends on the day if I feel like a 2x or a 0.1x engineer. Keep at it. You are not alone!

Re: Is something bugging you?

#134
post #95

Earlier quoted context omitted.

Except it doesn't actually explain in what it does: Is it fuzzing? Do you supply your own test cases? Is it testing hardware non-determinism?

Post author here. Sorry it was vague, but there's only so much detail you can go into in a blog post aimed at general audiences. Our documentation ( https://antithesis.com/docs/ ) has a lot more info. Here's my attempt at a more complete answer: think of the story of the blind men and the elephant. There's a thing, called fuzzing, invented by security researchers. There's a thing, called property-based testing, inven…

> most software is non-deterministic

Doesn't Antithesis rely on the fact that software is always deterministic? Reproducibility appears to be its top selling feature – something that wouldn't be possible if software were non-deterministic.

Re: Is something bugging you?

#135
post #92

> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size. I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out. But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to…

I'm tired of hearing about 10x engineers. I just want to be a good 1x engineer. Or good at anything in life realy.

Spend less time on HN and you might get more done.

Re: Is something bugging you?

#136
post #92

Earlier quoted context omitted.

I'm tired of hearing about 10x engineers. I just want to be a good 1x engineer. Or good at anything in life realy.

The “10x engineer” comes from the observation that there is a 10x difference in productivity between the best and the worst engineers. By saying that you want to be a 1x engineer, you’re saying you want to be the least productive engineer possible. 1x is not the average, 1x is the worst.

the worst engineer certainly has negative productivity, so I'm not sure that your explanation can possibly be the correct one.

Re: Is something bugging you?

#137

Earlier quoted context omitted.

The “10x engineer” comes from the observation that there is a 10x difference in productivity between the best and the worst engineers. By saying that you want to be a 1x engineer, you’re saying you want to be the least productive engineer possible. 1x is not the average, 1x is the worst.

the worst engineer certainly has negative productivity, so I'm not sure that your explanation can possibly be the correct one.

I’m explaining what the terms “10x” and “1x” mean, not asserting that the original observation is correct under all circumstances.

Re: Is something bugging you?

#138
post #95

Earlier quoted context omitted.

Post author here. Sorry it was vague, but there's only so much detail you can go into in a blog post aimed at general audiences. Our documentation ( https://antithesis.com/docs/ ) has a lot more info. Here's my attempt at a more complete answer: think of the story of the blind men and the elephant. There's a thing, called fuzzing, invented by security researchers. There's a thing, called property-based testing, inven…

I remember watching the Strange Loop video on your testing strategy, and now I need to go back and relearn how it differed from model checking (ie Promela or TLA+). Model checking is probably the big QA story that tech companies ignore because it requires dramatically more education, especially from QA departments typically seen as "inferior" to SWE.

Video of [0] the Strangeloop talk [1].

[0] https://www.youtube.com/watch?v=4fFDFbi3toc [1] https://thestrangeloop.com/2014/testing-distributed-systems-...

Re: Is something bugging you?

#139
Great read. Great product. I've been an early user of Antithesis. My background is dependability and formal distributed systems.

This thing is magic (or rather, it's indistinguishable from magic ;-)).

If they told me I could test any distributed system without a single line of code change, do things like step-by-step debugging, even rollback time at will, I would not believe it. But Antithesis works as advertised.

It's a game-changer for distributed systems that truly care about dependability.

Re: Is something bugging you?

#140
post #63
post #53

FoundationDB is an impressive achievement, quite possibly the only distributed database out there that lives up to its strict serializability claims (see https://jepsen.io/consistency/models/strict-serializable for a good definition). The way they wrote it is indeed very interesting and a tool that does this for other systems is immediately worth looking at.

Is it that good? I've been tasked to deploy it for sometime and it always bit me in the ass for one reason or another. And I'm not the one who use it so I don't know if it's actually good. For now I much prefer redis.

It's great, but operationally there are lots of gotchas and little guidance.

We got bitten _hard_ in production when we accidentally allowed some of the nodes to get above 90% of the storage used. The whole database collapsed into a state where it could only do a few transactions a second. Then the ops team, thinking they were clever, doubled the size of the cluster in order to give it the resources it needed to get the average utilization down to 45%; this was an unforced error as that pushed the size of the cluster outside the fdb comfort zone (120 nodes) which is itself a problem. The deed was done though and pulling nodes was not possible in this state, so slowly, slooooowly... things got fixed.

We ended up spending an entire weekend slowly, slowly getting things back into a good place. We did not lose data, but basically prod was down for the duration, and we found it necessary to _manually_ evict the full nodes one at a time over the period.

Now, this was a few years ago, and fdb has performed wickedly fast, with utter, total reliability before that and since, and to this day the ops team is butthurt about fdb.

From an engineering perspective, if you aren't using java fdb is pretty not great, since the very limited number of abstraction layers that exist are all java-centric. There are many, many issues with the maximum transaction time thing, the maximum key size and value size and total transaction size issue, the lack of pushdown predicates (e.g., filtered scans can't be done in-place which means that in AWS, they cost a lot in inter-az network charge terms and also are gated by the network performance of your instances), and so on.

What ALL of these have issues have in common is that they bite you late in the game. The storage issue bites you when you're hitting the DB hard in production and have a big data set, the lack of abstractions means that even something as finding leaked junked keys turns out to be impossible unless you were diligent to manually frame all your values so you could identify things as more than just bytes, the transaction time thing is very weird to deal with as you tend to have creeping crud aspects and the lack of libraries that instrument the transactions to give you early warning is an issue, likewise for certain kinds of key-value pairs, there's a creeping size problem - hey, this value is an index of other values; if you're not very careful up front, you _will_ eventually hit either the txn size limit or the key limit. The usual workarounds for those is to do separate transactions - a staging transaction, then essentially a swap operation and then a garbage collection transaction - but that has lots of issues overtime when coupled with application failure.

There are answers to ALL of these, manual ones. For the popular languages other than java - Go, python, maybe Ruby - there _should_ be answers for them, but there aren't. These are very sharp edges. Those java layers are _also_ _not_ _bug_ _free_. So yeah, one has a reliable storage storage layer (a topic that has come up over and over again in the last few years) but it's the layer on top of that where all the bugs are, but now with constraints and factors that are harder to reason about than the usual storage layer.

One might say, hey, SQL has all of these problems too, except no. You can bump into transaction limits, but the limits are vastly higher than fdb and the transaction time sluggishness will identify it long before you run into the "your transaction is rejected, spin retrying something that will _never_ recover" sort of issue that your average developer will eventually encounter in fdb.

That said, I love fdb as a software achievement. I just wish they had finished it. For my current project, I have designed it out. I might be able to avoid all of the sharp edges above at this point, but since we are not a java shop, I also can't rely on all the engineers to even know they exist.

Post reply on HN