Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is something bugging you?

antithesis.com

Post reply on HN
Print view View original

361–370 of 438 posts

Re: Is something bugging you?

#361

Post by shuntress »

nlavezzo wrote: post #342
shuntress wrote: post #300

> a platform that takes your software and hunts for bugs in it Ok but, what actually IS it? It seems like it is a cloud service that will run integration tests. I have to figure out how to deploy to this special environment and I still have to write those integration tests using special libraries. But even after all that integration refactoring, how is this supposed to help me find actual bugs that I wouldn't already…

I'd suggest taking a dive into the docs - there is quite a lot there that should address some of these questions. That said, Antithesis doesn't require you to write manual tests, integration or otherwise. It requires your software system to be packaged in containers, which is fairly straightforward, and then requires a workload to be written which will emulate the normal functioning of the software system. So for exa…

"Workloads" seem to be effectively equivalent to integration tests.

I don't mean to poke holes but I'm having trouble seeing the value add here.

If I have to deploy to some new environment anyways and I have to tailor the "Workloads" anyways why would I pay extra for vendor lock-in?

The type of devious bug this is promising to find would be something like:

"The DB silently drops timezone from Dates because of the column type. This results in unexpected data being returned for users in different timezones from the server"

I just don't see how repeatably calling the API with an expanding set of random inputs helps find something like that.

Re: Is something bugging you?

#362

Post by dclowd9901 »

jimbokun wrote: post #35

> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size. I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out. But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to…

Yep. Often I find our most accelerative work is stuff that makes testing changes easy (a very simple to bootstrap staging environment) or creates a lot of guarantees (typescript).

Re: Is something bugging you?

#364

Post by randomdata »

loeg wrote: post #346
randomdata wrote: post #146

Earlier quoted context omitted.

I'm not sure your math works. What we do know is that the worst engineers provide negative productivity. If 1x is the worst engineer, then let's for the sake of discussion denote x as -1 in order for the product to be negative. Except that means the 10x engineer provides -10 productivity, actually making them the worst engineer. Therein lies a conflict. What we also know is that best engineer has positive productivit…

Engineers with negative productivity are vanishingly rare, soon to be terminated, and reasonable to exclude for the purpose of the comparison.

Okay, but it still doesn't work. The world's worst engineer who somehow managed to successfully contribute one line of code to something like GPT is way more productive than a great engineer who designed from top to bottom the best laid software ever conceived but was thrown away before seeing the light of day because the business changed direction.

Of course, that doesn't actually matter as the original study found a 10x difference in measuring performance, not productivity. There is nothing out there to suggest that some developers are 10x more productive outside of those who mixed up their p words. We're not actually talking about productivity; that was just a mistake. If one were to study productivity, I expect they would find that some engineers are many orders of magnitude more productive than the least productive engineers.

Re: Is something bugging you?

#365

Post by svat »

mjevans wrote: post #328
ianmcgowan wrote: post #268

Earlier quoted context omitted.

Sounds like "Knuth vs McIlroy", which has been discussed on hn and elsewhere before, and the general take is that it was somewhat unfair to Knuth. [1] https://homepages.cwi.nl/~storm/teaching/reader/BentleyEtAl8... [2] https://www.google.com/search?q=knuth+vs+mcilroy

This is the competition I was thinking of. I must have read it in a dead-image PDF version some other time on HN. This paper isn't the one I recall but the solution is exactly the sort I vaguely recalled. I'm trying to copy-in the program as it might have existed, with some obvious updates to work in today's shells ... #!/bin/sh tr -cs A-Za-z ' ' "${2:-/dev/stdin}" |\ tr A-Z a-z |\ sort |\ uniq -c |\ sort -rn |\ sed…

Just to be clear, it was not a competition. For more, please follow the links from some of the previous HN discussions, e.g. https://news.ycombinator.com/item?id=31301777.

[For those who may not follow all the links: Bentley asked Knuth to write a program in Pascal (WEB) to illustrate literate programming—i.e. explaining a long complicated program—and so Knuth wrote a beautiful program with a custom data structure (hash-packed tries). Bentley then asked McIlroy to review the program. In the second half of the review, McIlroy (the inventor of Unix pipes) questioned the problem itself (the idea of writing a program for scratch), and used the opportunity to evangelize Unix and Unix pipes (at the time not widely known or available).]

Re: Is something bugging you?

#366

Post by loeg »

randomdata wrote: post #364
loeg wrote: post #346

Earlier quoted context omitted.

Engineers with negative productivity are vanishingly rare, soon to be terminated, and reasonable to exclude for the purpose of the comparison.

Okay, but it still doesn't work. The world's worst engineer who somehow managed to successfully contribute one line of code to something like GPT is way more productive than a great engineer who designed from top to bottom the best laid software ever conceived but was thrown away before seeing the light of day because the business changed direction. Of course, that doesn't actually matter as the original study found…

> The world's worst engineer who somehow managed to successfully contribute one line of code to something like GPT is way more productive than a great engineer who designed from top to bottom the best laid software ever conceived but was thrown away before seeing the light of day because the business changed direction.

I reject that assertion.

> performance, not productivity

You keep saying this like it's a slam dunk refutation, but performance and productivity are highly related.

Re: Is something bugging you?

#367

Post by randomdata »

loeg wrote: post #366
randomdata wrote: post #364

Earlier quoted context omitted.

Okay, but it still doesn't work. The world's worst engineer who somehow managed to successfully contribute one line of code to something like GPT is way more productive than a great engineer who designed from top to bottom the best laid software ever conceived but was thrown away before seeing the light of day because the business changed direction. Of course, that doesn't actually matter as the original study found…

> The world's worst engineer who somehow managed to successfully contribute one line of code to something like GPT is way more productive than a great engineer who designed from top to bottom the best laid software ever conceived but was thrown away before seeing the light of day because the business changed direction. I reject that assertion. > performance, not productivity You keep saying this like it's a slam dunk…

> I reject that assertion.

Because you don't believe the worst developer contributed anything to GPT? Sure, in reality that's no doubt true, but it was only ever meant to be illustrative.

> but performance and productivity are highly related.

Not in any meaningful way. The study found that the fastest developer can perform a set of defined tasks in a 10th of the time of the slowest developer. That is what the 10x developer refers to. But being fast doesn't mean being productive.

Come to my backyard and we will each dig a hole of equal size. Let's assume that you can dig the hole in a 10th of the time I can – that you are the 10x hole digger. But, no matter how fast you are, neither of us will be productive.

Re: Is something bugging you?

#368

Post by ManuelKiessling »

khazhoux wrote: post #338
ManuelKiessling wrote: post #253

Earlier quoted context omitted.

What makes a car go fast? The brakes: https://manuel.kiessling.net/2011/04/07/why-developing-witho...

Literally the opposite of what makes a car go fast :-)

Is it?

How fast would you drive a car if I gave you the keys and told you „everything works perfectly fine, however the brakes have been removed“?

Re: Is something bugging you?

#369

Post by maronato »

shuntress wrote: post #300

> a platform that takes your software and hunts for bugs in it Ok but, what actually IS it? It seems like it is a cloud service that will run integration tests. I have to figure out how to deploy to this special environment and I still have to write those integration tests using special libraries. But even after all that integration refactoring, how is this supposed to help me find actual bugs that I wouldn't already…

Reading their docs it seems you got it pretty much correctly. You write integration tests (aka “workloads”) and then they run it under different scenarios.

This means they use their hypervisor to change random seeds, make http requests fail or take too long, break connections between servers, change the order of server responses, and all sorts of wild things you don’t usually control, but that happen in the real world. Then they compare the expected workload responses and figure out which conditions break your systems.

That’s why they sell yearly contracts - you’re supposed to pay them to keep your workloads running continuously all year to try all sorts of different combinations of failures.

Re: Is something bugging you?

#370

Post by dclowd9901 »

spinningD20 wrote: post #174
amw-zero wrote: post #107

Earlier quoted context omitted.

I do think that it was a mistake to use the word "all" and imply that there are absolutely no bugs in FoundationDB. However, FoundationDB is truly known as having advanced the state of the art for testing practices: https://apple.github.io/foundationdb/testing.html . So in normal cases this would reek of someone being arrogant / overconfident, but here they really have gotten very close to zero bugs.

The other issue I would point out is that building a database, while impressive with their quality, is still fundamentally different than an application or set of applications like a larger SaaS offering would involve (api, web, mobile, etc). Like the difference between API and UI test strategies, where API has much more clearly defined and standardized inputs and outputs. To be clear, I am not saying that you can't…

I’d go so far as to say it is actually impossible to do UI testing in some kind of web based product unless it came from the browser makers themselves.

I’d settle for decent heap debugging.

Post reply on HN