Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is something bugging you?

antithesis.com

Post reply on HN
Print view View original

241–250 of 438 posts

Re: Is something bugging you?

#241

Post by binarymax »

nlavezzo wrote: post #187
binarymax wrote: post #171

I got really excited about this, and I spent a little time looking through the documentation, but I can't figure out how this is different than randomizing unit tests? It seems if I have a unit test suite already, then that's 99% of the work? Am I misunderstanding? I am drawing my conclusions from reading the Getting Started series of the docs, especially the Workloads section: https://antithesis.com/docs/getting_sta…

Antithesis here - curious what part of the Getting Started doc gave you that impression? If you take a look at our How Antithesis Works page, it might help answer you question as to how Antithesis is different from just bundling your unit tests. https://antithesis.com/docs/introduction/how_antithesis_work... In short though, unit tests can help to inform a workload, but we don't require them. We autonomously explore…

Thanks for the response. The linked introduction does help. The workload page does give me that impression (and based on upvotes of my post it does to others as well)...so perhaps disambiguating that the void test*() examples on the workloads page are not unit tests might help!

Congrats on the launch and I'll consider using it for some of my projects.

Re: Is something bugging you?

#243

Post by spinningD20 »

btrettel wrote: post #26
Rygian wrote: post #4

The writing is really enjoyable. > Programming in this state is like living life surrounded by a force field that protects you from all harm. [...] We deleted all of our dependencies (including Zookeeper) because they had bugs, and wrote our own Paxos implementation in very little time and it _had no bugs_. Being able to make that statement and back it by evidence must be indeed a cool thing.

The earliest that I've seen the attitude that one should eliminate dependencies because they have more bugs than internally written code was this book from 1995: https://store.doverpublications.com/products/9780486152936 pp. 65-66: > The longer I have computed, the less I seem to use Numerical Software Packages. In an ideal world this would be crazy; maybe it is even a little bit crazy today. But I've been bitten too…

That tracks well (both the quotes and your thoughts).

One example that comes to mind where I want to roll my own thing (and am in the process of doing so) is replacing our ci/cd usage of jenkins that is solely for running qa automation tests against PR's on github. Jenkins does way way more than we need. We just need github PR interaction/webhook, secure credentials management, and spawning ecs tasks on aws...

Every time I force myself to update our jenkins instance, I buckle up because there is probably some random plugin, or jenkins agent thing, or ... SOMETHING that will break and require me to spend time tracking down what broke and why. 100% surface area for issues, whilst we use <5% of what Jenkins actually provides.

Re: Is something bugging you?

#244

Post by ransom1538 »

didgetmaster wrote: post #113
jimbokun wrote: post #35

> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size. I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out. But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to…

It seems like the industry would get a lot more 10x behavior if it was recognized and rewarded more often than it currently does. Too often, management will focus more on the guy who works 12 hour days to accomplish 8 hours of real work than the guy who gets the same thing accomplished in an 8 hour day. Also, deviations from 'normal' are frowned upon. Taking time to improve the process isn't built into the schedule;…

Honestly? You work at a place a manager hasn't heard "impact" yet? I thought managers at this point just walk around the office saying "impact".

Re: Is something bugging you?

#245

Post by pfdietz »

mamidon wrote: post #199

I can see how determinism can be achieved (not easy, but possible), and I can see how describing a few important system invariants can match 100's or 1000's of hand rolled tests, but I'm having a hard time understanding how it's possible to intelligently explore the inputs to generate. e.g. if I wrote a compiler, how would Antithesis generate mostly valid source code for it? Simply fuzzing utf8 inputs wouldn't get ve…

I don't know how they'd do compiler testing, but I know how I do it (testing Common Lisp), and can talk about that if you're interested.

But it would be cool to hear how they'd do it.

Re: Is something bugging you?

#246

Post by FridgeSeal »

dap wrote: post #192
nlavezzo wrote: post #165

Earlier quoted context omitted.

I think the reference to "all the bugs" here is basically that our insanely brutal deterministic testing system was not finding any more bugs after 100's of thousands of runs. Can't prove a negative obviously, but the fact that we'd gotten to that "all green" status gave us a ton of confidence to push forward in feature development, believing we were building on something solid - which, time has shown we were.

Thanks -- that's very clarifying! But isn't this circular? The lack of bugs is used as evidence of the effectiveness of the testing approach, but the testing approach is validated by...not finding any more bugs in the software?

Yeah but if your software is running in an environment that controls for a lot of non-determinism and can simulate various kinds of failures and degradations at varying rates, and do it all in accelerated time and your software is still working correctly; I think it’d be somewhat reasonable to assert that maybe the testing setup has done a pretty good job.

Re: Is something bugging you?

#247

Post by eschneider »

mettamage wrote: post #173
jimbokun wrote: post #35

> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size. I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out. But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to…

When I was in college, I've met a few people that coded _a lot_ faster than me. Typically, they started since they were 12 instead of 21 (like me). That's how 10x engineers exist, by the time they are 30, they have roughly 20 years of programming experience behind their belt instead of 10. Also, their professional experience is much greater. Sure, their initial jobs at 15 are the occassional weird gig for the uncle/a…

I'm not even sure that coding _much_ faster than necessary is even required to give a 3-5x multiple on "average", let alone "worst case" developers. Some of the biggest productivity wins can be had by being able to look at requirements, knowing what's right or wrong about them, and getting everyone on the same page so the thing only needs to be made once. Being good at test and debug so problems are identified and fixed _early_ are also big wins. Lots of that is just having the experience to recognize what sort of problem you're dealing with very quickly.

Being a programming prodigy is nice, but I don't think you even really need that.

Re: Is something bugging you?

#248

Post by llm_trw »

coldtea wrote: post #46
llm_trw wrote: post #12

Earlier quoted context omitted.

I have proved my code has no bugs according to the spec. I do not make the claim my spec has no bugs.

With formal proof systems, you can also claim that for your spec.

No you can't.

You can claim that your spec doesn't violate some invariants in a finite number of steps, you can't claim that the spec contains all the invariants the real system must have and that it doesn't violate them in number of steps + 1.

Re: Is something bugging you?

#249

Post by pfdietz »

iamnotsure wrote: post #180

"I love me a powerful type system, but it’s not the same as actually running your software in thousands and thousands of crazy situations you’d never dreamed of." Would not trust. Formal software verification is badly needed. Running thousands of tests means almost nothing in software world. Don't fool beginners with your test hero stories.

Formal verification require a formal statement of what the software is supposed to do.

But if you have that, you have a recipe for doing property based testing: generate inputs that satisfy the conditions specified in this formal description, then verify that the behavior satisfies the specification also.

And then run for millions and millions of inputs.

Is it really going to be worth proving the program correct, when you could just run an endless series of tests? Especially if the verifier takes forever solving NP hard theorem proving problems at every check in. Use that compute time to just run the tests.

Post reply on HN