Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is something bugging you?

antithesis.com

Post reply on HN
Print view View original

211–220 of 438 posts

Re: Is something bugging you?

#211

Post by eatonphil »

wwilson wrote: post #103
BoppreH wrote: post #97

Three thoughts: 1. It's a brilliant idea that came at the right time. It feels like people are finally losing patience with flaky software, see developer sentiment on: fuzzers, static typing, memory safety, standardized protocols, containers, etc. 2. It's meant to be niche. $2 per hour per CPU (or $7000 per year per CPU if reserved), no free tier for hobby or FOSS, and the only way to try/buy is to contact them. Ouch…

Thanks for your kind words! As I mention in this comment ( https://news.ycombinator.com/item?id=39358526 ) we are planning to have pricing suitable for small teams, and perhaps even a free tier for FOSS, in the future. Disclosure: Antithesis co-founder.

There a few FOSS projects I'd love to set this up for if you ever get to the free tier. :)

Re: Is something bugging you?

#212

Post by _dain_ »

randomdata wrote: post #166
wwilson wrote: post #164

Earlier quoted context omitted.

Nope. We’re emulating a deterministic computer, so your software can’t act nondeterministically if it tries.

Right, by emulating a deterministic computer you can ensure that the inputs to the software are always deterministic – something traditional computing environments are unable to offer for various reasons. However, if we pretend that software was somehow able to be non-deterministic, it would be able to evade your deterministic computer. But since software is always deterministic, you just have to guarantee determinis…

[I work at Antithesis]

>But since software is always deterministic, you just have to guarantee determinism in the inputs.

This is technically correct, but that's a very load-bearing "just". A lot of things would have to count as inputs. Think about execution time, for example. CPUs don't execute at the same speed all the time because of automatic throttling. Network packets have different flight times. Threads and processes get scheduled a little differently. In distributed/concurrent systems, all this matters. If you run the same workload twice, observable events will happen at different times and in different orders because of tiny deviations in initial conditions.

So yes, if you consider the time it takes to run every single machine instruction as an "input", then software is deterministic given the same inputs. But in the real world that's not actionable. Even if you had all those inputs, how are you going to pass them in? For all intents and purposes most software execution is non-deterministic.

The Antithesis simulation is deterministic in this way though. It is in charge of how long everything takes in "simulated time", right down to the running times of individual CPU instructions. Everything observable from within the simulation happens the exact same way, every time. You can compare a memory dump at the same (simulated) instant across two different runs and they will be bit-for-bit identical.

Re: Is something bugging you?

#213

Post by amw-zero »

spinningD20 wrote: post #174
amw-zero wrote: post #107

Earlier quoted context omitted.

I do think that it was a mistake to use the word "all" and imply that there are absolutely no bugs in FoundationDB. However, FoundationDB is truly known as having advanced the state of the art for testing practices: https://apple.github.io/foundationdb/testing.html . So in normal cases this would reek of someone being arrogant / overconfident, but here they really have gotten very close to zero bugs.

The other issue I would point out is that building a database, while impressive with their quality, is still fundamentally different than an application or set of applications like a larger SaaS offering would involve (api, web, mobile, etc). Like the difference between API and UI test strategies, where API has much more clearly defined and standardized inputs and outputs. To be clear, I am not saying that you can't…

There's a lot of assertions that I throw into business applications that would be very useful to test in this way. So I don't think this only applies to testing databases.

Also, when properties are difficult to think of, that often means that a model of the behavior might be more appropriate to test against, e.g. https://concerningquality.com/model-based-testing/. It would take a bit of design work to get this to play nicely with the Antithesis approach, but it's definitely doable.

Re: Is something bugging you?

#215

Post by rkangel »

crashabr wrote: post #185
rkangel wrote: post #115

Earlier quoted context omitted.

Systems Engineering has terminology for this distinction. Verification is "does this thing do what I asked it to do". Validation is "did I ask it to do the right thing".

Tangentially related, but I've recently started distinguishing verification and validation in my data cleaning work: verification refers to "is this dataset clean?" or the more precise "does this dataset confirm my assumptions about what a what a correct dataset should be given its focus" validation refers to "can it answer my questions?" or the more rigorous "can I test my hypotheses against this dataset?" So I find…

They're fairly standard terms from "old style" project management - they show up in the usual V Model of Waterfall vein.

E.g. see Wikipedia: https://en.m.wikipedia.org/wiki/Verification_and_validation

Re: Is something bugging you?

#216

Post by robocat »

JimDabell wrote: post #191
SkyBelow wrote: post #177

Earlier quoted context omitted.

Even if this was the origin of the term, it still doesn't make sense because the best engineers can solve problems the worst would never be able to do so. The difference between the best and worst is much more than 10x the worst. Maybe the worst who meets certain minimums at a company, but then the best would also be limited by those willing to work for what the company pays, and I hypothesis that the minimums of the…

It sounds like you disagree with the concept of a 10x engineer then. In which case you should avoid using the term, rather than making up a new definition.

Concepts and words change meaning and sometimes we all need to accept that the popular meaning is not the definition we use.

This is especially common when dealing with historical or academic definitions versus common modern usage. "Evolution" particularly annoys me.

You should avoid using the term, rather than using a definition at odds with common usage. Your usage is confusing - and that is why you are getting push-back.

The definition you have given is nonsensical - it can't be consistent over time or between companies because it depends on finding a minimum in a group. And a value that is strongly dependent on the worst developer is useless because it mostly measures how bad the worst developer is - it doesn't say anything about how good the best developer is.

Re: Is something bugging you?

#217

Post by sfink »

iamnotsure wrote: post #180

"I love me a powerful type system, but it’s not the same as actually running your software in thousands and thousands of crazy situations you’d never dreamed of." Would not trust. Formal software verification is badly needed. Running thousands of tests means almost nothing in software world. Don't fool beginners with your test hero stories.

That'll work great for your Distributed QSort Incorporated startup, where the only product is a sorting algorithm.

Formal software verification is very useful. But what can be usefully formalized is rather limited, and what can be formalized correctly in practice is even more limited. That means you need to restrict your scope to something sane and useful. As a result, in the real world running thousands of tests is practically useful. (Well, it depends on what those tests are; it's easy to write 1000s of tests that either test the same thing, or only test the things that will pass and not the things that would fail.) They are especially useful if running in a mode where the unexpected happens often, as it sounds like this system can do. (It's reminiscent of rr's chaos mode -- https://rr-project.org/ linking to https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo... )

Re: Is something bugging you?

#218

Post by zoogeny »

What is described in this post is the gold standard of software reliability testing. A world where all critical and foundational systems are tested to this level would be a massive step forward for technology in general.

I'm skeptical of their claims but inspired by the vision. Even taking into account my skepticism, I would prefer to deploy systems tested to this standard over alternatives.

Re: Is something bugging you?

#219

Post by mrkeen »

Communitivity wrote: post #106

There are situations where no bugs is an important requirement, if it means no bugs that cause a noticeable failure. Things such as planes, submarines, nuclear reactors. For those there is provably correct code. That takes a long time to write, and I mean a really long time. Applying that to all software doesn't make sense from a commercial perspective. There are areas where improvements can have a big impact though,…

> I don't see no bugs in a distributed database as important enough to delay shipping for 5 years

The marketplace has enough distributed databases with bugs. There's a nice catalogue of them at jepsen.io.

> For me it's much more important to ship something with no critical bugs early, get user feedback, iterate, then rinse and repeat continually.

* You can't really choose which bugs are critical if you're selling a database. A lost write is as critical as the customer deems it is.

* You're not limited to your own users' feedback. There's plenty of users out there who disapprove of a buggy database, so you can probably take their views onboard before release.

Re: Is something bugging you?

#220

Post by mrkeen »

Gehinnn wrote: post #202

Reading this article, I want the same now for js code that involves web-workers... How can I write code that involves a webworker in a way that I can simulate every possible CPU scheduling between the main thread in the webworker (given they communicate via post message and no shared array buffer)? Is it possible to write such brute force test in pure JS, without having to simulate the entire computer?

Use TLA+/PlusCal for this. It's what it's there for.
Post reply on HN