Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing is better than data structures and algorithms

nedbatchelder.com

Post reply on HN
Print view View original

151–160 of 178 posts

Re: Testing is better than data structures and algorithms

#151

Post by FridgeSeal »

yakshaving_jgt wrote: post #130
themafia wrote: post #65

Earlier quoted context omitted.

Macro Pierre White says "perfection is lots of little things done well." Which is something I've always agreed with, so, I never understand articles that seek to eschew an important part of releasing software because they believe their approach elsewhere is enough to overcome these intentionally suboptimal choices.

Almost all of professional software should be intentionally suboptimal. This is what we mean when we say that premature optimisation is the root of all evil.

We ought to ban people saying that quote, due to the way it has been abused to avoid considering performance _at all_.

“Intentionally suboptimal” is also a strange way of phrasing it, as it makes it sound a bit like you’re intentionally building something bad, as opposed to “only as good as it needs to be”.

Re: Testing is better than data structures and algorithms

#152

Post by rossant »

JoeAltmaier wrote: post #146
rossant wrote: post #127

Earlier quoted context omitted.

How much work was it to go from 1 hour to 1 week? How many issues have you discovered, what were they? Genuinely interested.

It took a big fraction of our energy to get to each new stage. Always it was something new and unexpected. It was quite a while ago, but lets see what I remember. Some good fraction were mismatches between the app (bot) state and the server state. A bot would be expecting a message and stall. The server thought it had said enough. The app side used a lot of libraries, which it turns out are never as robust as adverti…

Wow, really interesting write-up, thank you! It really proves the immense value of this kind of automated, realistic stress test.

Re: Testing is better than data structures and algorithms

#153

Post by quotemstr »

moron4hire wrote: post #144
quotemstr wrote: post #104

Earlier quoted context omitted.

> “Well, I’m going to do Sudoku and I’m going to have this class and first thing I’m going to do is write a bunch of tests.” But then he never got anywhere There's a blog post I read once and that I've since been unable to locate anywhere, even with AI deep research. It was a blow-by-blow record of an attempt to build a simple game --- checkers, maybe? I can't recall --- using pure and dogmatic test driven developmen…

Norvig mentions it in the article linked in the post to which you are replying. The game was Sudoku. The person was Ron Jeffries. https://ronjeffries.com/articles/-z022/01121/sudoku-again/

Thanks!

Re: Testing is better than data structures and algorithms

#154

Post by ChrisMarshallNY »

general1465 wrote: post #122
ChrisMarshallNY wrote: post #55

Earlier quoted context omitted.

I have not done it, myself, but I think that Xcode, for Apple stuff, can parallelize tests, across multiple machines (maybe VMs?). I would assume that Microsoft systems could do the same.

A lot of tests are sharing one resource (USB device) which can't be accessed in parallel. So that's my constraint which I need to live with and the main reason why I can't parallelize or offload testing to cloud. Otherwise yes, you can run tests in parallel in vstest. That's completely possible.

Oh yeah. I did a lot of hardware stuff.

Quite familiar with the drill. Carry on...

Re: Testing is better than data structures and algorithms

#155

Post by yakshaving_jgt »

FridgeSeal wrote: post #151
yakshaving_jgt wrote: post #130

Earlier quoted context omitted.

Almost all of professional software should be intentionally suboptimal. This is what we mean when we say that premature optimisation is the root of all evil.

We ought to ban people saying that quote, due to the way it has been abused to avoid considering performance _at all_. “Intentionally suboptimal” is also a strange way of phrasing it, as it makes it sound a bit like you’re intentionally building something bad, as opposed to “only as good as it needs to be”.

In general I avoid considering performance at all. Instead, I focus on adding testing and instrumentation. When the telemetry tells me I have a performance problem, then I can solve that part while being confident that the performance improvement doesn't change the functionality because I first invested in tests.

Re: Testing is better than data structures and algorithms

#156

Post by pfdietz »

fuzztester wrote: post #121
pfdietz wrote: post #112

Earlier quoted context omitted.

As I recall this was a book that included the orthodoxy at the time that random testing was the worst kind of testing, to be avoided if possible. That turned out to be bullshit. Today, with computers many orders of magnitude faster, using randomly generated tests is a very cost effective away of testing, compared to carefully handcrafted tests. Use extremely cheap machine cycles to save increasingly expensive human t…

Interesting. Don't remember that from the book, but then, I read it long ago. I agree that random testing can be useful. For example, one kind of fuzzing is using tons of randomly generated test data against a program to try to find unexpected bugs. But I think both kinds have their place. Also, I think the author might have mean that random testing is bad when used with a small amount of test data, in which case I'd…

Here is the quote (from the 3rd ed., page 41):

"In general, the least effective methodology of all is random-input testing—the process of testing a program by selecting, at random, some subset of all possible input values. In terms of the likelihood of detecting the most errors, a randomly selected collection of test cases has little chance of being an optimal, or even close to optimal, subset. Therefore, in this chapter, we want to develop a set of thought processes that enable you to select test data more intelligently."

You can immediately see the problem here. It's optimizing for number of tests run, not for the overall cost of creating and running the tests. It's an attitude suited to when running a program was an expensive thing using precious resources. It was very wrong in 2012 when this edition came out and even more wrong today.

Re: Testing is better than data structures and algorithms

#158

Post by pixl97 »

wubrr wrote: post #62
JackSlateur wrote: post #59

Earlier quoted context omitted.

Testing concurrency is extremely hard For instance, get sql queries; You ran them, and you have no issue; Is your code sane ? Or is it because one query ran 10ms earlier and, thus, you avoided the issue ? I truly wonder if there is real world tests around this; I bet there is only algorithm and fuzzing;

> Testing concurrency is extremely hard Writing a non-trivial concurrent system based on your understanding of the 'algorithm' , without relying on testing is much harder. > I truly wonder if there is real world tests around this Of course there are. There are many tools, methods, and test suites out there for concurrency testing, for almost any major language out there. Of course, understanding your algorithm, and t…

I mean if you're talking SQL on a large real database vs a small test db you can get some pretty big differences in performance and behavior. Of course query planning is something that should be monitored as an app is deployed and used, but testing never does seem to catch the edge cases.

Re: Testing is better than data structures and algorithms

#159

Post by EVa5I7bHFq9mnYK »

IvyMike wrote: post #66

Always gonna have to side with Peter Norvig on this one: https://pindancing.blogspot.com/2009/09/sudoku-in-coders-at-... > They said, “Look at the contrast—here’s Norvig’s Sudoku thing and then there’s this other guy, whose name I’ve forgotten, one of these test-driven design gurus. He starts off and he says, “Well, I’m going to do Sudoku and I’m going to have this class and first thing I’m going to do is write a bun…

If you write all the tests, I'm sure the LLM can figure out the implementation.

Re: Testing is better than data structures and algorithms

#160

Post by nijave »

sarchertech wrote: post #68
wubrr wrote: post #62

Earlier quoted context omitted.

> Testing concurrency is extremely hard Writing a non-trivial concurrent system based on your understanding of the 'algorithm' , without relying on testing is much harder. > I truly wonder if there is real world tests around this Of course there are. There are many tools, methods, and test suites out there for concurrency testing, for almost any major language out there. Of course, understanding your algorithm, and t…

I’ve never worked somewhere (in 20 years from big tech companies to small startups) that was generally and reliably testing for concurrency bugs. And I’ve seen dozens of bugs caused by people assuming that transactions (with the default isolation level) protect against race conditions.

Not sure about other languages but I believe the stock test tooling for `go` generates a random sorting seed and has a flag to run tests concurrently. You can also manually pass the seed to simulate a certain ordering.

While not perfect, our e2e tests caught a couple bugs running them with concurrency.

Post reply on HN