Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How SQLite is tested

sqlite.org

Post reply on HN
Print view View original

11–20 of 41 posts

Re: How SQLite is tested

#11

Post by ams6110 »

rpedela wrote: post #9
flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

If you don't actively maintain the tests, then yes you can get a wrong impression of correctness. But when done properly testing helps ensure a certain level of correctness and old bugs don't reappear. A good rule to follow is to create a test(s) for each bug and run the tests on every check-in which is pretty easy with continuous integration. This will mitigate most bugs before they reach production. And the ones th…

This "old bugs reappearing" how does this happen? People make the same logical mistakes in new code? Old buggy code is literally resurrected somehow?

Re: How SQLite is tested

#12

Post by ygra »

ams6110 wrote: post #11
rpedela wrote: post #9

Earlier quoted context omitted.

If you don't actively maintain the tests, then yes you can get a wrong impression of correctness. But when done properly testing helps ensure a certain level of correctness and old bugs don't reappear. A good rule to follow is to create a test(s) for each bug and run the tests on every check-in which is pretty easy with continuous integration. This will mitigate most bugs before they reach production. And the ones th…

This "old bugs reappearing" how does this happen? People make the same logical mistakes in new code? Old buggy code is literally resurrected somehow?

Making changes again at the same place where a bug was fixed previously is a common offender. At least at my workplace we have regressions every now and then (and thanks to no test suite those sometimes make their way to the customer, yay). Basically every time you touch code you have the potential to break something. Even more so when a bugfix was done by adding code, not by changing it. People might accidentally rewrite the fix. If you have no tests that ensure that functionality stays that way it can be easy to break things in the process.

Re: How SQLite is tested

#13

Post by ioddly »

I love the simulated random failures in malloc and I/O, it's something I haven't seen at all in other software. Maybe some people do it, but I think it should be more common.

Re: How SQLite is tested

#14

Post by rpedela »

ams6110 wrote: post #11
rpedela wrote: post #9

Earlier quoted context omitted.

If you don't actively maintain the tests, then yes you can get a wrong impression of correctness. But when done properly testing helps ensure a certain level of correctness and old bugs don't reappear. A good rule to follow is to create a test(s) for each bug and run the tests on every check-in which is pretty easy with continuous integration. This will mitigate most bugs before they reach production. And the ones th…

This "old bugs reappearing" how does this happen? People make the same logical mistakes in new code? Old buggy code is literally resurrected somehow?

Yeah I should have probably used the technical term: regression. It is very easy to do. Sometimes it happens when code is rewritten/refactored. Sometimes a new feature utilizes a buggy code path that was not previously used/tested. Etc.

Re: How SQLite is tested

#15

Post by cwzwarich »

flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

There are no automatically provable invariants that would cover SQLite's functionality. How does a tool automatically prove that a query compiler is correct, for example? I don't see an easy way, barring writing SQLite in Coq.

Re: How SQLite is tested

#16

Post by cwzwarich »

ams6110 wrote: post #11
rpedela wrote: post #9

Earlier quoted context omitted.

If you don't actively maintain the tests, then yes you can get a wrong impression of correctness. But when done properly testing helps ensure a certain level of correctness and old bugs don't reappear. A good rule to follow is to create a test(s) for each bug and run the tests on every check-in which is pretty easy with continuous integration. This will mitigate most bugs before they reach production. And the ones th…

This "old bugs reappearing" how does this happen? People make the same logical mistakes in new code? Old buggy code is literally resurrected somehow?

It's pretty easy to make the same mistake that originally caused a bug when rewriting code.

Re: How SQLite is tested

#17

Post by wmt »

ioddly wrote: post #13

I love the simulated random failures in malloc and I/O, it's something I haven't seen at all in other software. Maybe some people do it, but I think it should be more common.

Simulating malloc failures is also very easy to do in Windows with the MS Application Verifier.

Re: How SQLite is tested

#18

Post by gred »

My takeaway: if you want (meaningful) 100% code coverage, you might have to write 1,000 times more test code than actual production code.

Re: How SQLite is tested

#20

Post by qznc »

kkowalczyk wrote: post #8
flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

1. 3.8.0.2 fixed a logical error in the code. High-level language won't help you if you invert a logical condition or commit some more subtle logical mistake. 2. Our software also needs to be fast. No one needs a slow database engine. There is no high-level language that beats C on tasks like writing sqlite. 3. Finally, show me that unicorn high-level language with strong, automatically provable invariants and one pi…

Contenders would be ATS and Nimrod. Neither fulfills the "one piece of widely used, reliable software that was written in it" requirement, though.
Post reply on HN