Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How SQLite is tested

sqlite.org

Post reply on HN
Print view View original

21–30 of 41 posts

Re: How SQLite is tested

#21

Post by smtddr »

flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

So, how would you test database software?

This reminds me of an old Google(?) interview question.

"How do you test a Calculator after dropping it?" http://www.youtube.com/watch?v=ILkT_HV9DVU&t=46m50s

For all the possible things that could go wrong in a fancy graphing calculator, what should you do to ensure it works properly?

Re: How SQLite is tested

#22

Post by derefr »

kkowalczyk wrote: post #8
flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

1. 3.8.0.2 fixed a logical error in the code. High-level language won't help you if you invert a logical condition or commit some more subtle logical mistake. 2. Our software also needs to be fast. No one needs a slow database engine. There is no high-level language that beats C on tasks like writing sqlite. 3. Finally, show me that unicorn high-level language with strong, automatically provable invariants and one pi…

> 3. Finally, show me that unicorn high-level language with strong, automatically provable invariants and one piece of widely used, reliable software that was written in it.

Not implying that this is true now, but this is the specific goal of Rust, with Servo.

Re: How SQLite is tested

#23

Post by mkesper »

Wow. Impressive!

But what bugs me about sqlite is

a) suppressing errors giving surprising results (hint: try opening a misspelt database file)

b) "syntax error near 'as'": almost impossible to find errors in longer sql statements (are other engines better, giving line numbers?)

c) no full outer join https://www.sqlite.org/omitted.html which can be worked around by writing about the double code, thus giving chance for twice as much errors

I really love the simplicity of using it, though!

Re: How SQLite is tested

#24

Post by boyter »

gred wrote: post #18

My takeaway: if you want (meaningful) 100% code coverage, you might have to write 1,000 times more test code than actual production code.

That's been my experience in practice. For a critical piece of integration code I had about 100 times more code in the tests then in the function I was testing.

Re: How SQLite is tested

#25

Post by smrtinsert »

flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

Typical ivory tower hn blather honestly. I would love to hear a real solution instead of self aggrandizement and vague ideas.

Re: How SQLite is tested

#26

Post by cwzwarich »

derefr wrote: post #22
kkowalczyk wrote: post #8

Earlier quoted context omitted.

1. 3.8.0.2 fixed a logical error in the code. High-level language won't help you if you invert a logical condition or commit some more subtle logical mistake. 2. Our software also needs to be fast. No one needs a slow database engine. There is no high-level language that beats C on tasks like writing sqlite. 3. Finally, show me that unicorn high-level language with strong, automatically provable invariants and one pi…

> 3. Finally, show me that unicorn high-level language with strong, automatically provable invariants and one piece of widely used, reliable software that was written in it. Not implying that this is true now , but this is the specific goal of Rust, with Servo.

With its current design trajectory, Rust won't be able to prove interesting high-level invariants. It's going to give you the kinds of things you can prove with traditional ML-style type systems and affine types. The combination of the two lets you encode some interesting things, e.g. typestates, but they won't let you express something like "this browser engine correctly implements CSS".

Re: How SQLite is tested

#27

Post by jt884 »

flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

It's amazing that, even with a long history of high quality releases and an incredible dedication to QA, some pedantic hipster can come along and, without flinching, issue a behold-while-I-educate-thee piece on what SQLite is "doing wrong".

The kind of experienced required to develop something like SQLite and then test it to this degree can probably be counted in decades.

Who the f*ck are you? Some JavaScript developer on the ugly end of the Dunning-Kruger effect?

Re: How SQLite is tested

#28

Post by jt884 »

smrtinsert wrote: post #25
flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

Typical ivory tower hn blather honestly. I would love to hear a real solution instead of self aggrandizement and vague ideas.

My thoughts exactly.

Re: How SQLite is tested

#29

Post by rodgerd »

ams6110 wrote: post #11
rpedela wrote: post #9

Earlier quoted context omitted.

If you don't actively maintain the tests, then yes you can get a wrong impression of correctness. But when done properly testing helps ensure a certain level of correctness and old bugs don't reappear. A good rule to follow is to create a test(s) for each bug and run the tests on every check-in which is pretty easy with continuous integration. This will mitigate most bugs before they reach production. And the ones th…

This "old bugs reappearing" how does this happen? People make the same logical mistakes in new code? Old buggy code is literally resurrected somehow?

From my days as a release/build manager, the most common cause of this was people merging code from branches where fixes had been overlooked and similar horrors. The second highest would be people fixing something that "looked wrong" but was actually there to solve a problem.

Re: How SQLite is tested

#30

Post by happywolf »

SQLite can get so many tests partly due to the relatively stable requirements. In an environment with fast-changing requirements, it is not likely to have so many test cases for regression tests. In this type of environment, what I observed is either 1) the team has no time to update the initial test cases and causing a large batch of tests not being run 2) team spends significant time to fix and debug tests, slowing down the progress in adding new cases
Post reply on HN