Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How SQLite is tested

sqlite.org

Post reply on HN
Print view View original

31–40 of 41 posts

Re: How SQLite is tested

#31

Post by emn13 »

advisedwang wrote: post #4
flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

But is possible to create very serious bugs even with high level languages, and proving software in a way to provide good coverage is extremely hard.

Yeah - it's ludicrous to suggest provable constraints will ever really achieve anything close to the coverage these tests have.

Having said that, if you have the choice, it's still unwise to spend huge amounts of time writing+maintaining tests but then balk at the relatively short amount needed to use a strong type system. It won't catch lots of things, but it will catch quite a few. For the limited concepts it can express, it can ensure correctness more rigorously, easily and quickly than any testing can. Might as well use the easy method where you can, right?

But sqlite is old and extremely portable - the only language that springs to mind would be C++. And while that does provide advantages, it's a bit of a stretch to claim those would merit a rewrite.

Re: How SQLite is tested

#32

Post by [deleted] »

derefr wrote: post #22
kkowalczyk wrote: post #8

Earlier quoted context omitted.

1. 3.8.0.2 fixed a logical error in the code. High-level language won't help you if you invert a logical condition or commit some more subtle logical mistake. 2. Our software also needs to be fast. No one needs a slow database engine. There is no high-level language that beats C on tasks like writing sqlite. 3. Finally, show me that unicorn high-level language with strong, automatically provable invariants and one pi…

> 3. Finally, show me that unicorn high-level language with strong, automatically provable invariants and one piece of widely used, reliable software that was written in it. Not implying that this is true now , but this is the specific goal of Rust, with Servo.

[deleted]

Re: How SQLite is tested

#33

Post by adrianN »

kkowalczyk wrote: post #8
flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

1. 3.8.0.2 fixed a logical error in the code. High-level language won't help you if you invert a logical condition or commit some more subtle logical mistake. 2. Our software also needs to be fast. No one needs a slow database engine. There is no high-level language that beats C on tasks like writing sqlite. 3. Finally, show me that unicorn high-level language with strong, automatically provable invariants and one pi…

Airbus planes run software written in Ada. Ada does fairly well in the programming language shootout and is designed for writing reliable software.

Re: How SQLite is tested

#34

Post by prof_hobart »

flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

Your choice of language can help you avoid, or even completely eliminate, certain types of defect.

But no language is ever going to prevent you from coding the wrong intent into your application - your programming language has no idea whether (for example) you are intending to keep or omit a left join in your optimisation routine.

Re: How SQLite is tested

#35

Post by ZoFreX »

bane wrote: post #19

Anybody know how the defect rates in SQLite code compares to that of the NASA Shuttle dev team?

Shuttle software: "the last three versions of the program -- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors."

http://www.fastcompany.com/28121/they-write-right-stuff

Re: How SQLite is tested

#37

Post by chiph »

smtddr wrote: post #21
flyingmutant wrote: post #3

While that undoubtedly sounds very impressive, IMO tests should not be what you use to ensure that your software is correct — in my opinion, they can be actively a bit harmful by giving you a wrong impression of correctness. Just take a look at https://www.sqlite.org/changes.html — after the recent 3.8.0, there were 2 bug fix releases separated by 3-4 days (!), with bug descriptions sounding relatively, hmm, trivial,…

So, how would you test database software? This reminds me of an old Google(?) interview question. "How do you test a Calculator after dropping it?" http://www.youtube.com/watch?v=ILkT_HV9DVU&t=46m50s For all the possible things that could go wrong in a fancy graphing calculator, what should you do to ensure it works properly?

For interview purposes, I build a LEGO Mindstorms framework to push the buttons and check that the correct LCD segments light up. And wait several weeks while it runs through a sufficient number of iterations to give some assurance of correctness.

For real-world purposes, I see if it turns on and gives me 4 when I do 2+2. If it fails, I buy a new one off Amazon.

Re: How SQLite is tested

#38

Post by malkia »

mkesper wrote: post #23

Wow. Impressive! But what bugs me about sqlite is a) suppressing errors giving surprising results (hint: try opening a misspelt database file) b) "syntax error near 'as'": almost impossible to find errors in longer sql statements (are other engines better, giving line numbers?) c) no full outer join https://www.sqlite.org/omitted.html which can be worked around by writing about the double code, thus giving chance for…

a - This way you'll still have valid sqlite context (object), which would contain the error message (and the memory allocated to it), but it would be in erroneous state. That could embed much more information rather than just returning an error code. c - My guess is that would put more memory pressure in terms of rewriting the SQL statement, which could explode more than what a typical sqlite user would expect (there is much tighter memory usage limit, and it could be even more limited).

e.g. SQLite is not trying to do everything, and does not encourage you to. I guess you basically wrap your style around it, rather than other way around.

Re: How SQLite is tested

#39

Post by malkia »

warpech wrote: post #36

"More bugs have been introduced into SQLite while trying to get it to compile without warnings than have been found by static analysis."

I've seen (and done it) myself in other code - for example trying to shutdown integer-conversion code early on, I would (rather prematurely) decide what the type would be, and later not having the warning (as guiding light) I have to rediscover what I did and fix it back. Hit me couple of time when converting 32-bit -> 64-bit c/c++ code, while trying to shutdown the warnings (part of the project requirments).

Re: How SQLite is tested

#40

Post by im3w1l »

ZoFreX wrote: post #35
bane wrote: post #19

Anybody know how the defect rates in SQLite code compares to that of the NASA Shuttle dev team?

Shuttle software: "the last three versions of the program -- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors." http://www.fastcompany.com/28121/they-write-right-stuff

This seems very suspicious! How can you know how many bugs a program has?
Post reply on HN