SuperSQLite: SQLite library for Python (2018) • Hacker News Board

Re: SuperSQLite: SQLite library for Python (2018)

#21

Post by ritchiea » Fri, Aug 23, 2019, 1:09 PM UTC

firebacon wrote: post #12

johnisgood wrote: post #4

> SQLite is extremely reliable and durable for large amounts of data (up to 140TB). It is considered one of the most well-engineered and well-tested software solutions today, with 711x more test code than implementation code. I keep seeing this statement. Why is it considered one of the most well-engineered software?

About the well-tested bit: Per it's own documentation, SQLite has a massive test suite. [*Not all of] the test suite is actually open source though, so the overlap between commenters selling you on how well tested SQLite is and those that have seen how the sausage is made is probably [close to] zero. However, pointing this or any of the other practical shortcomings of SQLite out on hacker news is blasphemy and will i…

Why are portions of the test suite closed source?

Re: SuperSQLite: SQLite library for Python (2018)

#22

Post by mtw » Fri, Aug 23, 2019, 1:15 PM UTC

How accurate is this statement? "SQLite is faster than nearly every other database". It links to a page that's decades old.

Re: SuperSQLite: SQLite library for Python (2018)

#23

Post by breakingcups » Fri, Aug 23, 2019, 1:15 PM UTC

platz wrote: post #8

johnisgood wrote: post #4

> SQLite is extremely reliable and durable for large amounts of data (up to 140TB). It is considered one of the most well-engineered and well-tested software solutions today, with 711x more test code than implementation code. I keep seeing this statement. Why is it considered one of the most well-engineered software?

I would like not to have to deal with SQLITE_BUSY errors for once. It even throws when trying to obtain a connection. It got so bad I had to put a mutex around obtaining a sqlite connection.

It's possible you are using it wrong.

Re: SuperSQLite: SQLite library for Python (2018)

#24

Post by firebacon » Fri, Aug 23, 2019, 1:16 PM UTC

ritchiea wrote: post #21

firebacon wrote: post #12

Earlier quoted context omitted.

About the well-tested bit: Per it's own documentation, SQLite has a massive test suite. [*Not all of] the test suite is actually open source though, so the overlap between commenters selling you on how well tested SQLite is and those that have seen how the sausage is made is probably [close to] zero. However, pointing this or any of the other practical shortcomings of SQLite out on hacker news is blasphemy and will i…

Why are portions of the test suite closed source?

They charge money for it...

Note that I'm not saying all of the test suite should be open source. It's clearly a valid/cool business model and SQLite is, for a lot of use cases, an excellent piece of software that I have often used myself. The criticism was honestly more directed towards the cherry-picking of facts that sometimes happens in discussions on hacker news, especially those related to SQLite.

Re: SuperSQLite: SQLite library for Python (2018)

#25

Post by quietbritishjim » Fri, Aug 23, 2019, 1:18 PM UTC

firebacon wrote: post #13

PudgePacket wrote: post #10

Earlier quoted context omitted.

Not a direct answer but it's certainly the most used, which correlates highly with getting a lot of dev attention :) https://sqlite.org/mostdeployed.html

That completely depends on the chosen metric/definition though. For example, you could consider the most used database the one that handles the most queries per day. Not so clear who is the winner now. The claim that SQLite is one of the 5 most widely deployed pieces of software worldwide is completely unsubstantiated and most likely untrue once you count in things like ICU, the linux kernel, etc.

> ICU, the linux kernel, etc.

It seems unlikely that either of these are more widely deployed than SQLite, especially the Linux kernel.

Probably the vast majority of deployments of the Linux kernel include SQLite. Add in all the non-Linux devices (e.g. the billions of iOS and Windows devices) and that's probably far more than the number of Linux installations without SQLite.

I don't know much about ICU, and perhaps you're right that it's one of the most used bits of software. But I can certainly imagine embedded devices that never communicate with users and wouldn't need ICU but do need to store data.

Re: SuperSQLite: SQLite library for Python (2018)

#26

Post by tony » Fri, Aug 23, 2019, 1:19 PM UTC

Any features in this library you'd like to see standard library's sqlite3 [1]? Maybe a PEP [2, python enhancement proposal] could do it.

[1] https://docs.python.org/3/library/sqlite3.html

[2] https://www.python.org/dev/peps/pep-0001/

Re: SuperSQLite: SQLite library for Python (2018)

#27

Post by mytherin » Fri, Aug 23, 2019, 1:20 PM UTC

firebacon wrote: post #12

johnisgood wrote: post #4

> SQLite is extremely reliable and durable for large amounts of data (up to 140TB). It is considered one of the most well-engineered and well-tested software solutions today, with 711x more test code than implementation code. I keep seeing this statement. Why is it considered one of the most well-engineered software?

About the well-tested bit: Per it's own documentation, SQLite has a massive test suite. [*Not all of] the test suite is actually open source though, so the overlap between commenters selling you on how well tested SQLite is and those that have seen how the sausage is made is probably [close to] zero. However, pointing this or any of the other practical shortcomings of SQLite out on hacker news is blasphemy and will i…

Part of the SQLite test cases are open source (the logic tests), but indeed unfortunately the biggest part are not. You can purchase a license for these tests though, so saying nobody has seen them is likely incorrect.

Either way, the fact that it is DO-178B certified for use on aircraft should tell you something about the reliability of it. It is certainly more reliable than most other software, OSS or otherwise.

I am curious what practical problems you encountered with SQLite. Could you perhaps expand on that?

Re: SuperSQLite: SQLite library for Python (2018)

#28

Post by stedaniels » Fri, Aug 23, 2019, 1:23 PM UTC

nabdab wrote: post #11

This looks like a dead project that just bundles together the built-in sqlite3 And another wrapper library APSW which itself is more active than this project. Why is this on hn?

No commits in 9 months, "looks like a dead project".

Re: SuperSQLite: SQLite library for Python (2018)

#29

Post by ddorian43 » Fri, Aug 23, 2019, 1:24 PM UTC

Title is misleading. It needs to add something like `python-library`. I thought they supercharged sqlite itself.

Re: SuperSQLite: SQLite library for Python (2018)

#30

Post by firebacon » Fri, Aug 23, 2019, 1:26 PM UTC

quietbritishjim wrote: post #25

firebacon wrote: post #13

Earlier quoted context omitted.

That completely depends on the chosen metric/definition though. For example, you could consider the most used database the one that handles the most queries per day. Not so clear who is the winner now. The claim that SQLite is one of the 5 most widely deployed pieces of software worldwide is completely unsubstantiated and most likely untrue once you count in things like ICU, the linux kernel, etc.

> ICU, the linux kernel, etc. It seems unlikely that either of these are more widely deployed than SQLite, especially the Linux kernel. Probably the vast majority of deployments of the Linux kernel include SQLite. Add in all the non-Linux devices (e.g. the billions of iOS and Windows devices) and that's probably far more than the number of Linux installations without SQLite. I don't know much about ICU, and perhaps y…

My point was less about arguing which is more popular, but more that the same claim can equally be made for at least hundreds if not thousands of other software packages. Still, SQLite stands out to me as the only package I can recall that consistently puts this message all over the documentation front and center. Which is definitely still a useful signal, even if we both agree that it is, in fact, widely used.