61–70 of 78 posts
I've been toying with the idea of using SQLite as a data exchange format instead of JSON or XML. I can't stand navigating through complex JSON trees, I want to just use SQL queries. Has anyone else tried this?
> SQLite is extremely reliable and durable for large amounts of data (up to 140TB). It is considered one of the most well-engineered and well-tested software solutions today, with 711x more test code than implementation code. I keep seeing this statement. Why is it considered one of the most well-engineered software?
See: https://www.sqlite.org/testing.html https://www.sqlite.org/hirely.html
> SQLite is extremely reliable and durable for large amounts of data (up to 140TB). This has not been our experience. Our experience is that it sort of bogs down around 12GB in file size. linux box, ext3/4, spinning disk (not SSD)
Any features in this library you'd like to see standard library's sqlite3 [1]? Maybe a PEP [2, python enhancement proposal] could do it. [1] https://docs.python.org/3/library/sqlite3.html [2] https://www.python.org/dev/peps/pep-0001/
Not sure if it's in this library, but I'd want the ability to open a database that's already loaded in memory as a byte array.
Title is misleading. It needs to add something like `python-library`. I thought they supercharged sqlite itself.
Looks like it just bundles a full build of the sqlite shared library, along with some kind of gross hacks. It bundles pysqlite2, which notably is missing a ton of features and improvements from upstream. I'll just compile my own, thanks...
Fair enough if you want to compile your own version, but there's a lot of people and Python devs who don't know how / won't go through the effort and that's what this project is for.
It's also a work in progress and not meant to be released yet. Shoot me an e-mail ajay@plasticityai.com if you have suggestions for how to improve it.
Interesting pick from one of the links in the article: "SQLite has fantastic write performance as well. By default SQLite uses database-level locking (minimal concurrency), and there is an “out of the box” option to enable WAL mode to get fantastic read concurrency — as shown by this test. But lesser known is that there is a branch of SQLite that has page locking, which enables for fantastic concurrent write performa…
Earlier quoted context omitted.
In the standard library? Probably nothing. But if someone published an alternative APSW wheel with JSON1, ICU, and FTS5 enabled, I'd be happy.
I'm the APSW author. The binary builds for Windows are distributed with those extensions all compiled in, although my doc needs some updating. It is also only a single flag for other platforms to include all extensions during compilation. What can I do?
The problem for my case is that if I want to distribute anything with a Sqlite+JSON1 workflow I also have to host and distribute a version of sqlite3 or apsw with those flags enabled. If that's already done on Windows then great, but many people I'm sure (myself included) would like a cross-platform solution.
It would be great if we could use the Setuptools "extras" feature to select which flags we want enabled. Not sure what the technical requirements for that are.
Alternatively just having an "everything enabled" version of the package would i think be good for most people who dont need such fine control over their project dependencies. If anything, someone who cares so much about keeping the library minimal is probably in the minority and should be free to disable extensions as needed.
Earlier quoted context omitted.
I'm the APSW author. The binary builds for Windows are distributed with those extensions all compiled in, although my doc needs some updating. It is also only a single flag for other platforms to include all extensions during compilation. What can I do?
Whats the hold back to getting it via pip install via a wheel? I'm curious (happy APSW user here). Its not well elaborated from what I could find. This is something I would love to help with but not sure where to start, as I'm not sure where in the process of doing the builds it falls down. Would Cython help, perhaps?
I already do Windows builds[2] - 21, becoming 23 in next release so that isn't a difficulty.
The problem is tying all this together in a way that is useful, with sensible defaults and appropriate tools. APSW supports python versions going back 16 years! I welcome discussion at python-sqlite[3]
[1] https://rogerbinns.github.io/apsw/download.html#easy-install...
[2] https://rogerbinns.github.io/apsw/download.html#source-and-b...