My understanding of serverless = easily scalable, managed service. But somehow the word annoys people. Maybe we should find a better word?
I never found things like DBaaS annoying at all.
SQLite Is Serverless
281–290 of 453 posts
Re: SQLite Is Serverless
#282Earlier quoted context omitted.
I will shamelessly plug myself again. I am the main author of RediSQL [1] and I am about to launch a managed service for it. It will allow to write SQL (SQLite dialect) against an HTTP or Redis protocol, to make thing clearer: https://simplesql.carrd.co/ Eventually it will upload your database to an S3 bucket for backup. [1]: https://redisql.com/
Looks pretty cool! I can imagine it would be useful for data-science projects, if the pricing is right.
What pricing would feel right to you?
Re: SQLite Is Serverless
#283Earlier quoted context omitted.
Yes, for Access (and sqlite?) it is the client computers that do the edits directly to the database file, rather than via a mediating server side process, so they are particularly vulnerable to network interruptions while in the middle of an edit. Cable is far more reliable than wifi and so far fewer interruptions. I think that's it.
Ok but you cant really say you support network access if you rely on it to be flawless. Working fast, maybe but not reliably working if your network is not super reliable is not really support.
It was a different time.
Re: SQLite Is Serverless
#284Earlier quoted context omitted.
It might have been designed for it. It was poorly designed for it. It was a clustfuck of corruption when actually utilized
It depended. It suited a small cabled office with 10 to 15 computers fine. I saw such setups work fine for 12 years without corruption. But when wifi came along people started connecting that way, sometimes unintentionally, and corruption became an issue. So then we bit the bullet and moved the backend to a server database. Not a single database corruption since.
But this is the difference between supporting concurrent access or not. The fact that the feature relied so much on network reliability means that race conditions still existed.
Re: SQLite Is Serverless
#285I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…
I’d love a SQLite to macOS Excel (or any macOS spreadsheet application) workflow so less technical users can do analysis. Has anybody pulled this off?
Re: SQLite Is Serverless
#286"Microsoft Azure Cosmo DB and Amazon S3 are examples of a neo-serverless databases." Can we back away a bit from the bandwagoning of misused terminology? Serverless literally means "running your apps on somebody else's server". S3 is not a server you run your apps on, it is SaaS that you manipulate through an API - you don't put your apps on it. If S3 is serverless, then literally every network service of any kind is…
I always preferred XaaS nomenclature myself. "XaaS with autoscaling"... is probably better than "Serverless"
Re: SQLite Is Serverless
#287Earlier quoted context omitted.
Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!
Do you generate the file from scratch every time or do you modify the previous one as new data arrives?
It is a bit of a hybrid.
In the early stages of development of whatever the ETL process is, I keep the database and just empty it out each time. As I got more of a sense of what I needed, I started DROPing my TABLEs more often and remaking them. Eventually I would make the whole database from scratch once I was along the way and had most everything fleshed out.
Re: SQLite Is Serverless
#288I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…
I've wondered about this too, but have not gotten around to trying it yet.
We get a gnarly csv log file back from our sensors in the field, which is really a "flattened" relational data model. What I mean by that is a file with "sets" of records of various lengths, all stacked on top of each other. So, if you open it in Excel, (which many users do), the first set of 50 rows may be 10 columns wide, the next 100 rows will be 20 columns wide, the next 45 wide, etc. And, the columns for each of these record sets have different names and data types.
Converting to JSON is obvious, but I've thought about just creating a SQLite file with tables for each of the sets of records. Then, as others have said, can use one of any number to tools to easily query/examine the file. Also can easily import into a pandas data frame.
One concern is file size. Any comments on this? I can try it, but wonder if anyone knows off the top of their heads if a large JSON file converted to an SQLLite file would be a lot larger or smaller?
edit: clarity
Re: SQLite Is Serverless
#289I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…
I was under the impression SQLite files were not supposed to be moved across architectures.
Re: SQLite Is Serverless
#290I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…
I’d love a SQLite to macOS Excel (or any macOS spreadsheet application) workflow so less technical users can do analysis. Has anybody pulled this off?
You mean like the .excel command?
"... causes them to accumulate output as Comma-Separated-Values (CSV) in a temporary file, then invoke the default system utility for viewing CSV files (usually a spreadsheet program) on the result. This is a quick way of sending the result of a query to a spreadsheet for easy viewing"