31–40 of 71 posts
Only tangentially related: what is the best general-use single hand tool? My feeling is that it would be some kind of small sturdy knife. "When all you have is a knife, a lot of stuff looks passably doable"
Context is everything. Outdoors? Probably a very sharp machete. Working in a genius bar? Probably that weird-ass hextool screwdriver thing.
Can Datasette be used with a Postgres RDS database with hundred GBs? Feels like it's mostly designed for smaller things? Also if it can, how would I go about it and if it can't, any similar tools I can use? This looks really cool!!
This is a self-plug but shapedata.io does similar things on top of Postgres -- it is much younger than Datasette, so has far fewer features, but should give you a similar exploration experience!
I tried datasette for the first time not long ago. To me it wasn't obvious what it gave me compared to my local Postgres setup. I have a single docker-compose file with Postgres + pgAdmin and a host mount which I use for data exploration and ad-hoc analysis. I get a nice Web UI and can ingest quickly using the awesome Postgres FDWs. Compared to datasette, it allows me to also transform the data by creating derived vi…
I'm showing my age, but the first thing I thought of when reading the headline was the Commodore Datasette. https://en.m.wikipedia.org/wiki/Commodore_Datasette
I had also assumed it would be a good unique name that I could subscribe to search results for... only to find that the Commodore 64 vintage computing community is still way more active than I had ever imagined!
I tried datasette for the first time not long ago. To me it wasn't obvious what it gave me compared to my local Postgres setup. I have a single docker-compose file with Postgres + pgAdmin and a host mount which I use for data exploration and ad-hoc analysis. I get a nice Web UI and can ingest quickly using the awesome Postgres FDWs. Compared to datasette, it allows me to also transform the data by creating derived vi…
datasette publish vercel til.db --app simon-tils
My https://til.simonwillison.net site (and many other sites I run) is deployed like that, using GitHub Actions: https://github.com/simonw/til/blob/81fdedb8e9b047cb059ef22c6...
See also the baked data pattern where I talk about this kind of deployment in a lot of detail: https://simonwillison.net/2021/Jul/28/baked-data/
For ad-hoc data analysis - assuming you already know your way around PostgreSQL - your pgAdmin in Docker solution is hard to beat.
How often do you create a new database?
My favourite thing about working with SQLite is that the overhead of new database creation is equivalent to the overhead of creating a file. In a given day I might casually create dozens of new databases, then throw away most of them. I never found myself doing that with MySQL or PostgreSQL.
Earlier quoted context omitted.
In my case, I log the GPS data from my car, but wanted a private way to explore that (obviously sensitive) data. https://datasette.io/plugins/datasette-cluster-map is a really neat example of a "just works" plugin: If you have latitude and longitude columns, it will plot them! But there are numerous plugins for certain types of visualizations, there's a bunch for handling editing, inserts, uploads, etc. Some of the e…
Will datasette automatically provide useful graph views of time series? Or is there a good plug-in of that? One thing I’m trying to get better at is resource tracking (mostly energy) but it’s pretty clunky using a spreadsheet (likely because I’m not super knowledgeable and really don’t care).
I've been hoping to build a new one around Observable Plot for a while now.
This plugin still requires you to pick what columns you would like to see plotted. Ideally I'd like it to take an informed guess based on the data and maybe offer you some thumbnails of suggested charts which you could then click to view larger.
Interesting project. I would've liked to have heard a brief comparison with Excel - what makes Datasette something that one should reach for over that? Is it just the ability to use SQL in-place?
Right now I'd say the big things are:
1. Size of data. Datasette is a great tool for when your data is too big for Excel but not so big that you need a data warehouse - think 100MB to 10GB range
2. Interoperability and automation. Building tools in other languages that interoperate with an Excel spreadsheet - a web crawler that updates it constantly over time for example - is possible but in my experience pretty hard. Writing to a SQLite file is usually much easier in comparison
3. Publishing data. This is a core concept in Datasette: the ability to publish your data online so other people can explore and interact with it.
4. Plugins. Writing plugins for Datasette is pretty easy, and there are over 100 now: https://datasette.io/plugins - so if it doesn't do something you need, you can add that feature independently of the core project
5. SQLite generally. Some projects are a better fit for a database than a spreadsheet, and SQLite is a /really/ good (often underestimated) database.
Can Datasette be used with a Postgres RDS database with hundred GBs? Feels like it's mostly designed for smaller things? Also if it can, how would I go about it and if it can't, any similar tools I can use? This looks really cool!!
It used Django for the authentication layer but can otherwise work against any PostgreSQL database.
I partly built it to help explore what Datasette could look like if it expanded to work with more databases than SQLite. That's still something I'm considering doing in the future, via a plugin hook, but it's not on my short-term roadmap.