Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Datasette is my data hammer

jeremiak.com

Post reply on HN
Print view View original

41–50 of 71 posts

Re: Datasette is my data hammer

#41

Post by plaidfuji »

frogulis wrote: post #3

Only tangentially related: what is the best general-use single hand tool? My feeling is that it would be some kind of small sturdy knife. "When all you have is a knife, a lot of stuff looks passably doable"

For general house projects, I’d say a Visegrip. I’m not saying it can do everything well, but it could be used as a hammer, pliers, adjustable wrench, and I’ve even used it as a screwdriver in a couple of situations. It’s not the best at any of those things, but it can do all of them if necessary.

Re: Datasette is my data hammer

#42

Post by simonw »

Two things I love about this post:

1. It's by a data journalist. Datasette was originally designed for data journalism (though it's useful for all sorts of other things) so it's really great to see it being enthusiastically used in this way

2. Jeremia talks at length about Datastte's URLs! This is a key design principle of Datasette: anything you see (a filtered table, SQL query results, a configure chart visualization) should be reflected in the URL of the page, such that you can bookmark it and share it with others. And any time you can see data you should be able to add .json to that URL (or .csv or, via plugins, other extensions like .atom or .geojson) to get that data back in a useful format.

Re: Datasette is my data hammer

#43

Post by sally_glance »

mritchie712 wrote: post #33
sally_glance wrote: post #8

I tried datasette for the first time not long ago. To me it wasn't obvious what it gave me compared to my local Postgres setup. I have a single docker-compose file with Postgres + pgAdmin and a host mount which I use for data exploration and ad-hoc analysis. I get a nice Web UI and can ingest quickly using the awesome Postgres FDWs. Compared to datasette, it allows me to also transform the data by creating derived vi…

Does pgadmin have drag and drop to import files? So often I have a few parquet or CSVs from various sources (e.g. S3) and need to quickly summarize or join them. Quick imports and summarization is one thing I've always found painful in most tools.

If it has I haven't used it yet - I just have a collection of snippets to create foreign tables (e.g. one based on a CSV I put into the host mount before).

Re: Datasette is my data hammer

#44

Post by frogulis »

plaidfuji wrote: post #41
frogulis wrote: post #3

Only tangentially related: what is the best general-use single hand tool? My feeling is that it would be some kind of small sturdy knife. "When all you have is a knife, a lot of stuff looks passably doable"

For general house projects, I’d say a Visegrip. I’m not saying it can do everything well, but it could be used as a hammer, pliers, adjustable wrench, and I’ve even used it as a screwdriver in a couple of situations. It’s not the best at any of those things, but it can do all of them if necessary.

Have you used it as a screwdriver by holding a screwdriver bit or using the grip directly on the screw? Or something else I'm not imagining?

Re: Datasette is my data hammer

#45

Post by sally_glance »

simonw wrote: post #37
sally_glance wrote: post #8

I tried datasette for the first time not long ago. To me it wasn't obvious what it gave me compared to my local Postgres setup. I have a single docker-compose file with Postgres + pgAdmin and a host mount which I use for data exploration and ad-hoc analysis. I get a nice Web UI and can ingest quickly using the awesome Postgres FDWs. Compared to datasette, it allows me to also transform the data by creating derived vi…

One big advantage is in publishing data online: that as one of Datasette's key original features, and it's something I use constantly: datasette publish vercel til.db --app simon-tils It can deploy databases directly to Cloud Run, Heroku, Fly and Vercel. It can cost pennies per month depending on database size and traffic. My https://til.simonwillison.net site (and many other sites I run) is deployed like that, using…

That publish command looks sweet! For confidential data I'd have to setup some integration with our infra to get the required access control though. Maybe rather go with the package option using docker and a pipeline that deploys to our k8s where I could handle auth using an existing proxy...

My current workflow doesn't really involve creating databases, I mostly just create foreign tables, normal tables and views in a default DB. From time to time I then just dump-backup everything away and start fresh.

Thanks for the response and your tools in general though, I liked how everything just worked when I tried it :-)

Re: Datasette is my data hammer

#46

Post by simonw »

sally_glance wrote: post #45
simonw wrote: post #37

Earlier quoted context omitted.

One big advantage is in publishing data online: that as one of Datasette's key original features, and it's something I use constantly: datasette publish vercel til.db --app simon-tils It can deploy databases directly to Cloud Run, Heroku, Fly and Vercel. It can cost pennies per month depending on database size and traffic. My https://til.simonwillison.net site (and many other sites I run) is deployed like that, using…

That publish command looks sweet! For confidential data I'd have to setup some integration with our infra to get the required access control though. Maybe rather go with the package option using docker and a pipeline that deploys to our k8s where I could handle auth using an existing proxy... My current workflow doesn't really involve creating databases, I mostly just create foreign tables, normal tables and views in…

For confidential data you could use one of the plugins that add authentication - datasette-auth0 or datasette-auth-github or datasette-auth-passwords are options there - but yeah, you're probably better off deploying to internet hosting if you already have an auth proxy setup somewhere.

Re: Datasette is my data hammer

#47

Post by sally_glance »

simonw wrote: post #46
sally_glance wrote: post #45

Earlier quoted context omitted.

That publish command looks sweet! For confidential data I'd have to setup some integration with our infra to get the required access control though. Maybe rather go with the package option using docker and a pipeline that deploys to our k8s where I could handle auth using an existing proxy... My current workflow doesn't really involve creating databases, I mostly just create foreign tables, normal tables and views in…

For confidential data you could use one of the plugins that add authentication - datasette-auth0 or datasette-auth-github or datasette-auth-passwords are options there - but yeah, you're probably better off deploying to internet hosting if you already have an auth proxy setup somewhere.

Oh wow thx for the pointer, I overlooked those. For business I would need a more generic oauth- or saml-based one to work with our Keycloak, but for quick private/hobby sharing the password one looks perfect.

Re: Datasette is my data hammer

#48

Post by misterS »

klondike_klive wrote: post #28
jdblair wrote: post #26

I'm showing my age, but the first thing I thought of when reading the headline was the Commodore Datasette. https://en.m.wikipedia.org/wiki/Commodore_Datasette

I immediately thought of the musician Datassette - proponent of the micro-genre Business Funk. https://www.youtube.com/watch?v=zgWa7NoJGWc

I did, too :)

For those that don't know: Datasette is the author/curator of https://musicforprogramming.net

Re: Datasette is my data hammer

#49

Post by kreelman »

simonw wrote: post #37
sally_glance wrote: post #8

I tried datasette for the first time not long ago. To me it wasn't obvious what it gave me compared to my local Postgres setup. I have a single docker-compose file with Postgres + pgAdmin and a host mount which I use for data exploration and ad-hoc analysis. I get a nice Web UI and can ingest quickly using the awesome Postgres FDWs. Compared to datasette, it allows me to also transform the data by creating derived vi…

One big advantage is in publishing data online: that as one of Datasette's key original features, and it's something I use constantly: datasette publish vercel til.db --app simon-tils It can deploy databases directly to Cloud Run, Heroku, Fly and Vercel. It can cost pennies per month depending on database size and traffic. My https://til.simonwillison.net site (and many other sites I run) is deployed like that, using…

Your preaching to the choir Simon.

...But the tenor ...whose pitch is a little off... up the back, Is listening even though he's heard it before :-)

I think the ability to so quickly create and consume an SQLite DB is one of the big advantages here. I need to use bigger DBs like PostGres and MSSQL (in previous roles) for bigger tasks, but SQLite is down there with fopen in terms of time to create a DB.

I find myself imagining schemas for problems that I'd like to put into SQLite and possibly DataSette.

Thanks for making DataSette Simon.

Post reply on HN