Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Observable 2.0, a static site generator for data apps

observablehq.com

Post reply on HN
Print view View original

91–100 of 157 posts

Re: Observable 2.0, a static site generator for data apps

#91

Post by chrisjc »

kuatroka wrote: post #83

A couple of questions: 1. Let's say I got a Sqlite/Duckdb database file on my server. It's got multiple tables and some of them 100M to 150M records. I want to create a plot/table that would have a slider/filter to only bring and show a slice of data at a time. Since it's statically generated data, how is this interactivity achieved? All the possible facets of data filtered by which ever way will be generated? Won't…

Came here with similar questions and Cmd-F "DuckDB". See the comment about "data loaders". Seems like a "data loader" would provide most of what you're asking about.

I'm also thinking that a "data loader" combined with duckdb-wasm and arrow would be a pretty nice combination. I imagine that it might not be too difficult to switch two between two implementations of the "data loader" as needed. Switch between reading from a remote system (in your case DuckDB on a server) and DuckDB running locally in the browser (that can interact with its own remote or local data sources).

edit: welp https://observablehq.com/framework/lib/duckdb

Re: Observable 2.0, a static site generator for data apps

#92

Post by chrisjc »

mbostock wrote: post #56
polskibus wrote: post #54

Earlier quoted context omitted.

Thank you Mike for pushing the visualisation envelope for so many years. Is the new Framework going to support virtualized data access for data sets too large to be sent over network (think of a pivot table that allows to browse huge data warehouse) - it is impossible to prepare entire file upfront, so data queries must happen incrementally with users actions? Or is it completely the other direction from where your v…

If you generate Apache Parquet files you can use DuckDB to make range requests and not download everything to the client. This is pretty magical and allows you to have surprisingly large datasets still queryable at interactive speeds. But the general idea is to not send everything the client — to be more deliberate and restrictive in what you send, and also what you show. So you probably shouldn’t use this for a gene…

It's not always clear which pushdowns are available in DuckDB. For instance, while x = y has been available for a while, x in (y, z, ...) hasn't. The DuckDB team seems very eager and motivated to get all the pushdown functionality working though, so hopefully becomes a non-issue soon (perhaps already in 0.10.0).

Another way to use DuckDB if you're warehouse supports it would be e2e Arrow (no col->row->col overhead).

    warehouse --> ADBC --> arrow --> DuckDB
Of course this differs in that you would be reading from the warehouse directly, but in my experience fully pre-aggregating data and then staging it (keeping parquet files on S3 up to date) might solve one issue, but results in unimaginable issues. Perhaps the sweet spot might be something like Iceberg in the middle?

    warehouse --> iceberg table (parquet) --> DuckDB
> craft corresponding data loaders

Do you have an example of this using DuckDB? I'm very interested in seeing an actual implementation of Observable's data loaders combined with DuckDB (or any other SQL DB)

edit: nm, found it. https://observablehq.com/framework/lib/duckdb

edit2: eh, I didn't even realize who i was responding to, lol. The more I read into this, the more I see how this is all heavily based on static files. So the static parquet files thing makes more sense, the solution I added makes little. Although I guess you could add a static iceberg table and interact with its manifest with the duckdb iceberg extension.

Re: Observable 2.0, a static site generator for data apps

#93

Post by fredguth »

mbostock wrote: post #9

Hey, HN. We’re thrilled to release Observable Framework today — a new open-source tool for developing data apps. I highly recommend viewing this example report adapted from our internal dashboard analyzing web logs: https://observablehq.com/framework/examples/api/ This technique of “just plot everything” (7.6M requests as a scatterplot) has revealed surprising insights we’ve used to optimize our servers and better co…

Interesting to see ObservableHQ making strides towards dashboards, similar to what Quarto and Evidence are doing.

Observable Notebooks reactivity feels intuitive, much like spreadsheets, but the lack of self-hosting options is no-go Drawback in my work context.

Re: Observable 2.0, a static site generator for data apps

#94

Post by mythmon_ »

chrisjc wrote: post #90
mythmon_ wrote: post #32

Earlier quoted context omitted.

Observable engineer here. I haven't looked into Malloy much, but Framework's data loaders are very flexible. If you can write a script or binary that uses Malloy and writes to stdout, it can be a data loader. For example, although we use SQL a lot in our internal usage of Framework, Framework doesn't actually have any specific SQL support. We just use our database's normal bindings.

Can you link to the "data loader" API or perhaps even a "data loader" example implementation for something similar to malloy, duckdb, or any other DB/SQL data source/provider? Would love to see it, thanks in advance! edit: found it https://observablehq.com/framework/lib/duckdb

The docs for data loaders are here: https://observablehq.com/framework/loaders. The simple version is they are simply programs that write their output to standard out. Very Unixey. When those programs are referenced in client side parts of the JS, they are reactively run when in development, and prebuilt for deployment.

I don't think we have any full examples of using a database yet, but we have written a bit about using DuckDB via its Node bindings here: https://observablehq.com/framework/lib/duckdb

I imagine that either Malloy's CLI or its Python bindings would fit very well here.

Re: Observable 2.0, a static site generator for data apps

#95

Post by ddanieltan »

mbostock wrote: post #58
zurfer wrote: post #52

Earlier quoted context omitted.

I also love how Observable Plot looks, but agree with top poster, the things that keep me from switching are: - Python wrapper - Out of the box interactivity

We’re working on the interactivity, but we’re not going to do a Python wrapper as the goal of Observable Framework (and Plot) is to leverage web technologies with JavaScript in the front-end for interactive graphics — while doing whatever language you like, including Python, on the back-end for data analysis. There is a R wrapper for Observable Plot ( https://github.com/juba/obsplot ) and so I imagine someone could w…

FYI: Same dev building obsplot is also building a Python version at https://github.com/juba/pyobsplot

Re: Observable 2.0, a static site generator for data apps

#96

Post by dleeftink »

kepano wrote: post #53

I appreciate the nod to "File over app"[1] in the announcement. It's so cool that a Markdown file with code blocks can be the source for complex data visualizations and dashboards. Interoperability of this kind makes me giddy. I played around with editing an Observable site from Obsidian and it works great[2]. [1]: https://stephango.com/file-over-app [2]: https://twitter.com/kepano/status/1758202572446581025

My mind immediately went to how these Dasboards could be integrated in Obsidian, and seeing the `import` dependency graph reflected in Obsidian's graph view.

Re: Observable 2.0, a static site generator for data apps

#97

Post by dleeftink »

tophtucker wrote: post #57
simonw wrote: post #47

There's an almost bewildering amount of interesting ideas buried in this. Things like data loaders which are ANY script that can output data (as JSON or something else) to standard output. Markdown files with ```js blocks in that get executed. The reinvention of the core Observable notebook to avoid custom syntax. This is really big.

Yeah data loaders are like a UNIX pipe to a reactive notebook cell (?). There needn’t be any question of “do the data loaders support this or that”; it doesn’t even have a concept of “supporting” beyond supporting stdout… Still thinking through how to understand it myself!!

Scripting with hot-reload?

Re: Observable 2.0, a static site generator for data apps

#98

Post by tophtucker »

skybrian wrote: post #82
tophtucker wrote: post #57

Earlier quoted context omitted.

Yeah data loaders are like a UNIX pipe to a reactive notebook cell (?). There needn’t be any question of “do the data loaders support this or that”; it doesn’t even have a concept of “supporting” beyond supporting stdout… Still thinking through how to understand it myself!!

Data loaders seem like an interesting way to define a multi-language build system without having to write a makefile. Lots of build systems do this, but the boundaries between build steps often isn't as clean and uniform as having a single output per build step and relying on a file naming convention. It's not truly reactive if you have to do a build to make anything happen. But maybe that doesn't matter, as long as…

Yeah there are some open issues about more granular rebuilds and chaining data loaders. https://github.com/observablehq/framework/issues/638, https://github.com/observablehq/framework/issues/332

It’s kinda cool to think about the shearing layers of reactivity. Reactivity is what originally drew me to Observable. But the way notebooks have to be recomputed live for every viewer every time makes them feel silly for, like, a BI dashboard that changes daily at most. Like they only have one pace layer. Like they’re trying so hard to be _live_ that they can’t be _fast_! Idk. I guess even a chalkboard is reactive on the timescale of “someone noticing some information and telling it to someone who writes it down” lol.

Re: Observable 2.0, a static site generator for data apps

#99

Post by one_buggy_boi »

What would be a good design pattern to put these dashboards behind auth? I suppose since they're static files you could just serve them with something like FastAPI or Spring Boot and have your CI/CD refresh the static files throughout the day on shared storage?

Re: Observable 2.0, a static site generator for data apps

#100

Post by tophtucker »

kuatroka wrote: post #83

A couple of questions: 1. Let's say I got a Sqlite/Duckdb database file on my server. It's got multiple tables and some of them 100M to 150M records. I want to create a plot/table that would have a slider/filter to only bring and show a slice of data at a time. Since it's statically generated data, how is this interactivity achieved? All the possible facets of data filtered by which ever way will be generated? Won't…

Good questions.

1. It’s just JavaScript so you can fetch stuff dynamically too (see https://observablehq.com/framework/lib/duckdb). But yeah, only client-side. (Though see https://github.com/observablehq/framework/issues/234.)

2. Sure, it’s all open source, I bet you could make that work. Or `yarn deploy` to Observable and configure sharing there (though it wouldn’t let you charge others).

3. Yup. Which is part of the appeal of model of running data loaders at build time: you can query some private data and viewers would only be able to see the final result set. (The lack of something like this has always been a huge problem for Observable notebooks. You’d make some great query-driven charts and then couldn’t make it public without some awkward manual dance of downloading and re-uploading a file to a fork of the notebook.)

4. I wish I knew! It’s being tracked here https://github.com/observablehq/plot/issues/1711. Lately there’s been a lot more work on Framework naturally but now that that’s out…

5. Another good question. We’re definitely interested in tailoring it more to this sort of use case but lots is TBD!

Post reply on HN