Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Observable 2.0, a static site generator for data apps

observablehq.com

Post reply on HN
Print view View original

121–130 of 157 posts

Re: Observable 2.0, a static site generator for data apps

#121

Post by kuatroka »

ddanieltan wrote: post #19

I'm super excited to try this out! Couple of questions since I see @mbostock active in the comments. 1. Is the flexibility of languages used in data loaders/backend going to eventually come to the front end/ui? Or will the paradigm always be bring-your-own-language for the data loading but build your dashboard with observablejs/observable plot? 2. Considering ObservableJS is supported by Quarto, can we look forward t…

3. Apache echarts are much more interactive out of the box. The API is indeed clunky, but they’ve got all the chart type and all interactions you might need. IMHO, Plot in comparison, is very limited in interactivity and even chart types ( there are no heat maps or donuts).

echarts have a huge example library with clear examples and though Plot has it too, the library is not thought out well. You might looks at an example in the Plot Library only to realize later that it’s a D3 example. On the good side, the API in Plot is much cleaner and easier to work with.

Re: Observable 2.0, a static site generator for data apps

#122

Post by kuatroka »

recifs wrote: post #117
chrisjc wrote: post #91

Earlier quoted context omitted.

Came here with similar questions and Cmd-F "DuckDB". See the comment about "data loaders". Seems like a "data loader" would provide most of what you're asking about. I'm also thinking that a "data loader" combined with duckdb-wasm and arrow would be a pretty nice combination. I imagine that it might not be too difficult to switch two between two implementations of the "data loader" as needed. Switch between reading f…

See the example at https://huggingface.co/spaces/observablehq/fpdn where DuckDB is used both as a data loader (to download and digest 200GB worth of source data into a small 8MB parquet file) and on the client-side to allow the user to do live search queries on the minimized data. Server-side, we're using duckdb-the-binary, and client-side we're using duckdb-wasm.

So the 200Gb loading and digesting part is totally separate from the Observable Framework, right? You just do it with a standard ( non wasm duckdb as part of ETL) and later you just direct Observable Framework to read and plot the 8Gb file? Thanks

Re: Observable 2.0, a static site generator for data apps

#123

Post by kuatroka »

tophtucker wrote: post #100
kuatroka wrote: post #83

A couple of questions: 1. Let's say I got a Sqlite/Duckdb database file on my server. It's got multiple tables and some of them 100M to 150M records. I want to create a plot/table that would have a slider/filter to only bring and show a slice of data at a time. Since it's statically generated data, how is this interactivity achieved? All the possible facets of data filtered by which ever way will be generated? Won't…

Good questions. 1. It’s just JavaScript so you can fetch stuff dynamically too (see https://observablehq.com/framework/lib/duckdb ). But yeah, only client-side. (Though see https://github.com/observablehq/framework/issues/234 .) 2. Sure, it’s all open source, I bet you could make that work. Or `yarn deploy` to Observable and configure sharing there (though it wouldn’t let you charge others). 3. Yup. Which is part of…

Thank you I wonder if for #3 there is a way of somehow to keep the data hidden and only let people see the chart without hacking their way to see the underlying data

Re: Observable 2.0, a static site generator for data apps

#124

Post by kuatroka »

Are you planning to add new UI components like Data Table or other in the future or it’s purely Plotting and Data ingestion and the UI through Tailwind or CSS or would it be possible to add UI libraries like shadcn or DaisyUI to make it a full fledged web site? Thanks

Re: Observable 2.0, a static site generator for data apps

#125

Post by anentropic »

amcaskill wrote: post #16
mbostock wrote: post #13

Earlier quoted context omitted.

Yep, Evidence is doing good work. We were most directly inspired by VitePress; we spent months rewriting both D3’s docs ( https://d3js.org ) and Observable Plot’s docs ( https://observablehq.com/plot ) in VitePress, and absolutely loved the experience. But we wanted a tool focused on data apps, dashboards, reports — observability and business intelligence use cases rather than documentation. Compared to Evidence, I’d…

One of the founders of Evidence here. Thanks the kind words Mike - that means a lot coming from you. I think that distinction is right -- we are focused on making a framework that is easy to use with a data analyst skill set, which generally means as little javascript as possible. As an example, the way you program client-side interactions in Evidence is by templating SQL which we run in duckDB web assembly, rather t…

This looks very interesting to me, I'm building a BI reporting tool in my company at the moment, but browsing the docs I felt what I was missing was a clear overview of the architecture.

e.g. you say above that Evidence takes templated SQL and runs it in DuckDB WASM

and then in the docs there's various https://docs.evidence.dev/core-concepts/data-sources/#suppor... like Snowflake, MySQL etc

I guess I am wondering where and when the queries are happening

If I set up a Snowflake data source is it doing a build-time import (like in the new Observable, from this thread) into DuckDB? or DuckDB is connecting to the sources via extensions?

Where does the data live?

My question is really just "how does it work?" and the "What is Evidence? > How does Evidence work?" section on the docs homepage doesn't really answer that at all, it's just a list of things that it does.

Re: Observable 2.0, a static site generator for data apps

#126

Post by shiandow »

mbostock wrote: post #14
0cf8612b2e1e wrote: post #5

Sounds a bit like the “baked data” pattern. Which I think is a really good idea. I have long been toying with how to use Datasette to make a deliverable dashboard, so this is interesting.

Yes! It first felt counterintuitive and constraining to prepare data ahead of time, rather than just loading whatever you want on the fly. But we’ve found this to be a great discipline in practice because it forces you to think about what data you actually need to show, and to shift as much of the compute to build time to reduce file sizes (through aggregation and filtering) and accelerate page load. And once you bec…

Realistically what's the biggest you can make the dataset if you need to prebuild everything? Aggregation and filtering help, but quickly become impractical if you want people to change the filters dynamically.

Re: Observable 2.0, a static site generator for data apps

#128

Post by severo_bo »

kuatroka wrote: post #122
recifs wrote: post #117

Earlier quoted context omitted.

See the example at https://huggingface.co/spaces/observablehq/fpdn where DuckDB is used both as a data loader (to download and digest 200GB worth of source data into a small 8MB parquet file) and on the client-side to allow the user to do live search queries on the minimized data. Server-side, we're using duckdb-the-binary, and client-side we're using duckdb-wasm.

So the 200Gb loading and digesting part is totally separate from the Observable Framework, right? You just do it with a standard ( non wasm duckdb as part of ETL) and later you just direct Observable Framework to read and plot the 8Gb file? Thanks

nope, Observable Framework data loader accesses the 200GB dataset. The code is here: https://huggingface.co/spaces/observablehq/fpdn/blob/main/do...

Re: Observable 2.0, a static site generator for data apps

#129

Post by laurels-marts »

mbostock wrote: post #9

Hey, HN. We’re thrilled to release Observable Framework today — a new open-source tool for developing data apps. I highly recommend viewing this example report adapted from our internal dashboard analyzing web logs: https://observablehq.com/framework/examples/api/ This technique of “just plot everything” (7.6M requests as a scatterplot) has revealed surprising insights we’ve used to optimize our servers and better co…

Very impressive and will definitely server many use-cases. However, it's static site with data refresh at build time. Does this mean there cannot be user-based row-level security (i.e. selective access)?

One of the main selling points of the clunky, general purpose drag-and-drop BI tools (Power BI, Tableau etc.) is selective access. This is especially important in larger enterprises and for customer-facing dashboards.

For example, you're an enterprise manufacturing and selling IoT devices and have many different corporate customers. When you build a dashboard you want to make sure that each customer can see the data that belongs to their account and potentially, have further user-based restrictions. Obviously this goes against the idea of creating pre-aggregated datasets and instant loads but it's a massive multi-billion gap that currently is being filled by inferior tools to D3/Plot/Framework. This is something that Observable could develop in the future given what I'm seeing now and considering how relatively close already you are to this. Framework could serve both types of needs - static sites and dynamic, user-based more fully-featured sites for Enterprise needs.

Re: Observable 2.0, a static site generator for data apps

#130

Post by mbostock »

time4tea wrote: post #114
mbostock wrote: post #43

Earlier quoted context omitted.

I don’t think the `npm install` will scare people away (Evidence uses that, too), and we’ve definitely tried to make the onboarding process as guided as possible (shout-out to clack.cc for a great CLI prompt library): https://observablehq.com/framework/getting-started And plus you can import libraries directly from a CDN rather than needing to use npm or yarn to manage dependencies. (Though we plan on supporting the…

Importing libs from a cdn is a big no-no for almost any system I work on - they are just another way of surveilling users on the Internet, and building information about the insides of organisations.

We plan on downloading the imported libraries during build so that they are self-hosted, so it’s effectively another way to install them without having to run `npm install`. We also plan on supporting importing libraries from node_modules. And you can already import local modules so you can install libraries by manually downloading them, too.
Post reply on HN