Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Observable 2.0, a static site generator for data apps

observablehq.com

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Observable 2.0, a static site generator for data apps

#141

Post by mythmon_ »

kuatroka wrote: post #123
tophtucker wrote: post #100

Earlier quoted context omitted.

Good questions. 1. It’s just JavaScript so you can fetch stuff dynamically too (see https://observablehq.com/framework/lib/duckdb ). But yeah, only client-side. (Though see https://github.com/observablehq/framework/issues/234 .) 2. Sure, it’s all open source, I bet you could make that work. Or `yarn deploy` to Observable and configure sharing there (though it wouldn’t let you charge others). 3. Yup. Which is part of…

Thank you I wonder if for #3 there is a way of somehow to keep the data hidden and only let people see the chart without hacking their way to see the underlying data

During our early exploration, someone made a data loader that returned an entire svg of a chart, instead of the data for a chart. I think it was a headless browser running Observable Plot, but I imagine there are lots of ways to generate charts in a data loader.

Re: Observable 2.0, a static site generator for data apps

#143

Post by amcaskill »

anentropic wrote: post #125
amcaskill wrote: post #16

Earlier quoted context omitted.

One of the founders of Evidence here. Thanks the kind words Mike - that means a lot coming from you. I think that distinction is right -- we are focused on making a framework that is easy to use with a data analyst skill set, which generally means as little javascript as possible. As an example, the way you program client-side interactions in Evidence is by templating SQL which we run in duckDB web assembly, rather t…

This looks very interesting to me, I'm building a BI reporting tool in my company at the moment, but browsing the docs I felt what I was missing was a clear overview of the architecture. e.g. you say above that Evidence takes templated SQL and runs it in DuckDB WASM and then in the docs there's various https://docs.evidence.dev/core-concepts/data-sources/#suppor... like Snowflake, MySQL etc I guess I am wondering whe…

Thanks for the kind words.

That’s good feedback on the docs. The tool has evolved pretty dramatically from where it started and we should revisit those diagrams.

Evidence is a static site generator.

Queries against your sources happen at build time and save to parquet.

Queries against the built in DuckDB web assembly instance happen at runtime.

Sources (snowflake, Postgres, csv files etc.) run at build time.

Pages in evidence are defined as markdown files. You write markdown, components, and code fences.

SQL code fences in pages run in the built in duck db wasm instance which can query across the results from all of your sources. These queries run in the client. We call this feature universal SQL, and it’s quite new.

You can read about universal SQL here if it’s of interest. https://evidence.dev/blog/why-we-built-usql/

You can template those SQL queries to accept input from input components. This enables you to build extremely performant client side interactions.

Under the hood, Evidence is built on svelte and compiles to a svelte kit application, and you can extend your project with custom svelte components.

Hope that’s helpful — we’re very active in our slack if you ever want to say hi!

Re: Observable 2.0, a static site generator for data apps

#144

Post by laurels-marts »

mbostock wrote: post #56
polskibus wrote: post #54

Earlier quoted context omitted.

Thank you Mike for pushing the visualisation envelope for so many years. Is the new Framework going to support virtualized data access for data sets too large to be sent over network (think of a pivot table that allows to browse huge data warehouse) - it is impossible to prepare entire file upfront, so data queries must happen incrementally with users actions? Or is it completely the other direction from where your v…

If you generate Apache Parquet files you can use DuckDB to make range requests and not download everything to the client. This is pretty magical and allows you to have surprisingly large datasets still queryable at interactive speeds. But the general idea is to not send everything the client — to be more deliberate and restrictive in what you send, and also what you show. So you probably shouldn’t use this for a gene…

I’ve been searching for alternatives to crossfilter.js for slicing large multidimensional datasets at sub 10ms speeds. Would DuckDB-Wasm be a suitable alternative?

Re: Observable 2.0, a static site generator for data apps

#145

Post by tophtucker »

skybrian wrote: post #104
tophtucker wrote: post #98

Earlier quoted context omitted.

Yeah there are some open issues about more granular rebuilds and chaining data loaders. https://github.com/observablehq/framework/issues/638 , https://github.com/observablehq/framework/issues/332 It’s kinda cool to think about the shearing layers of reactivity. Reactivity is what originally drew me to Observable. But the way notebooks have to be recomputed live for every viewer every time makes them feel silly for, l…

Another interesting example: since Go packages use minimum version selection, publishing a new version of a package doesn't do anything right away. Someone has to notice it (perhaps reading a release announcement) and bump the version number on a dependency. Then, after testing, they might release a new version, which again, doesn't do anything until projects downstream from them decide to upgrade. That's deliberate,…

Haha yeah, that’s well put — “latest” depends on time, and something that depends on time is not deterministic, so those principles conflict. Currently a Framework project has the opposite priorities as Go and always gets the latest version of the data.

When I first started working at Observable I explained reactivity to my now-wife as “when one thing changes, other things that depend on it update automatically.” She was at an asset management job at the time working in Excel all day, which of course is also reactive in that sense. But she interpreted my statement differently; she said “Oh that’s terrible! If a number changes I want to know.” E.g. if the estimated budget for a project had ballooned, she wanted to hear about it so she could check why, ask other people about it, try to contain the damage, warn others, make other decisions differently, etc. “Automatic” would be harmful, like in your Go example. She was also thinking about coarser-grained data than I was — a single very impactful number, not a stream of mousemoves or pageloads or ad clicks or whatever.

Re: Observable 2.0, a static site generator for data apps

#146

Post by allisonhorst »

CJefferson wrote: post #59

Can anyone with knowledge of both systems compare this to quarto for me?

Hi there! I'm Allison, I've spent lots of time working in R, R Markdown, and more recently Quarto for environmental data science work & teaching. Working in Quarto is actually what introduced me to Observable ("what is this 'OJS' business?")! Now I'm a Developer Marketing Manager at Observable and am really enjoying working in Observable Framework.

I still love Quarto. It lets me tinker, explore & troubleshoot in R easily while building things (I still have a bit of a hard time troubleshooting data loaders). I like that I can use Quarto for dashboards, but also for a personal blog, a scientific article, a nicely formatted PDF, etc. -- it feels like I only need to know one system to be able to create a bunch of different products. There are options for people like me who are more data scientist, less developer to quickly customize some biggies like fonts, background colors, etc. right in the yaml. There are really nice helpers (e.g. for cross referencing and citations, among others) that are great for researchers.

What I like about Observable Framework is that getting started is so fast - minutes from install, to a dashboard I can preview & update locally, to deploy - with really clear prompts to walk me through it. I do really like data loaders. It feels better to just prep my data in an R script (or Python, or whatever else), then access the data I need from the loader (rather than to pass outputs from R code to an OJS cell in Quarto). Using any JS libraries and components feels more streamlined in Observable Framework (not sure why - the syntax isn't that different, though Framework uses import rather than require now). And dashboard layouts are easier for me in Framework using our grid class.

I like how focused Observable Framework feels. Quarto feels awesome with is breadth of possibilities (output to PDF, or Word, or slides, or blogs, or ebooks, or websites, or ...!), whereas Framework feels sleek in its focus and design for developers creating beautiful, fast data apps & dashboards.

Re: Observable 2.0, a static site generator for data apps

#148

Post by Optimal_Persona »

mbostock wrote: post #138
apitman wrote: post #105

Earlier quoted context omitted.

That's an interesting quote. What is the difference between charting and graphing in this context?

See Leland Wilkinson’s The Grammar of Graphics . He describes the difference between a chart typology (a fixed set of chart types with a fixed set of configuration options) and a grammar of graphics (a set of orthogonal primitives that can be composed in arbitrary ways).

Wilkinson inspired Hadley Wickham to articulate a Layered Grammar of Graphics embodied in his ggplot R visualization package.

https://vita.had.co.nz/papers/layered-grammar.html

Re: Observable 2.0, a static site generator for data apps

#149

Post by RyanHamilton »

kuatroka wrote: post #121
ddanieltan wrote: post #19

I'm super excited to try this out! Couple of questions since I see @mbostock active in the comments. 1. Is the flexibility of languages used in data loaders/backend going to eventually come to the front end/ui? Or will the paradigm always be bring-your-own-language for the data loading but build your dashboard with observablejs/observable plot? 2. Considering ObservableJS is supported by Quarto, can we look forward t…

3. Apache echarts are much more interactive out of the box. The API is indeed clunky, but they’ve got all the chart type and all interactions you might need. IMHO, Plot in comparison, is very limited in interactivity and even chart types ( there are no heat maps or donuts). echarts have a huge example library with clear examples and though Plot has it too, the library is not thought out well. You might looks at an ex…

I can second a +1 vote for eCharts. I have used them for fast updating charts (10x second) and they work and look great. I've only had 2-3 small niggles with them in 2years of heavy usage.

Re: Observable 2.0, a static site generator for data apps

#150

Post by RyanHamilton »

farhanhubble wrote: post #107

I love how much is possible with the Observable framework and support for libraries like d3.js. However many data apps cannot precompute their outputs. For example, a pipeline that extracts text from documents based on what a user queries, cannot precompute the results and any visualizations must be updated every time. The best hack to accomplish this seems to be rebuilding the app on each update. Or is there another…

I took the query only what you need approach with PulseUI: https://www.timestored.com/pulse/ It emphasises not sending all data to the client. For real-time and very large data sets this is the only option.
Post reply on HN