Viewing profile — psfried

Yes, and this is an important point! This is the reason for our current approach for sqlite derivations. You can absolutely just store all the data in the sqlite database, as long …

I agree completely! We've always talked about this, but we haven't really seen a clear way to package it into a good developer UX. We've got some ideas, though, so maybe one day we…

The main benefit isn't necessarily that it's _streaming_ per se, but that it's _incremental_. We typically see people start by just incrementally materializing their data to a dest…

To my knowledge, nobody's implemented parquet fragment files. But it supports compression of JSONL out of the box. JSON compresses very well, and compression ratios approaching 10/…

I don't think it's correct to say that JSONL is any more vulnerable to invalid data than other message framings. There's literally no system out there that can fully protect you fr…

Newton at least became quite wealthy. Dunno about the others

In contrast to jebarker's comment, I actually think it's really interesting that a concept coming from game engine development actually seems quite applicable in some very differen…

I feel like "data products" was a great idea, but difficult to implement in practice. There's kind of a paradox where you need a platform in order to host your data products, but a…

Post explaining why we chose Typescript for realtime data tranformations in Flow, and how it enables end-to-end static type checking of streaming data pipelines.

> But it is binary, so can’t be viewed or edited with standard tools, which is a pain. I've heard this sentiment expressed multiple times before, and a minor quibble I have with it…

That's an interesting statistic about Julia. Makes me curious what the stats would be for Rust.

I get the gist of what you're saying, and broadly agree that seasoned programmers tend to develop a strong sense of professional humility. I have to say that I think the analogy go…

First let me say that I think Timely Dataflow and Materialize are both super cool. The two approaches are quite different, in part because they solve slightly different problems. O…

IANAE on Flink, especially when it comes to the internals. But I think that the decomposition of computations into distinct map and reduce functions seems to afford a bit more flex…

Apart from Google, which has a patent related to their 2004 paper, I don't know how much people are trying to "take credit" for map-reduce. I'm certainly not. But I do think the ap…

I agree with this. As soon as the MapReduce paper came out, people were criticizing it for a lack of novelty, claiming that so-and-so has been using these same techniques for years…

Versioning is indeed an issue, but that's the case for anything with long-lived state. Our current rely on JSON schemas, TypeScript, and built-in testing support to help ensure com…

Another possible explaination is simply that people have gotten better at designing programming languages. Or that newer languages are better adapted to solving the problems we now…