Viewing profile — ignoreusernames

> Heresy I suppose, but doesn't feel (to me) like it "specifically" has to be cons cells, as long as it's a "list" of some sort Yeah, "cons" is definitely a implementation detail. …

> Homoiconicity, as I understand, is that the code is structured data that is easy to programmatically modify I think a more accurate description is that lisp code is just cons cel…

If safe keeper exposes the changes to the tables somehow, a type2 scd is just a windowed lag over the primary key sorted by the timestamp

[dead]

Even when the layout is friendly to simd, auto vectorization can be finicky. As a programmer, it’s really annoying to be constantly inspecting compiler output to see if the code wa…

Yeah, especially a bloomfilter which has a pretty easy formula for its false positive rate.

Don’t you think that the provider of the LLM is also a dimension on these discussions about responsibility? We often talk about the tech itself (LLM driven development) but how we …

I think this argument only holds if you believe that LLMs are at a point where it can handle any combination of craziness that you throw at it. From my own experience working with …

Can you elaborate on this please? Do you mean that’s basically impossible for rust std to provide a default runtime that makes “everyone” (embedded on one end and web on the other)…

As of now I don’t think there’s an alternative. I’m not a Rust expert but the core issue to me is that “async” goes beyond just having a Futures scheduler. Async stuff usually need…

I may have missed something, but how does “sans-io” deal with CPU heavy code? For example, if there’s some heavy decoding/encoding required on the data? Does the event loop only dr…

Agree with the other commenters that the title is a bit too dramatic. The content was well written and got the point across. I still don’t have enough experience to have a strong o…

Another strategy to avoid redistribution is simply having a big enough number of partitions and assign ranges instead of single partitions. A bit more complex on the coordination s…

> The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea. I haven't though much about it, but I believe the ideal reference implem…

> Folks I know in the industry are not very happy with databricks Yeah, big companies globing up everything does not lead to a healthy ecosystem. Congrats on the founders for their…

I recommend anyone who works with databases to write a simple engine. It's a lot simpler than you may think and it's a great exercise. If using python, sqlglot ( https://github.com…

Same thing with columnar/vectorized execution. It has been known for a long time that's the "correct" way to process data for olap workflows, but only became "mainstream" in the la…

> Spark is "in-memory" in the sense that it isn't forced to spill results to disk between operations I see your point, but that's only true within a single stage. Any operator that…

just out of curiosity, why do you say that spark is "in-memory"? I see a lot people claiming that, including several that I've interviewed in the past few years but that's not very…

> Most Parquet files are bulk exports from various data analysis pipelines or databases, often appearing as full snapshots rather than incremental updates I'm not really familiar o…

From the announcement “As of now, we have mined 1,580 PySpark tests from the Spark codebase, among which 838 (53.0%) are successful on Sail. We have also mined 2,230 Spark SQL stat…

This is a fair argument that's often brought up but I never see actual raw data backing it up. Housing is fucked in several places around the world, including a bunch of countries …

Early Athena (managed prestodb by AWS) had a similar bug when measuring colunar file scans. If it touched the file, it considered the whole file instead of just the column chunks r…

Great series. If I'm not mistaken, there's an additional layer to the unreliable narrator part because the book is supposed to be a translation of that biography. So, when certain …

100% agree. mapReduce hype always seemed strange to me because it's basically the volcano paper from the 90s but with custom user defined operators instead of pre baked ones in a m…