Viewing profile — willvarfar

Exciting! I have been on a similar odyssey making a 'zero copy' Java library that supports protobuf, parquet, thrift (compact) and (schema'd) json. It does allocate a long[] and br…

With a long time in the industry and seeing how so many big software companies work, this really really chimed with me. Many/most teams and projects and busy work are not actually …

Yeah it's pretty obscure, sorry. It's called cogroup in Spark and similar architectures. It does a group-by to convert data into the format (key_col_1, ... key_col_n) -> [(other_co…

This video is very liberal but does a good job of explaining which companies and industries pay for breaks and which don't. And uses soy bean farmers as a prominent example of a gr…

Can join cardinality can be tackled with cogroup and not expanding the rows until final write?

everyone is expecting everyone to actually go Gripen with Rolls Royce or MECA engines?

Having a pending order that can be cancelled is negotiation leverage?

And what about all those huge pending orders for F35 in ... Denmark and Canada? Etc.

The very last clip in the video says that it is kids in affluent families taking that direction.

(I work a lot with BigQuery's BigLake adaptor and it's basically caching the metadata of the iceberg manifest and parquet footers in Bigtable (this is Google) so query planning is …

Greenland and Denmark have always been encouraging minerals deals etc, they just haven't materialized.

Seriously, this is not what big data does today. Distributed query engines don't have the primitives to zip through two tables and treat them as column groups of the same wider log…

specifically I've discovered how to 'trick' mainstream cloud storage and mainstream query engines using mainstream table formats how to read parallel arrays that are stored outside…

crazy to think that soon not being able to successfully complete the captcha will be a signal that the user is human

I had a great euphoric epiphany feeling today. Doesn't come along too often, will celebrate with a nice glass of wine :) Am doing data engineering for some big data (yeah, big enou…

I agree that social media is a net negative, but want to also point out that before social media it was the mainstream press and TV have been shaping society for decades. Things li…

I've always noticed and wondered, so I guess it's easy to overlook but it's there.

Presumably there will soon be banner ads in Claude Code then?

I remember when syntax highlighting was introduced in Borland's Turbo Pascal editor (on DOS). It was a very major usability improvement and put TP's IDE at the forefront of getting…

Kudos, neat digging and writeup that makes us think :) If you merge linear probed tables by iterating in sorted hash order then you are matching the storage order and can congest p…

This is going to be an interesting take but I think it is plausible that we'll see a quiet growth in American tech companies having even bigger offshore campuses instead. Google Zu…

A really clear explanation! So if I were running a provider I would be caching popular prefixes for questions across all users. There must be so many questions that start 'what is'…

absolutely, I almost missed it too. Took me a while to work out!