Viewing profile — squarecog

Here are a couple of talks about orchestration and choreography, workflows vs sagas, etc. Both very technical, both with very little "sell" despite being given by a Camunda co-foun…

LLMs are Foundational Models, but not all Foundational Models are LLMs. If you are interested in this space, it does probably make sense to look up what Stable Diffusion (Stability…

LinkedDataHub, a "RDF-native notebook", is not to be confused with LinkedIn DataHub, which is a metadata store/crawler/ui for your data systems: https://datahubproject.io/ .

I should've added, there's an obvious example for the "SaaS control plane" separation, which is equivalent: "stop processing job X that is destabilizing the cluster" should be proc…

These terms usually show up in the context of networking protocols. Cloudflare has a very quick explainer: https://www.cloudflare.com/learning/network-layer/what-is-th... . To make…

Now? No idea. At the time I left in 2016, all Storm usage was replaced by Heron, that was the point of making it API-compatible.

I was in charge of the Twitter data platform team at the time we developed Heron and deprecated Storm. The Mesos component of your retelling is not quite right. Take a look at this…

Donated to Apache in 2018. https://blog.twitter.com/engineering/en_us/topics/open-sourc... (and open-sourced in 2016, having started development in 2014).

Why is this odd? If he started two years later and there was not a trace of the Prize work at the company, that would be an indicator that the competition was not important. If he …

> So basically all research projects have to redirect funds to cover part of the "open access fees" Yes, the cost of publishing results, as well as review, editing, etc, should be …

While HDFS is indeed used for exporting old data and storing some partition mapping metadata, it's clear from the blog post that MetricsDB is much more reliant on BlobStore as well…

I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is …

Can you explain which part, in your opinion, is facepalm worthy? In your example conversation, the developer starts off by demonstrating that they are not good at thinking through …

I helped build Twitter's data platform, 2010-2016. There isn't an "analysis server" and analyzing user activity is not done on a "user database backup" at Twitter's scale, though i…

(ex-twitter engineer) I left before this project got started, and do not have any insider info on how they did it. Given what I know about the number of places tweet length assumpt…

hdf5, Feather, Arrow, protobufs, json, xml -- all solve the problem of binary representation of data on disk. They all leave the question of how to map said data to a specific prob…

Stopping when you hit 95% confidence is a classic failure mode. Yes, if you are doing classic t-test based A/B testing, you have to wait until a pre-determined threshold; otherwise…

You may be interested in what Wes McKinney has to say on the topic: http://wesmckinney.com/blog/outlook-for-2017/ (you can read/write using pyarrow)

Yes: https://twitter.com/aneeshs/status/773547228694589440 "We use GraphJet for serving contextual, personalized content recommendations in real time, such as "You May Like", email…