Viewing profile — shalabhc

> meaning being able to rename fields freely, is a must. avro supports field renames though. 3. on second thought i believe you'd only have to deploy when you choose. the next buil…

Did you look at other formats like Avro, Ion etc? Some feedback: 1. Dense json Interesting idea. You can also just keep the compact binary if you just tag each payload with a schem…

For all interested in this topic, I highly recommend the book Designing Data Intensive Applications https://www.goodreads.com/book/show/23463279-designing-data-... . It goes into n…

+1 What could be useful here is if postgres provided a way to determine the latest frozen uuid. This could be a few ms behind the last committed uuid but should guarantee that no n…

We have been using pex to deploy code to running docker containers in ECS to avoid the cold start delay. Cuts down the iteration loop time for development significantly. https://da…

The question is whether the concept of data is essential to how we structure computation. Computation is a physical process and any model we use to build or describe this process i…

I would suggest the author also look at Amazon Ion: * It can be used as schema-less * allows attaching metadata tags to values (which can serve as type hints[1]), and * encodes blo…

While well known for this paper and "information theory", Shannon's master's thesis* is worth checking out as well. It demonstrated some equivalence between electrical circuits and…

> Instead, we can check whether any of the writes between the begin timestamp and commit timestamp overlap with our transaction’s read set. Do you handle the case where the actual …

"Lots of Little Databases" reminded me of https://www.actordb.com/ which does lots of server-side sqlite instances, but the project now looks defunct.

Global consistency is expensive, both latency-wise and cost-wise. In reality most apps don't need global serializability across all objects. For instance, you probably don't need s…

Check out jwz's blog posts, eg https://www.jwz.org/blog/2023/01/mozilla-orgs-25th-anniversa... You may find other interesting articles linked from here: https://en.wikipedia.org/wi…

I'd be curious is Cython was evaluated as an alternative. With less of an impedance mismatch with Python and capable of similar speedups it might be a good fit for this use case.

Is EKS safe for multi-tenant use? When we looked it appeared unsafe if we want to run our users code next to each other because of possible isolation issues.

I agree with your sentiment. K8s may have some more controls for the "incremental deployment" case but I'm less confident about the isolation between pods to run user provided code…

(author here) There are two sides to the problem: the build and the deploy. > it feels like just having a build machine with storage would immediately solve the problem, there woul…

Thanks for the interesting links - I'll check them out! We would need not just another CI but also another container platform because launching a docker container is also slow. Irr…

Getting the message to the right container is one bottleneck. Currently this is routed through a couple of hops and includes some polling. This could all be optimized (if we had a …

(author here) Firecracker is definitely very interesting. Would require more ops work for us to run bare metal EC2 (we currently use Fargate). IIUC reusing pre-existing environment…

dagster.io is an open source Python library for building data pipelines. When using Dagster your project may depend on other Python data analysis and warehouse libraries like panda…

(author here) I agree it would be fantastic to have sub second deploys! Doing this for user specified Python environments is challenging in different ways than doing it for a JS SD…

(author here) You can already do local development with dagster, if you set up a local environment. Some users may not want to set up a local environment with all dependencies, sec…

https://shalabh.com It's a pelican based static site. More details of the tech used is here: https://shalabh.com/pages/site-tech.html

To add a specific example. A common pattern is 'put some objects in memcache' in front of the DB. Imagine if you can: 1. define a view representing the object fields you want to ca…