Ok this was a nice read. Very expressive and I agreed with the majority of the points. I'm not going to lie, I expected "and this is why we built ", but it never came. Which leads me to my question, and forgive me for my impatience, but _what_ are you selling? Is it a platform? Ide? Books? So many words and I ended with more questions than I started. On any who, good read indeed.
> I'm not going to lie, I expected "and this is why we built ", but it never came. Uh... > Implementing it is more than I can do alone, which is why my cofounders, Daniel Mills and Skylar Cook, and I are starting Cambra. We are developing a new kind of programming system that rethinks the traditional internet software stack on the basis of a new model.
Composition shouldn't be this hard
21–30 of 78 posts
Re: Composition shouldn't be this hard
#22 There are many examples of models that enable coherent systems within specific domains:
- Type systems in programming languages catch many logic errors and interface misuses
- The relational model in databases enables programmers to access incredible scale and performance with minimal effort.
[...]
So coherent systems are great: everyone should just buy into whatever model will most effectively do the job. Right? Unfortunately, the listed models are all domain-specific–they don’t generalize to other contexts. And most modern internet software is not domain-specific. Modern applications typically span a wide variety of domains, including web and API serving, transaction processing, background processing, analytical processing, and telemetry. That means that trying to keep a system coherent limits what that system can ultimately do. As one implements more capabilities, application requirements push us outside of a single domain, forcing us to reach for components with a different internal model. So, bit by bit, our system fragments.
The problem of course is that type systems and databases are not meaningfully "domain-specific." They aren't technical magic bullets but they separately provide real value for the use cases of "web and API serving, transaction processing, background processing, analytical processing, and telemetry." So then why hasn't the industry settled on a specific type system? Why do database vendors (and the SQL standard) keep breaking the relational model in favor of something ad hoc and irritating?I believe the real problem is that software is symbolic and the problems it solves usually aren't. Writing an application means committing to a certain set of symbolic axioms and derivation schemas, and these are never going to encapsulate the complexity of the real world. This relates to Greenspun's 10th rule:
Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
Or in a modern context, C++/C# and managing a huge amount of configuration data with a janky JSON/XML parser, often gussied up as an "entity component system" in game development, or a "DSL" in enterprise. The entirely equivalent alternative is a huge amount of (deterministic!) compile-time code generation. Any specific symbolic system small enough to be useful to humans is eventually going to go "out of sync" with the real world. The authors hint at this with the discrepancy between SQL's type system and that of most programming languages, but this is a historical artifact. The real problem is that language designers make different tradeoffs when designing their type system, and I believe this tradeoff is essentially fundamental. Lisp is a dynamically-typed s-expression parser and Lisp programs benefit from being able to quickly and easily deal with an arbitrary tree of whatever objects. In C#/C++ you would either have to do some painful generics boilerplate (likely codegen with C#) or box everything as System.Object / void pointer and actually lose some of the type safety that Lisp provides. OTOH Idris and Lean can do heterogeneous lists and trees a little more easily, but that cost is badly paid for in compilation times, and AFAICT it'll still demand irritating "mother may I?" boilerplate to please the typechecker. There is a fundamental tradeoff that seems innate to the idea of communicating with relatively short strings of relatively few symbols.This sounds like Godel incompleteness, and it's a related idea. But this has more to do with cognition and linguistics. I wish I was able to write a little more coherently about this... I guess I should collect some references and put together a blog at some point.
Re: Composition shouldn't be this hard
#23We lucked into filesystems that have open structures (even if the data is opaque). Perhaps we should be pushing for "in-memory filesystems" as a default way of storing runtime data, for example.
Re: Composition shouldn't be this hard
#24This resonates a lot if you’ve ever worked in system integration. In practice, most of the complexity comes exactly from what’s described here: every system has a rich internal model, but the moment data crosses a boundary, everything degrades into strings, schemas, and implicit contracts. You end up rebuilding semantics over and over again (validation, mapping, enrichment), and a lot of failures only show up at runt…
I think die-hard fans of static typing mostly fail to acknowledge this objective reality and its implications. Every time they encounter this problem again and again, they approach it as if nobody thought of this before, and didn’t develop reliable abstractions to productively work in these environments.
Re: Composition shouldn't be this hard
#25I've also felt that composition is unnecessarily hard in web & data infrastructure. I do game development as a hobby, and web + data technology is in the stone age compared to game engines when it comes to composability. For example, say I develop some object (scene) in Godot Engine. It interacts with the environment using physics simulation, renders 3D graphics to the screen with some shaders and textures, and plays…
It is kind of broken now, much thanks to using web applications (and applications that are basically just wrappers for web applications), but I don't know I if want to go back.
On one side it was much easier when I could hack together a program that was good enough (since everything was the same bland grey).
On the other hand some programs certainly looks nicer today.
And it has become easier to compose logic with solutions like Maven, Nuget and the various frontend package managers.
But yes, we lost drag and drop UI development, we lost consistency and we lost a lot of UX (at least temporarily).
Re: Composition shouldn't be this hard
#26Re: Composition shouldn't be this hard
#27Obviously a lot of this you can piece together today, in fact Snowflake itself does a lot of it. But the other part of the article makes me think they understand the even harder part of the problem in modern enterprises, which is that nobody has a clear view of the model they're operating under, and how it interacts with parts of the business. It takes insane foresight and discipline to keep these things coherent, and the moment you are trying to integrate new acquisitions with different models you're in a world of pain. If you can create a layer to make all of this explicit - the models, the responsibilities, the interactions, and the incompatibilities that may already exist, then mediate the chaos with some sort of AI handholding layer (because domain experts and disciplined engineers aren't always going to be around to resolve ambiguities), then you can solve both a huge technical problem but a much more complicated ecological one.
Anyway, whatever they're working on, I think this is the exact area you should focus on if you want to transform modern enterprise data stacks. Throwing AI at existing heterogenous systems and complex tech stacks might work, but building from scratch on a system that enforces cohesion while maintaining agility feels like it's going to win out in the end. Excited to see what they come up with!
Re: Composition shouldn't be this hard
#28I used to believe this, but after working at a successful SaaS I have come to believe that correctness and unambiguity are not entirely necessary for successful software products.
It was a very sad realization that systems can be flaky if there is enough support people to solve edge case problems. That features can be delivered while breaking other features as long as enough users don't run into the middle of that venn diagram, etc.
Fact is it always comes down to economics, your software can afford to be as broken and unpredictable as your users will still be willing to pay money for it.
Re: Composition shouldn't be this hard
#29Ok this was a nice read. Very expressive and I agreed with the majority of the points. I'm not going to lie, I expected "and this is why we built ", but it never came. Which leads me to my question, and forgive me for my impatience, but _what_ are you selling? Is it a platform? Ide? Books? So many words and I ended with more questions than I started. On any who, good read indeed.
> I'm not going to lie, I expected "and this is why we built ", but it never came. Uh... > Implementing it is more than I can do alone, which is why my cofounders, Daniel Mills and Skylar Cook, and I are starting Cambra. We are developing a new kind of programming system that rethinks the traditional internet software stack on the basis of a new model.
Re: Composition shouldn't be this hard
#30I think anything that can change this has to be simple enough that it'd be more effective to just explain the system and implement it, than wax about the general outline of part of the problem. Especially since the real target audience for an initial release by necessity needs to understand it.
There are some big leaps we could make with having code be more flat. Things like having the frontend and backend handler in the same file under the same compiler/type checker. But somebody will want to interact with a system outside of the 'known-world' and then you're writing bindings and https://xkcd.com/927/
At the end of the day I think the core tension is that once the speed of light is noticeable to your usecase things become distributed, which creates the desire for separate rate-of-change. I'm not sure what would 'solve' that.
AI will be a plus, for the fact that a single team can be in charge of more of the parts leading to a more coherent whole.
Hope OP builds some nice tools, but I've seen too many of these attempts fail to get excited about "i think we found it".