Job offers require experience in technologies that you won't ever need building solo project. I'm not surprised when those big scale technologies get shoehorned into small project for the sake of learning, showcasing "look I know that one" etc.
Only totally missing this point could explain why someone would make this hyperbole rant page
What the hell have you built
131–140 of 239 posts
Re: What the hell have you built
#132Earlier quoted context omitted.
This comment makes this thread a great time capsule. Given that the website is now over 10 years old, it perfectly illustrates how much 'best practices' and architectural complexity (and cloud bills) have changed since then.
No no, don't give me this. I was there before 10 years ago. I remember the pain in the ass that was hosting your own web server and own hardware, dealing with networking issues with cisco switches and thinking about getting a ccna. I remember the days of trying to figure out php and ranodm ass modules or how python and wsgi fit together on a slow ass windows machine instead of just spinning up an app and doing networ…
I know people who can wrangle k8s and set up rules in whatever it's called to spin up and down the whole kaboodle of services effortlessly. It's like they know a whole level of programming I'm not familiar with, at all. I know the dotnet stuff pretty well, after many years fiddling with it. I do stuff in dotnet now I didn't have even terminology to talk about before. What they do in k8s and friends reminds me of that.
I personally don't care for it and if I design something I make it so it avoids that stuff if I can at all help it. But I've come to see that it can have real value.
The thing is though, that then you really need someone to be that very competent ops person. If you're a grug like me, you don't get many shots to be good at something. I probably don't have the years in me to be good at both ops and "pure" programming.
So if you are a startup and you're not some kind of not only very smart but smart, fast and with taste, maybe pick your battles.
If you are great at the ops side, ok, maybe design it from that perspective and hire a bunch of not-made-of-unobtainium regular middle-of-the-road coders to fill in what the microservices and stuff should contain and manage those. This requires money for a regular hiring budget. (Or you are supersmart and productive and "play pretend enterprise" with all roles yourself. But I have never seen such a person.)
Or focus on a tight design which can run without any of that, if you come more from the "I'm making a single program" part of the world.
Tinkering syndrome can strike in any kind of design, so you need personal maturity whatever path you choose.
Re: What the hell have you built
#133Earlier quoted context omitted.
I personally wouldn't like to put caching in Postgres, even though it would work at lower scales. But at that scale I don't really need caching anyway. Having the ephemeral data in a different system is more appealing to me as well. The caching abstractions your frameworks have are also likely designed with something like Redis in mind and work with it out of the box. And often you can just start with an in-memory ca…
>I personally wouldn't like to put caching in Postgres, even though it would work at lower scales. Probably should stop after this line - that was the point of the article. It will work at lower scales. Optimize later when you actually know what to optimize.
My point is more that at that scale I'd try to avoid caching entirely. Unless you're doing analytical queries over large tables, Postgres is plenty fast without caching if you're not doing anything stupid.
Re: What the hell have you built
#134"Maybe Redis for caching". Really that's going way too far - you do NOT need Redis for caching. Just put it in Postgres. Why go to this much trouble to put people in their place for over engineering then concede "maybe Redis for caching" when this is absolutely something you can do in Postgres. The author clearly cannot stop their own inner desire for overengineering.
just use the filesystem, it's superfast and reliable.
Re: What the hell have you built
#135"Maybe Redis for caching". Really that's going way too far - you do NOT need Redis for caching. Just put it in Postgres. Why go to this much trouble to put people in their place for over engineering then concede "maybe Redis for caching" when this is absolutely something you can do in Postgres. The author clearly cannot stop their own inner desire for overengineering.
That seemed odd to me too, they're talking about single server, which to me would mean running postgres on the application server itself.
In that scenario, the last thing you need is another layer between application and database.
Even in a distributed environment, you can scale pretty far with direct-to-database as you say.
Re: What the hell have you built
#136"Maybe Redis for caching". Really that's going way too far - you do NOT need Redis for caching. Just put it in Postgres. Why go to this much trouble to put people in their place for over engineering then concede "maybe Redis for caching" when this is absolutely something you can do in Postgres. The author clearly cannot stop their own inner desire for overengineering.
The sentiment here is right, but redis does make a difference at scale. I built a web app this year on AWS lambda that had up to 1000/requests/second and at that scale, you can have trouble with Postgres, but redis handles it like it’s nothing. I think that redis is a reasonable exception to the rule of ”don’t complicate things” because it’s so simple. Even if you have never used it before, it takes a few minutes to…
Postgres itself has no issue with 1000 simple requests per second. On normal notebook hardware you'll get easily several thousand requests per second if you're just looking up small amounts of data by the primary key. And that is without any optimization and with non-DB overhead included. The actual performance you can get is probably quite a bit higher, but I've seen ~4-6k requests per second on naive, unoptimized endpoints that just look up some data in Postgres.
Re: What the hell have you built
#137Earlier quoted context omitted.
I think that's a slightly different set of things to what OP is complaining about though. They're much more reasonable, but also "outside" of the application. Having secret management or CI (pretty much mandatory!) does not dictate the architecture of the application at all. (except the caching layer. Remember the three hard problems of computer science, of which cache invalidation is one.) Still hoping for a good "s…
Cache invalidation is replacing one logical thing with a new version of the same logical thing. So technically that’s also naming things. Doubly so when you put them in a kv store.
That angle seems potentially insightful, and I'm going to have to think about it, but to me, cache invalidation seems more like replacing one logical thing with nothing. It may or may not get replaced with a new version of the same logical thing later if that's required.
Re: What the hell have you built
#138It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but organisational issues: You want straightforward, self-contained deployments for one, instead of uploading files onto your single server. If the process crashes or your harddisk dies, you want redundancy so even those twelve customers can still access the appli…
>It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but organisational issues: You want straightforward, self-contained deployments for one, instead of uploading files onto your single server ... You can get all that with a monolith server and a Postgres backend.
In this job market, how am I supposed to get hired without the latest buzzwords on my resume? I can’t just have monolithic server and Postgres!
(Sarcasm)
Re: What the hell have you built
#139It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but organisational issues: You want straightforward, self-contained deployments for one, instead of uploading files onto your single server. If the process crashes or your harddisk dies, you want redundancy so even those twelve customers can still access the appli…
I'm not sure why your architecture needs to be complex to support CI pipelines and proper workflow for change management. And some of these guidelines have grown into satus quo common recipes. Take your starting database for example, the guideline is always "sqlite only for testing, but for production you want Postgres" - it's misleading and absolutely unnecessary. These defaults have also become embedded into PaaS s…
I've been involved in bootstrapping the infrastructure for several companies. You always start small, and add more components over time. I dare say, on the projects I was involved, we were fairly successful in balancing complexity, but some things really just make sense. Using a container orchestration tool spares you from tending to actual Linux servers, for example, that need updates and firewalls and IP addresses and managing SSH keys properly. The complexity is still there, but it shifts somewhere else. Looking at the big picture, that might mean your knowledge requirements ease on the systems administration stuff, and tighten on the cloud provider/IaC end; that might be a good trade off if you're working with a team of younger software engineers that don't have a strong Linux background, for example, which I assume is pretty common these days.
Or, consider redundancy: Your customers likely expect your service to not have an outage. That's a simple requirement, but very hard to get right, especially if you're using a single server that provides your application. Just introducing multiple copies of the app running in parallel comes with changes required in the app (you can't assume replica #1 will handle the first and second request—except if you jump through sticky session hoops, which is a rabbit hole on its own), in your networking (HTTP requests to the domain must be sent to multiple destinations), and your deployment process (artefacts must go to multiple places, restarts need to be choreographed).
Many teams (in my experience) that have a disdain for complex solutions will choose their own, bespoke way of solving these issues one by one, only to end up in a corner of their own making.
I guess what I'm saying is pretty mundane actually—solve the right problem at the right time, but no later.
Re: What the hell have you built
#140Or build your microservices as a monolith using a “local” async service mesh (no libs or boilerplate needed, its just an async interface for each service) and service namespaced tables in your DB, then just swap in a distributed transport on a per-case basis if you ever need to scale.