I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…
I like a similar quote from Steven Pressfield: “The athlete knows the day will never come when he wakes up pain-free. He has to play hurt.” This applies to ourselves more than our systems though.
Squeeze the hell out of the system you have
151–160 of 383 posts
Re: Squeeze the hell out of the system you have
#152The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
There are "tall" applications and "wide" applications. Almost all advice you ever read about database design and optimization is for "tall" applications. Basically, it means that your application is only doing one single thing, and everything else is in service of that. Most of the big tech companies you can think of are tall. They have only a handful of really critical, driving concepts in their data model. Facebook…
- Sellers (Amazon, third party, retail store) - Inventory (forecasting, recommendation) - Customers (comments, ratings, returns, preferences) - Warehouses (5+ distinct types, filled with custom machines) - Transit Options (long haul, air, vans, cars, bikes, walking, boats) - Delivery Partners (DSP, Flex, Fedex, UPS - forecasting capacity here) - Routing (between warehouses, within warehouses, to specific homes) - skipping AWS - skipping billing - skipping advertising on amazon.com (bidding, attribution, etc)
There's optimizations and metrics collected and packages transition between all these layers. There's hundreds of "neat projects" running to special case different things; all them useful but adding complexity.
For example ordering prescriptions off Amazon pharmacy needs effectively its own website and permissions and integrations. Probably distinct sorting machines with supporting databases for them. Do you need to log repairs on those machines? Probably another table schema.
You want to normalize international addresses? And fine tune based on delivery status and logs and customer complaints and map data? Believe it not like 20 more tables. Oh this country has no clear addresses? Need to send experienced drivers to areas they already know. Need to track that in more tables.
Re: Squeeze the hell out of the system you have
#153There's definitely a subset of the industry which seems to really love complexity, and I suspect a large part of that is caused by the incentives involved and the need for "growth" and justification for one's continued employment.
Re: Squeeze the hell out of the system you have
#154Earlier quoted context omitted.
Rumsfeld's got some great quotes, most of which were delivered in the context of explaining how the Iraq war turned into such a clusterfuck, and boy could that whole situation have used the kind of leadership Donald Rumsfeld's quotes would lead you to believe the man could've provided.
If I remember it correctly (it was a long time ago), he never fully supported the war. It didn't take a genius to notice that the goals set by the presidency were (literally) impossible and not the kind of thing you do achieve a war. But whatever position he had, Iraq turning into a clusterfuck wasn't a sign of bad leadership by his part. It was a sign of bad ethics, but not leadership. His options were all of gettin…
> In the first emergency meeting of the National Security Council on the day of the attacks, Rumsfeld asked, "Why shouldn't we go against Iraq, not just al-Qaeda?" with his deputy Paul Wolfowitz adding that Iraq was a "brittle, oppressive regime that might break easily—it was doable," and, according to John Kampfner, "from that moment on, he and Wolfowitz used every available opportunity to press the case."
https://en.wikipedia.org/wiki/Donald_Rumsfeld#Military_decis...
Re: Squeeze the hell out of the system you have
#155Earlier quoted context omitted.
100%, it makes it easy for bad programmers to write bad performing queries, but you can easily write performant code. Btw that’s a feature - letting people ramp up to full db knowledge is beneficial, you don’t want to be spending your time writing performant queries before you need to.
> it makes it easy for bad programmers to write bad performing queries That is true of every ORM in existence. The easiest thing to do is naively follow the object graph in code, because that's what the ORM gives you. If the ORM was to somehow add friction here to encourage some other approach it would be panned as "too hard!!1" and fade away into obscurity.
Re: Squeeze the hell out of the system you have
#1561. Efficiency, which I define as minimising losses i.e not writing things inefficiently, avoiding artificial complexity, bloat and keeping code simple. Performance hits here can also be a 1000 cuts problem when depending on many 3rd party pieces while having people chanting "premature optimisation" at you.
2. Optimization, which I define as employing specialist algorithms (which sometimes come in the form of entire tech stacks these days) with the cost of added complexity (and potentially performance trade) to get performance beyond the basic or naive yet efficiently implemented methods. The cost benefit ratio to these is not always worth it, especially in the beginning.
Hopefully the point I'm trying to make should be obvious, that attempting #2 before #1 is a bad idea, and in less explicit words I suspect this is kind of what the author is getting at... Yet it's not all that uncommon to see someone trying to fit a turbocharger to a cheese skateboard with 64 triangular wheels.
Re: Squeeze the hell out of the system you have
#157I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…
Rumsfeld's got some great quotes, most of which were delivered in the context of explaining how the Iraq war turned into such a clusterfuck, and boy could that whole situation have used the kind of leadership Donald Rumsfeld's quotes would lead you to believe the man could've provided.
> Q: Could I follow up, Mr. Secretary, on what you just said, please? In regard to Iraq weapons of mass destruction and terrorists, is there any evidence to indicate that Iraq has attempted to or is willing to supply terrorists with weapons of mass destruction? Because there are reports that there is no evidence of a direct link between Baghdad and some of these terrorist organizations.
> Rumsfeld: Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.
> And so people who have the omniscience that they can say with high certainty that something has not happened or is not being tried, have capabilities that are ...
https://archive.ph/20180320091111/http://archive.defense.gov...
Re: Squeeze the hell out of the system you have
#158YAGNI? But you could say that about a lot of things and this also gives you other options like releasing to a subset of customers (rolling release system like LTS etc.)
I work somewhere that has done this but for other driving reasons but it is a scalability dream. It is the web equivalent of desktop software running in Citrix! I haven’t seen anyone tune SQL in 4 years there whereas it has been a regular pastime everywhere else!
Yes you can’t do this for social networks but you can do it for most “customer with isolated clients” type shops which is most companies.
Re: Squeeze the hell out of the system you have
#159The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
But joins should never impact performance in a large way if they're on the same server and properly indexed. "It's truly amazing how much faster everything is when you eliminate joins" is just not true if you're using joins correctly. Sadly, many developers simply never bother to learn.
On the other hand, having to write a piece of data to 20 different spots instead of 1 is going to be dramatically slower performance-wise, not to mention make your queries tremendously more complex and bug-prone, when you remember to update a value in 18 spots but forget about 2 of them.
You mention cheap storage as an advantage for denormalizing, but storage is the least problem here. It's a vastly larger surface area for bugs, and terrible write performance (that can easily chew up read performance).
Re: Squeeze the hell out of the system you have
#160The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
There are "tall" applications and "wide" applications. Almost all advice you ever read about database design and optimization is for "tall" applications. Basically, it means that your application is only doing one single thing, and everything else is in service of that. Most of the big tech companies you can think of are tall. They have only a handful of really critical, driving concepts in their data model. Facebook…