Really curious how much can be attributed to using an ORM.
Squeeze the hell out of the system you have
81–90 of 383 posts
Re: Squeeze the hell out of the system you have
#82Earlier quoted context omitted.
Single Table Design is the way forward here. I can highly recommend The DynamoDB Book [0] and anything (talks, blogs, etc) that Rick Houlihan has put out. In previous discussions the author shared a coupon code ("HACKERNEWS") that will take $20-$50 off the cost depending on the package you buy. It worked earlier this year for me when I bought the book. It was very helpful and I referred back to it a number of times.…
And if you don't want to spend money, you can get idea from this article: https://www.alexdebrie.com/posts/dynamodb-single-table/ Im really curious about real life performance on different databases, especially in situation where RAM is smaller than database size.
That article didn't appear to be suggesting single-table design was appropriate for general purpose RMDBSes (or any database other than DynamoDb).
Re: Squeeze the hell out of the system you have
#83I'll probably get down-voted for saying this (again), but a key way to squeeze unimaginable amounts of performance is to lean into stored procedures . Look, I get it, the devx sucks. And it feels proprietary, icky, COBOL-like experience. It means you have to dwell in the database. What are you, a db admin?! But I'm telling you, the payoff is worth it. (and also, if you ship it you own it so yes you're a db admin). My…
I can't disagree with the results, SPs can change your life. HOWEVER, they require significant discipline and regular audits. All the code for them needs to be in source control with a Process for deployment to the DB. You also need a test suite as part of the Process which runs against a staging server with a comparable configuration to prod. The SPs need to be regularly dumped and compared against what's in source…
[deleted]
Re: Squeeze the hell out of the system you have
#84The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
> Design your application's hot path to never use joins. Grab (uber of asia) did this religiously and it created a ton of friction within the company due to the way the teams were laid out. It always required one team to add some sort of API that another team could take advantage of. Since the first team was so busy always implementing their own features, it created roadblocks with other teams and everyone started po…
Hard to follow the link. How would you join two tables between teams that don't communicate?
Re: Squeeze the hell out of the system you have
#85> Split up the monolith into multiple interconnected services, each with its own data store that could be scaled on its own terms. Just to note: you don't have to split out all the possible microservices at this junction. You can ask, "what split would have the most impact?" In my case, we split out some timeseries data from Mongo into Cassandra. Cass's table structure was a much better fit — that dataset had a well…
Is interesting that the idea of micro services is throw like a obvious "solution".
Is not.
"Scale-up" MUST be the "obvious" solution. What is missed by many, and this article touch (despite saying that micro-services is a "solid" choice) is that "Scale-up" is "scale-out" without breaking the consistency of the DB.
Is a lot you can do to squeeze, and is rare you need to ignore join, data validations and other anti-patterns that are normally trow casual when problems of performance happens.
Re: Squeeze the hell out of the system you have
#86I wonder if moving the db on beefy dedicated hardware with ton of ram and nvme would solve the problem. Preferably physicaly connected to the web serveurs.
Cost: a fraction of the developper cost.
I see so many things done on the cloud that 10X their complexity because of it. Modern hardware in increadibly powerfull.
Re: Squeeze the hell out of the system you have
#87Earlier quoted context omitted.
Žižek has a followup to that quote: "What he forgot to add was the crucial fourth term: the "unknown knowns," the things we don't know that we know." I've found it's really critical during the project planning phase to get to not just where the boundaries of our knowledge are, but also where are the things we're either tacitly assuming or not even aware that we've assumed. An awful lot of postmortems I've been a part…
> An awful lot of postmortems I've been a part of have come down to "It didn't occur to us that could happen." Would that not be an unknown unknown?
Usually there's a tacit assumption of how the system works, how the users are using the system, or something else about the system or the environment that causes that - it's not that the answer wasn't known, it's that it was assumed to be something it wasn't and nobody realized that was an assumption and not a fact.
Re: Squeeze the hell out of the system you have
#88Ugh. I had a colleague that addressed any scaling problem by putting a cache in front of the DB. Praised for solving the immediate problem, but shouldered none of the costs. I admit in the face of finding Prod/market fit, you do the expedient thing, but damned if I'm not often at the receiving end of these sorts of decisions.
Interestingly, I often ask candidates about optimising a slow running db query and the majority of people jump to adding caching and very few ask if they can run an explain or see the indexes.
"I would make the slow query faster" seems too obvious an answer for an interview question.
Re: Squeeze the hell out of the system you have
#89The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
My hot take: always use a materialized view or a stored procedure. Hide the actual, physical tables from the Application's account! The application doesn't need to know how the data is physically stored in the database. They specify the logical view they need of the data. The DBAs create the materialized view/stored procedure that's needed to implement that logical view. Since the application is never directly access…
But for the saves the structure is visible?
Re: Squeeze the hell out of the system you have
#90Earlier quoted context omitted.
My hot take: always use a materialized view or a stored procedure. Hide the actual, physical tables from the Application's account! The application doesn't need to know how the data is physically stored in the database. They specify the logical view they need of the data. The DBAs create the materialized view/stored procedure that's needed to implement that logical view. Since the application is never directly access…
But for the saves the structure is visible?
You can update underlying data via a materialized view.