Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

Post reply on HN
Print view View original

141–150 of 383 posts

Re: Squeeze the hell out of the system you have

#141

Post by michael1999 »

killjoywashere wrote: post #108
KnobbleMcKnees wrote: post #56

Earlier quoted context omitted.

That was Donald Rumsfeld!? I always assumed this came from some techie or agile guru given how much it's used as a concept in project planning.

As a military officer who was watching CNN live from inside an aircraft carrier (moored) when he said that, being in charge of anti-terrorism on the ship at the time, it was absolutely foundational to my approach to so many things after that. Here's the actual footage: https://www.youtube.com/watch?v=REWeBzGuzCc Rumsfeld was complicated, but there's no doubt he was very effective at leading the Department. I think mo…

I'll support you there. In any sensible reading of Nuremberg, they all deserve to hang from the neck until dead. But the central moral failure was Bush. Letting Cheney hijack the vp search, and then pairing him up with Rumsfeld was a bad move, and obviously bad at the time. Those two had established themselves as brilliant but paranoid kooks with their Team B fantasies in the 70s, and should never have been allowed free rein.

Re: Squeeze the hell out of the system you have

#142

Post by latchkey »

tedunangst wrote: post #84
latchkey wrote: post #31

Earlier quoted context omitted.

> Design your application's hot path to never use joins. Grab (uber of asia) did this religiously and it created a ton of friction within the company due to the way the teams were laid out. It always required one team to add some sort of API that another team could take advantage of. Since the first team was so busy always implementing their own features, it created roadblocks with other teams and everyone started po…

Hard to follow the link. How would you join two tables between teams that don't communicate?

You don’t, that’s the problem.

Re: Squeeze the hell out of the system you have

#143

Post by endisneigh »

feoren wrote: post #115
endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

There are "tall" applications and "wide" applications. Almost all advice you ever read about database design and optimization is for "tall" applications. Basically, it means that your application is only doing one single thing, and everything else is in service of that. Most of the big tech companies you can think of are tall. They have only a handful of really critical, driving concepts in their data model. Facebook…

I agree with your sentiment, but even enterprises work on multiple “tall” features.

If they didn’t then I’d change my advice to be simply multi tenant per customer and replicate into a column store for cross customer analytics.

What advice would you give for a “wide” application?

Re: Squeeze the hell out of the system you have

#144

Post by feoren »

pipe_connector wrote: post #135
feoren wrote: post #115

Earlier quoted context omitted.

There are "tall" applications and "wide" applications. Almost all advice you ever read about database design and optimization is for "tall" applications. Basically, it means that your application is only doing one single thing, and everything else is in service of that. Most of the big tech companies you can think of are tall. They have only a handful of really critical, driving concepts in their data model. Facebook…

I agree with the characterization of applications you've laid out and think everyone should consider whether they're working on a "tall" (most users use a narrow band of functionality) or a "wide" (most users use a mostly non-overlapping band of functionality) application. I also agree with your take that tall applications are generally easier to build engineering-wise. Where I disagree is that I think in general wid…

> I've worked on a ton of wide applications, and each of them eventually became loathed by users and really hard to design features for.

Yes, I agree that this is the fate of most. But I refuse to believe it's inevitable; rather, I think it comes from systemic flaws in our design thinking. Most of what we learn in a college database course, most of what we read online, most all ideas in this space, transfer poorly to "wide" design. People don't realize this because those approaches do work well for tall applications, and because they're regarded religiously. This is why I call them so much harder.

Re: Squeeze the hell out of the system you have

#145

Post by prmph »

sainez wrote: post #59
i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Great point about working on teams. For the vast majority of tasks, people are only marginally better or worse than each other. A few people with decent communication will outpace a "star" any day of the week. I try to remind myself of this fact when I'm frustrated with other people. A bit of humility and gratitude go a long way.

Hmmmm, I really don't think this is true all (or even most of) the time. It probably depends on the task at hand, but if leading small teams of all kinds has taught me anything, it's that I'd prefer a tiny team (or even one person) who is at least above average competence, and is reflective of the work they are doing, than several people of average or below-average competence.

It's eye opening how many people are outright lazy with thought, don't care about the joy of doing something well (apart from whatever extrinsic rewards are attached to the work). Many team members can actually produce negative value.

It seems that people who are really capable of (or care about) conscientious, original thought in problem solving and driving projects forward are few. Count yourself lucky if you get to manage one of these people, they can produce incredible value when well directed.

Re: Squeeze the hell out of the system you have

#146

Post by feoren »

xyzzy123 wrote: post #128
feoren wrote: post #115

Earlier quoted context omitted.

There are "tall" applications and "wide" applications. Almost all advice you ever read about database design and optimization is for "tall" applications. Basically, it means that your application is only doing one single thing, and everything else is in service of that. Most of the big tech companies you can think of are tall. They have only a handful of really critical, driving concepts in their data model. Facebook…

Thanks I think this is a really interesting way to look at things. What is the market for "wide" applications though? It seems like any particular business can only really support one or two of them, for some that will be SAP and for others it might be Salesforce (if they don't need much ERP), or (as you mentioned) some giant semi homebrewed Oracle thing. Usually there is a legacy system which is failing but still ru…

I think you're getting the idea -- both your points kinda highlight that this is something that companies want, but are not really getting.

As for the market, various sources have the "enterprise software market", whatever that means, at somewhere around $100 billion to $300 billion. We also see companies trying over and over to do this kind of thing. The demand is clearly there.

Certainly the mandate "help run the business" is a wide concern, and that's an OK working definition of "enterprise", and what most existing solutions are trying to do. There are hundreds of interconnected concerns, lots of things to coordinate, etc.

There are other wide concerns, though. Almost anything in engineering and science. Take, for example, the question "how can we reduce our greenhouse gas emissions?" which a lot of companies are asking (or being forced to ask). If you wanted to build a SAAS product for helping companies reduce their GHG, you've got a wide problem, because there are a thousand activities that can emit GHG, and any given company is going to be doing dozens of them at once. But each company is different. Each state and country thinks of things differently. You might not even have the same calculations state-to-state.

Hard problems in science and engineering are just naturally cross-disciplinary, meaning your system has to know a lot of things about a lot of subjects. There are just thousands of little complicating differences and factors. If you're trying to solve a problem like this, absolutely do not de-normalize your database.

Re: Squeeze the hell out of the system you have

#147

Post by justinlloyd »

Squeeze what you've got, as hard as you can, then realize after squeezing for a while that if you squeezed here, here, and also... here, by changing how you think about a problem, suddenly you've got a lot left to get.

I spent two or so months optimizing the crap out of a majestic monolith and went from under 2K RPS when the PM thought, and the team repeatedly reported, that everything had been squeezed as much as it could, then changing the hardware, which got us to less than 3200 RPS, then to 4K RPS after just a few days of tinkering, to 10K RPS with a bit more effort, to 40K RPS a week or so later. "Oh that's, enough, we don't need to go further." I then changed "quite a bit of stuff" which then jumped us to 2M+ RPS, and then a month later, a consistent 40M+ RPS with low latency on a single box and there is still some juice left in the box should we want to go a little harder.

Right now we're not even touching 5% of the capacity of what we can pull from, it was that much of a change, simply by changing how we think about the problems. Moving from the old server to the new server let us jump from around 1800 RPS to a hair over 3000 RPS. Adding more hardware didn't fix our underlying problems. Adding more complexity was just punting the problem down the road. But changing how to think about the problem? _That_ changed the problem. And changed our answer to the problem.

Re: Squeeze the hell out of the system you have

#148

Post by roughly »

waprin wrote: post #105
roughly wrote: post #76

Earlier quoted context omitted.

Žižek has a followup to that quote: "What he forgot to add was the crucial fourth term: the "unknown knowns," the things we don't know that we know." I've found it's really critical during the project planning phase to get to not just where the boundaries of our knowledge are, but also where are the things we're either tacitly assuming or not even aware that we've assumed. An awful lot of postmortems I've been a part…

I really enjoy the concept of unknown knowns, but I don’t agree with your example, which is an unknown unknown. To me the corporate version of the unknown known is when a a project is certainly doomed, for reasons everyone on the ground knows about, yet nobody wants to say anything and be the messenger that inevitably gets killed, as long as paycheck keeps clearing. An exec ten thousand feet from the ground sets a “v…

Yeah, you’re right. I was reaching for an example, but yours are much better and better capture the idea.

Re: Squeeze the hell out of the system you have

#149

Post by taylodl »

downWidOutaFite wrote: post #131
taylodl wrote: post #66

Earlier quoted context omitted.

My hot take: always use a materialized view or a stored procedure. Hide the actual, physical tables from the Application's account! The application doesn't need to know how the data is physically stored in the database. They specify the logical view they need of the data. The DBAs create the materialized view/stored procedure that's needed to implement that logical view. Since the application is never directly access…

This doesn't work because DBAs are rarely on the dev team's sprint schedule. If the DBAs are blocking them devs can and will figure out how to route around the gatekeepers. In general, keep the logic in the app not the db.

We have sprints. We also have super responsive DBAs. Keeping the logic in the app is the path to unresponsive database calls. Been there, done that. Not going back to that crap.

In all seriousness, I won't work for an organization that works the way you describe. It's a red flag and a sign of organizational issues, personality issues, and ineffective management. Don't need to waste my time at a place like that.

Re: Squeeze the hell out of the system you have

#150

Post by nerdponx »

bootsmann wrote: post #137
nerdponx wrote: post #127

Earlier quoted context omitted.

I'm working on something right now with the Python ORM SQLAlchemy. It turns out that getting it to use RETURNING with INSERT is not trivial and requires you to set the non-obvious option `expire_on_commit=False`, which doesn't guarantee use of RETURNING, but is supposed to use it if your db driver and database happen to support it and the ORM happens to support it for that particular combination of driver and databas…

There is certainly an API to inspect your query, you can just call print() on the object iirc.

The problem is with using `session.add(obj)` instead of `session.scalars(insert(TheClass).returning(TheClass), data)`. If there's a way to get generated SQL from an AsyncSession, please do let me know.
Post reply on HN