41–50 of 94 posts
Some a prime example of a service that naturally peaks at round hours. We have a habbit of never scheduling long running processes at round hours. Usually because they tend to be busier. https://hakibenita.com/sql-tricks-application-dba#dont-sched...
> Most meetings start on the hour, some on the half, but most on the full. It sounds obvious to say it aloud, but the implication of this has rippled through our entire media processing infrastructure.
When you can control when it happens, you can often jitter things. For instance the naive approach of rate limiting users down to quantized times (eg: the minute, the hour, etc.) leads to every client coming back at the same time. The solution there is to apply a stable jitter so different clients get different resets.
That pattern does not go all that well with meetings as they need to happen when they happen, which is going to be mostly the hour and 30 minutes etc. However often the lead up time to those meetings is quite long, so you can do the work needed that should happen on the hour, quite a bit ahead of time and then apply the changes in one large batch on the minute.
You have similar problems quite often with things like weekly update emails. At scale it can take you a lot of time to prepare all the updates, often more than 12 hours. But you don't want the mails to come in at different times of the day so you really need to get the reports prepared and then send them out when ready.
> sudo echo $NUM_PAGES > /proc/sys/vm/nr_hugepages This won't work :) echo will run as root but the redirection is still running as the unprivileged user. Needs to be run from a privileged shell or by doing something like sudo sh -c "echo $NUM_PAGES > /proc/sys/vm/nr_hugepages" The point gets across, though, technicality notwithstanding.
Or echo $NUM_PAGES | sudo tee /proc/sys/vm/nr_hugepages I've always found it odd that there isn't a standard command to write stdin to a file that doesn't also write it to stdout. Or that tee doesn't have an option to supress writing to stdout.
If you happen to have moreutils installed, you can do that with pee
echo $NUM_PAGES | sudo pee 'cat > /proc/sys/vm/nr_hugepages'Note that they were running Postgres on a 32 CPU box with 256GB of ram. I'm actually surprised that it handled that many connections. The data implies that they have 4000 new connections/sec...but is it 4000 connections handled/sec?
It’s likely an actual CPU would’ve handled this load just fine.
I'm not working at this company but I found that these types of problems can often be simplified in the architecture. > Most meetings start on the hour, some on the half, but most on the full. It sounds obvious to say it aloud, but the implication of this has rippled through our entire media processing infrastructure. When you can control when it happens, you can often jitter things. For instance the naive approach o…
Can’t believe they needed this investigation to realize they need a connection pooler. It’s a fundamental component of every large-scale Postgres deployment, especially for serverless environments.
can't believe postgres still uses a process-per-connection model that leads to endless problems like this one.
Earlier quoted context omitted.
Or echo $NUM_PAGES | sudo tee /proc/sys/vm/nr_hugepages I've always found it odd that there isn't a standard command to write stdin to a file that doesn't also write it to stdout. Or that tee doesn't have an option to supress writing to stdout.
> I've always found it odd that there isn't a standard command to write stdin to a file that doesn't also write it to stdout If you happen to have moreutils installed, you can do that with pee echo $NUM_PAGES | sudo pee 'cat > /proc/sys/vm/nr_hugepages'
Can’t believe they needed this investigation to realize they need a connection pooler. It’s a fundamental component of every large-scale Postgres deployment, especially for serverless environments.
Some a prime example of a service that naturally peaks at round hours. We have a habbit of never scheduling long running processes at round hours. Usually because they tend to be busier. https://hakibenita.com/sql-tricks-application-dba#dont-sched...
I wish more applications would adopt the "H" option that Jenkins uses in it's cron notation - essentially it is a randomiser, based on some sort of deterministic hashing function. So you say you want this job to run hourly and it will always run at the same minute past the hour, but you don't know (or care) what that minute that is. Designed to prevent the thundering herd problem with scheduled work.
[1] https://github.com/puppetlabs/puppet/blob/main/lib/puppet/pa...