221–230 of 479 posts
People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…
People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…
A lot of that is due to absolutely lousy code. We had a system management backend at my last company. Loading the users list was unbearably slow; 10+ seconds on a warm cache. Not too terrible, except that most user management tasks required a page reload, so it was just wildly infuriating. Eventually I took a look at the code for the page, which queried LDAP for user data and the database for permissions data. It did…
$6 VPS can handle 500,000 requests daily
On this server, I have PHP-fpm workers, nginx and MariaDB
The average CPU usage about 30%, load average is about 0.5
Earlier quoted context omitted.
My $5/mo server can handle several thousand requests per second. It’s mostly a question of what server software you use. If you use some node, python, ruby thing, it’s going to be slow as shit and need a reverse proxy in front of it. If you use a fast compiled language with a good framework, you can rip through requests no problem. I tried a bunch of different stuff and ended up using Haskell - all of its popular web…
I'm curious if you evaluated Rust?
Earlier quoted context omitted.
Django's prefetch_related mechanism essentially implements the pattern you describe here for you: https://docs.djangoproject.com/en/3.2/ref/models/querysets/#...
Thanks. Yeah, I think I used that years ago when I first ran into this problem, and it worked well. Whether one uses an ORM or not, one needs to know how to use one's tools. My problem (not just with Django, but with ORMs in general) is how they make bad code look good. Like the following (I don't know Django well anymore, but something like this): users = User.objects.all() for u in user: print(u.name, len(u.comment…
users = User.objects.all()
for u in user:
print(u.name, len(u.comments))
Django ORM has a succinct way of doing the "SELECT COUNT(*)" pattern:
users = User.objects.all()
for u in user:
print(u.name, u.comments.count())
users = User.objects.annotate(n_comments=Count("comments"))
for u in user:
print(u.name, u.n_comments)Earlier quoted context omitted.
>There is not even mention of a dB query. Did you read the post?
> These benchmarks show that a very cheap server can easily handle 50 requests a minute to a "full stack" website. I did, and all I see is someone spinning some numbers idly, like, hey, if I can lay 1 brick every second, then with 20000 people we can build a house in one second! So good! a) entirely and totally lacking in experience running a heavy load website. b) 50 requests a minute is so atrociously bad, it’s not…
This is shifting the goal posts. Your initial comment claimed that there are no database queries being made.
In any case, you've amended your argument, I have no further comments.
Earlier quoted context omitted.
What Andy giveth, Bill taketh away.[0] I'm more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from pretty high memory overhead, which leads to suboptimal resource utilization. I've worked with a few different systems that compile their queries into C/C++ (which is transparent to the developer). Those tend to be significantly faster or can use fewer nodes to p…
What reminded me of this the other day is how MacOS will grow your cursor if you “shake” it to help you find it on a big screen. I was thinking about how they must have a routine that’s constantly taking mouse input, buffering history, and running some algorithm to determine when user input is a mouse “shake”. And how many features like this add up to eat up a nontrivial amount of resources.
People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…
A lot of that is due to absolutely lousy code. We had a system management backend at my last company. Loading the users list was unbearably slow; 10+ seconds on a warm cache. Not too terrible, except that most user management tasks required a page reload, so it was just wildly infuriating. Eventually I took a look at the code for the page, which queried LDAP for user data and the database for permissions data. It did…
for each user in (get_freeipa_users | grep_attribute uid):
email = (get_freeipa_users | client_side_find user | grep_attribute email)
last_change = (get_freeipa_users | client_side_find user | grep_attribute krblastpwdchange)
expiration = (get_freeipa_users | client_side_find user | grep_attribute krbpasswordexpiration)
# Some slightly incorrect date math...
send_email
Earlier quoted context omitted.
>There is not even mention of a dB query. Did you read the post?
> These benchmarks show that a very cheap server can easily handle 50 requests a minute to a "full stack" website. I did, and all I see is someone spinning some numbers idly, like, hey, if I can lay 1 brick every second, then with 20000 people we can build a house in one second! So good! a) entirely and totally lacking in experience running a heavy load website. b) 50 requests a minute is so atrociously bad, it’s not…
That was a typo, the worst performance they tested was 54 reqs / second.
Earlier quoted context omitted.
What reminded me of this the other day is how MacOS will grow your cursor if you “shake” it to help you find it on a big screen. I was thinking about how they must have a routine that’s constantly taking mouse input, buffering history, and running some algorithm to determine when user input is a mouse “shake”. And how many features like this add up to eat up a nontrivial amount of resources.
That particular example seems like something that's probably a lot cheaper than you'd initially think. The OS has to constantly take mouse input anyway to move the pointer and dispatch events to userspace. It also needs to record the current and new position of the mouse pointer to dispatch the events. Detecting whether the mouse is being "shaken" can be done with a ring buffer of mouse velocities over the last secon…