[deleted]
My £4 a month server can handle 4.2M requests a day
231–240 of 479 posts
Re: My £4 a month server can handle 4.2M requests a day
#232One of my proudest moment in my career is when I lowered our app processing time from ~8hrs to 17 minutes. When I deployed my first update, it reduced it to 2 hours. The sysadmin immediately contacted me that there was something unusual. I confirmed the results but he was skeptical. Then with my second update, he told me that the app must be broken or that the script must be dying. There is no way it could complete t…
> no one was allowed on the website until the back up was done
I'm assuming this was an internal website and backups were scheduled for evenings/weekends?
Re: My £4 a month server can handle 4.2M requests a day
#233Earlier quoted context omitted.
> now you can get single servers for ~$4K with 4T of RAM Does the $4K include the cost of the RAM? Where can I find these servers? Thanks!
The more I read comments on subjects I am intimately familiar with, the more I realize most people who comment on HN don't really know what they're talking about and mostly make things up. To answer your question, you can't find these servers because they don't exist. A server with 4T of RAM will cost you at a minimum $20,000 and that will be for some really crappy low-grade RAM. Realistically for an actual server th…
For $4k you could get an (old) refurbished R820 with 1.5TB of (old) DDR3 RAM.
Which is a far cry from the claimed 4TB, but still, damn.
Re: My £4 a month server can handle 4.2M requests a day
#234Earlier quoted context omitted.
What reminded me of this the other day is how MacOS will grow your cursor if you “shake” it to help you find it on a big screen. I was thinking about how they must have a routine that’s constantly taking mouse input, buffering history, and running some algorithm to determine when user input is a mouse “shake”. And how many features like this add up to eat up a nontrivial amount of resources.
Simply moving my USB mouse consumes 10% of my CPU. Computers these days...
What's behind the mouse cursor while you're doing it? Could it be the UX/UI layer keeping state up-to-date?
Other possibility, do you have a gaming mouse with 1000Hz polling rate configured?
Re: My £4 a month server can handle 4.2M requests a day
#235People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…
Turns out cloud iaas is severely overpriced and it will always be cheaper to run on prem...let's see how long this lasts
Re: My £4 a month server can handle 4.2M requests a day
#236Earlier quoted context omitted.
I would guess they may be muxed over fewer sockets, by their LBs, but that's not strictly necessary. I'm not sure exactly what you mean by "run out of TCP sockets", but theoretically speaking, the only limitation is how much memory is available to store the necessary info about the socket (like address/protocol info and process info). In practice, OS's do have a "max socket" or "max FD" limit, but that's usually conf…
> I'm not sure exactly what you mean by "run out of TCP sockets", but theoretically speaking, the only limitation is how much memory is available to store the necessary info about the socket (like address/protocol info and process info). Probably the 65k port limit since each connection will get assigned a remote port, which can be solved by binding to multiple local ports and using a load balancer in front or using…
You can have more than one connection per port, so you don't even need the load balancer or more interfaces! :)
Re: My £4 a month server can handle 4.2M requests a day
#237Earlier quoted context omitted.
Simply moving my USB mouse consumes 10% of my CPU. Computers these days...
What's behind the mouse cursor while you're doing it? Could it be the UX/UI layer keeping state up-to-date? Other possibility, do you have a gaming mouse with 1000Hz polling rate configured?
Yeah, it's a high quality mouse. But the only excuse for this is it's slightly cheaper to make everything USB. PS/2 worked much better. It was limited to 200Hz but needed no polling. Motherboards just stopped providing the port.
Re: My £4 a month server can handle 4.2M requests a day
#238People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…
What Andy giveth, Bill taketh away.[0] I'm more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from pretty high memory overhead, which leads to suboptimal resource utilization. I've worked with a few different systems that compile their queries into C/C++ (which is transparent to the developer). Those tend to be significantly faster or can use fewer nodes to p…
I work in the Analytics space and been mostly on Java and I am so glad that other people feel the same.
At this point, people have become afraid of suggesting something other than Spark.
I see something written in Rust to be much better at problems like this.
I love the JVM but it works well with transactional workloads and starts showing its age when its dealing with analytical loads.
The worst thing is then people start doing weak references and weird off the heap processing usually by a senior engineer but really defeats the purpose of the JVM
Re: My £4 a month server can handle 4.2M requests a day
#239Re: My £4 a month server can handle 4.2M requests a day
#240The comments in this thread surprise me quite a lot. I suppose it shouldn't, but it does. This post + the response calls to the surface how badly basic system operations knowledge is needed in the industry and how much of it is missing from the toolkit of most developers.
> basic system operations knowledge
Perhaps you can suggest a book or roadmap to learn it?