My 1cpu 2gb ram server: - nginx - slack bot behind Django - OpenVPN - Minecraft server (20 players) - tunneling (reverse proxy for local dev)
Press X to doubt.
My £4 a month server can handle 4.2M requests a day
431–440 of 479 posts
Re: My £4 a month server can handle 4.2M requests a day
#432The comments in this thread surprise me quite a lot. I suppose it shouldn't, but it does. This post + the response calls to the surface how badly basic system operations knowledge is needed in the industry and how much of it is missing from the toolkit of most developers.
Any recommended reading? I've bought a few things in that vein but I'm self-taught and always looking to improve on the ops/perf side.
I don't have any one complete book that I can recommend, and I don't even really have a great reading list for this. But I'll make an attempt to share what I think is useful as a starting point.
1. Systems Operations is first and foremost about understanding systems, in all of their complexity, which means understanding the internals of your OS primarily.
2. Performance and networking, in particular, are super important areas to focus on understanding when it comes to learning the topic to help with software development.
3. A lot of it is about understanding concepts in abstract and being able to extrapolate to other situations and apply these concepts, so there's actually quite a lot of useful information that can be learned on one OS and still applied to another OS (or on one game engine and applied to another, et al).
Here's a few books I think are worth reading, not in any particular order of prevalence, but loosely categorized
Databases:
High Performance MySQL: https://www.amazon.com/gp/product/1449314287/
SQL Queries for Mere Mortals: https://www.amazon.com/gp/product/0321992474/
The Art of SQL: https://www.amazon.com/gp/product/0596008945/
Networking:
TCP/IP Illustrated: https://www.amazon.com/exec/obidos/ISBN=0201633469/wrichards... (updates on author's site at http://www.kohala.com/start/tcpipiv1.html)
The TCP/IP Guide: https://www.amazon.com/TCP-Guide-Comprehensive-Illustrated-P...
UNIX Network Programming: https://www.amazon.com/dp/0131411551
Beej's Guide to Network Programming: http://beej.us/guide/bgnet/
Operating Systems:
Operating Systems Concepts: https://www.amazon.com/Operating-System-Concepts-Abraham-Sil... (various editions, I have the 7th edition... I recommend you find the latest)
Modern Operating Systems: https://www.amazon.com/Modern-Operating-Systems-Andrew-Tanen... (the "Tanenbaum Book")
Operating Systems Design and Implementation: https://www.amazon.com/Operating-Systems-Design-Implementat-... (the other one, the "MINIX Book")
Windows Internals:
Part 1: https://www.amazon.com/Windows-Internals-Part-architecture-m...
Part 2: https://www.amazon.com/Windows-Internals-Part-2-7th/dp/01354... (I had the pleasure of being taught from this book by Mark Russinovich and David Solomon at a previous employer, was an amazing class and these books are incredible resources even applied outside of Windows, we used 5th edition, I linked 7th, which has the 2nd part pending publication).
MacOS Internals:
Part 1: https://www.amazon.com/MacOS-iOS-Internals-User-Mode/dp/0991...
Part 2: https://www.amazon.com/MacOS-iOS-Internals-II-Kernel/dp/0991...
Part 3: https://www.amazon.com/MacOS-iOS-Internals-III-Insecurity/dp...
Linux Kernel Programming:
Part 1: https://www.amazon.com/Linux-Kernel-Development-Cookbook-pro...
Part 2: https://www.amazon.com/Linux-Kernel-Programming-Part-Synchro...
The Linux Programming Interface: https://www.amazon.com/Linux-Programming-Interface-System-Ha...
General Systems Administration:
Essential Systems Administration: https://www.amazon.com/gp/product/0596003439/
UNIX and Linux Systems Administration Handbook: https://www.amazon.com/UNIX-Linux-System-Administration-Hand...
The Linux Command Line and Shell Scripting Bible: https://www.amazon.com/Linux-Command-Shell-Scripting-Bible/d...
UNIX Shell Programming: https://www.amazon.com/Unix-Shell-Programming-Stephen-Kochan...
BASH Hackers Wiki: https://wiki.bash-hackers.org/
TLDP Advanced BASH Scripting Guide: https://tldp.org/LDP/abs/html/
The Debian Administrator's Handbook: https://debian-handbook.info/browse/stable/
TLDP Linux System Administrator's Guide: https://tldp.org/LDP/sag/html/index.html
Performance & Benchmarking:
Systems Performance: https://www.amazon.com/Systems-Performance-Brendan-Gregg-dp-... (this is Brendan Gregg's book where you learn about the magic of dtrace)
BPF Performance Tools: https://www.amazon.com/Performance-Tools-Addison-Wesley-Prof... (the newer Brendan Gregg book about BPF, stellar)
The Art of Computer Systems Performance Analysis: https://www.cse.wustl.edu/~jain/books/perfbook.htm (no longer available from Amazon, but is available direct from publisher. This is basically the one book you should read about creating and structuring benchmarks or performance tests)
I guess that's a "reading list", but this is just a small part of what you need to know to excel in systems operations.
I would say for the typical software developer writing web applications, the most important thing to know is how databases work and how networking works, since these are going to be the primary items affecting your application performance. But there's obviously topics not included in this list that are also worth understanding, such as browser/DOM internals, how caching and CDNs work, and web-specific optimizations that can be achievable with HTTP/2 or QUIC.
For the average software developer writing desktop applications, I'd say make sure you /really/ understand OS internals... at the base everything you do on a computer system is based on what the OS provides to you. Even though you are abstracted (possibly many layers) away from this, being able to peel back the layers and understand what's /really/ happening is essential to writing high-quality application code that is performant and secure, as well as making you a champ at debugging issues.
If you're trying to get into systems operations as a field, this is just a brush over the top surface and there's a lot deeper diving required.
Re: My £4 a month server can handle 4.2M requests a day
#433The comments in this thread surprise me quite a lot. I suppose it shouldn't, but it does. This post + the response calls to the surface how badly basic system operations knowledge is needed in the industry and how much of it is missing from the toolkit of most developers.
> basic system operations knowledge Perhaps you can suggest a book or roadmap to learn it?
I replied to your sibling comment with something approaching a book list.
Re: My £4 a month server can handle 4.2M requests a day
#434Earlier quoted context omitted.
Yeah, I see this a lot. I think it's especially easy to introduce this kind of "accidentally quadratic" behaviour using magical ORMs like Django's, where an innocent-looking attribute access like user.groups can trigger a database query ... access user.groups inside a loop and things get bad quickly. In the case of groups and permissions there's probably only a few of each, so fetching all of them is probably fine. B…
I’m coming around to considering Django-seal to be mandatory for non-trivial applications. It will “seal” a query set so that you can’t make accidental DB lookups after the initial fetch. That way you can be more confident that you are doing the right joins in the initial query, and you are safe from the dreaded O(N) ORM issue. SqlAlchemy has this as part of the ORM, it should really be part of Django IMO.
[deleted]
Re: My £4 a month server can handle 4.2M requests a day
#435Earlier quoted context omitted.
Yeah, I see this a lot. I think it's especially easy to introduce this kind of "accidentally quadratic" behaviour using magical ORMs like Django's, where an innocent-looking attribute access like user.groups can trigger a database query ... access user.groups inside a loop and things get bad quickly. In the case of groups and permissions there's probably only a few of each, so fetching all of them is probably fine. B…
I’m coming around to considering Django-seal to be mandatory for non-trivial applications. It will “seal” a query set so that you can’t make accidental DB lookups after the initial fetch. That way you can be more confident that you are doing the right joins in the initial query, and you are safe from the dreaded O(N) ORM issue. SqlAlchemy has this as part of the ORM, it should really be part of Django IMO.
I've never heard of django-seal before, thanks!
Re: My £4 a month server can handle 4.2M requests a day
#436Earlier quoted context omitted.
I get a kick out of stuff like this - I’m mostly an exec these days, but I recently prototyped a small database system to feed a business process in SQLite on my laptop. To my amusement, my little SQLite prototype smoked the “enterprise” database. Turns out that a MacBook Pro SSD performs better than the SAN, and the query planner needs more tlc. We ended up running the queries off my laptop for a few days while the…
Who the fuck thinks it's a good idea to run database on networked storage?
The same people who will run a clustered database on VMs.
Re: My £4 a month server can handle 4.2M requests a day
#437Earlier quoted context omitted.
> now you can get single servers for ~$4K with 4T of RAM Does the $4K include the cost of the RAM? Where can I find these servers? Thanks!
The more I read comments on subjects I am intimately familiar with, the more I realize most people who comment on HN don't really know what they're talking about and mostly make things up. To answer your question, you can't find these servers because they don't exist. A server with 4T of RAM will cost you at a minimum $20,000 and that will be for some really crappy low-grade RAM. Realistically for an actual server th…
Maybe they downloaded more RAM...
Re: My £4 a month server can handle 4.2M requests a day
#438Earlier quoted context omitted.
I get a kick out of stuff like this - I’m mostly an exec these days, but I recently prototyped a small database system to feed a business process in SQLite on my laptop. To my amusement, my little SQLite prototype smoked the “enterprise” database. Turns out that a MacBook Pro SSD performs better than the SAN, and the query planner needs more tlc. We ended up running the queries off my laptop for a few days while the…
Who the fuck thinks it's a good idea to run database on networked storage?
Isn't that what's happening if you use any managed database product? They have probably colocated everything as much as possible and used various proprietary tricks to cut latency, but still.
Re: My £4 a month server can handle 4.2M requests a day
#439Earlier quoted context omitted.
It's death by a thousand paper cuts. Lots of things that aren't really that slow in isolation, but in aggregate (or under pressure) they slow down the system and become impossible to measure. Let's do web development. Since you mentioned payloads: today they're bigger, and often come with redundant fields, or sometimes they're not even paginated! This slows down the database I/O, requires more cache space, slows down…
That's interesting, thanks for sharing. > All those things are very simple, and you certainly know all of them. I wonder about that. Most of the people I graduated with didn't know about complexity. Some had never touched a relational database, most probably didn't know about views. I doubt most of them knew what serialization mean.
I wonder if it’s a matter of background. I never really had tutorials when starting out. I never had good documentation, even.
(Sorry in advance for the rant)
I also remember piercing together my first programs from other people’s code. Whenever I needed an internet forum I’d build one. Actually, all my internet friends, even the ones who didn’t go into programming, were doing web forums and blogs from scratch!
Today people consider that a heresy. “How dare you not use Wordpress”.
My generation just didn’t care, we built everything from scratch because it was a badge of honor to have something made by us. We didn’t care about money, but we ended up with abilities that pay a lot of cash. People who started programming post the 2000s just didn’t do it...
I think it is visible that I sorta resent the folks (both the younger, and the older who arrived late at the scene) constantly telling me I shouldn’t bother “re-inventing the wheel”. Well, guess what: programming is my passion, fuck the people telling me to use Unity to make my game, or Wordpress to do my blog.
Re: My £4 a month server can handle 4.2M requests a day
#440Earlier quoted context omitted.
You’d be amazed at how cheap it is. You can usually get an MP to do your bidding or say your piece in Parliament for a donation of £10k or less. I’ve never heard of anyone paying more.
It would be a contempt of Parliament to either request or agree to such an arrangement [1]. If you have definitive evidence of this having happened in the recent past, you really should pass it on to the Parliamentary Commissioner for Standards. [1] https://erskinemay.parliament.uk/section/5023/corruption-or-...
This is why it’s a donation to the party - not to them or their office - although they can want a campaign contribution, which again doesn’t count as going to them.
I’ve had an MP literally solicit this from me - he emailed a bunch of local businesses basically stating his price list. He’s no longer an MP, or in politics - but you see this stuff everywhere, all the time. Any member’s bill you see has almost certainly been sponsored.