I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…
We 30x'd our Node parallelism
141–150 of 261 posts
Re: We 30x'd our Node parallelism
#142I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…
>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he…
Then, you take a look at node. You look at a getting started tutorial. Its javascript on the front, and on the back. The JSON in between is "native" and is convenient and easy to use, easy to read, lightweight, and just makes a lot of intuitive sense- especially when I had found myself neck deep in XML in previous jobs for the same tasks. I had a nice looking HTML5 web app running in a few minutes- my mind was blown. Then you take a look at the frameworks- express and hapi, and the vast module ecosystem- and how easy it was to build a simple CRUD website with leveldb, or mysql, or really an endless array of storage options. And people were using those options! It wasn't just the bog standard RDBMS being used every place, with your only real choice being mysql, postgres, or if you had money, Oracle. Building endpoints with routes in these frameworks made your code so easy to divide up along clear lines, and there just wasn't the endless miles of boilerplate/scaffold code, and ugly syntax and type systems to fight with and plan ahead of. Things Just Worked. Turning around a code change was a matter of seconds, not a minutes long build process- I had never felt so productive- and writing code was fun again! Deploys were easy, restarts were fast. Rollbacks, when necessary, were painless. There was a plugin/module for everything (too much in hindsight).
Now, this was 6 years ago. Go was around, but still kind of a blip on the radar, Ruby/Python were probably the closest real contenders. Ruby had lost steam, I honestly took some cursory looks at it, but it didn't seem to have traction. Python, suffered from its single threadedness and GIL, and its popularity with the ML crowd- Flask and such existed, but was pretty rudimentary compared to what Express/Hapi were offering, and no one seemed that interested in those projects. I like Go a lot, and for a pure backend service, it might be my go-to today, as one of the original arguments for Node was "its the same language on the front and the back, no more delineation between FE and BE developers, anyone can jump in and fix the bugs!" Which, along the lines of my original comment, don't really work out in reality, at least not on larger systems. People drawn to FE work usually have never done real systems development and don't understand how things work under the hood- which isn't a problem, until one day it is and then its a huge one.
The dynamic typing argument... is somewhat valid, but I found that enforcing api contracts with hapi/joi gave you the equivalence of type safety at your interface borders, while still giving you the flexibility of dynamic typing within your code. In fact, Joi went even farther than just type checking, it could check that your int was within range for the field, that your dates were formatted properly, etc... In mega large codebases, this will come back to bite you, but I found the plugin architecture of Hapi really discouraged that kind of crap from leaking in and it was easy to build truly modularized code.
The performance ceilings aren't that different, and not that impactful, at least not until you get to FANG scale, and I mean literally only FANG scale. We were running a billion dollar business with on 8 fairly small VMs for the API layer, which handled all of the ecommerce transaction handling. I remember at one point we encountered a memory leak of some sort in node, and the instances were falling over and dying about once an hour, but restarting and recovering- this was causing a few % error rates to our customers. I was insistent that we get all hands on deck to figure this out ASAP, and our head of Ops type person said "kevstev, we can throw hardware at this problem to meet SLOs until you get it under control. Your monthly server costs are less than my studio apartment cost me per month in Jersey City 15 years ago."
You just have to have a basic understanding of whats going on at an architectural level, something a few hours of doing the right reading and experimenting can get you if you have the proper background. The number of gotchas to avoid to get that performance were an order of magnitude, if not more, fewer than in a language like C++ (Which I feel has actually gotten so complicated and difficult to grok its become a parody of itself- and I say that as someone who used it and adored it for 15 years).
Re: We 30x'd our Node parallelism
#143Earlier quoted context omitted.
>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he…
> safety and correctness features You can achieve safety and correctness features for node via good lint rules and typescript/flow.
Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?
Re: We 30x'd our Node parallelism
#144Earlier quoted context omitted.
Async is just modern cooperative multitasking, and just like the 90s, it's easy to accidentally lock the whole system.
Yeah, I remember just how nice it was going from Cooperative MT to Preemptive multi tasking -- the general view was that anything that only did Cooperative MT was just a Toy. I'd bet that the orders of magnitude of speed from Moore's law did in CMT by making PMT doable without a huge speed hit. Nothing I've seen from async is cleaner, easier to maintain, or better from a cognitive load POV. It's just more efficient f…
https://dev.to/nestedsoftware/is-cooperative-concurrency-her...
This article does a good overview
Re: We 30x'd our Node parallelism
#145Earlier quoted context omitted.
> Invalid or broken content should return an error, the parser shouldn't try to "fix" it. You would have a really difficult life in web scraping. You do not have the guarantees of well-formed data. Instead you get HTML with mismatched tags, JSON with newlines in the middle of strings, content that claims it's UTF-8 but upon closer inspection is actually GB2312, pagination endpoints with off-by-one errors, etc. It's a…
I'm actually quite experienced with web scraping, mostly using PHP and XPath, but also with Javascript as well as a custom approach written in Rust. I know in detail what an inconsistent mess everything is. That's why I'm so uncomfortable handling things like bank transfers over such inconsistent, buggy systems, which is what Plaid does. It's not read-only: https://plaid.com/use-cases/consumer-payments/ Not to say I…
I have no affiliation with plaid, I've honestly only heard negative things about those guys, I was only empathizing with the difficulties in maintaining thousands of different scrapers and why I felt a scripting language provided far more latitude to get things done.
Re: We 30x'd our Node parallelism
#146I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…
>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he…
Re: We 30x'd our Node parallelism
#147Nobody involved in this project should be allowed to ever be in the same room as a computer again.
Why? They had a 12 factor -ish app that scaled the normal way; run more copies. Eventually that got expensive. They had the observability to figure out what was making it expensive and whether or not their fixes had an effect. They then saved $300,000. Seems like everything went right to me. I would be worried if the blog post was "we randomly tweaked some stuff and we can't measure it but it's a little better" or "w…
So, yes, horizontal scaling is good, especially for stateless workloads - but that doesn't mean you run the most hopelessly under-performing code imaginable on each node, so you basically have to scale out like this! I mean, seriously, 4000 containers to serve 4000 concurrent requests? I mean, I can't even...
I honestly can't believe the attempts in this thread to justify such an utterly, horrendously bad architecture - there are 1001 better, simpler even, ways to approach this.
Yes, premature optimisation is bad, but optimisation here was nowhere near premature.
Re: We 30x'd our Node parallelism
#148I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…
It's only tangentially related to your question, but I can't help but ask this question: why people use JSON instead of protobufs at all?
I'm mostly a client-side developer, and most of my server-side experience is in hobby projects; still, I always used protobufs and loved it. They never damaged my feature velocity, apart from an hour to set up the build system in the beginning, and type safety helped me quite a few times when I forgot to sync changes in protocol on client and server side. Are there some secret advantages of going with json that I don't see because of limited experience?
Re: We 30x'd our Node parallelism
#149This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for an API workload is absolute insanity.
These engineers are building stuff for banking. Banking!! There is literally no way I'm going near Plaid with a very long bargepole after reading this.
It I was someone senior at Plaid, I'd be pulling this blog post before it harms reputation any further.
Re: We 30x'd our Node parallelism
#150The insistence on using Javascript is just beyond lunacy at this point.