> We were running 4,000 Node containers
LOL
We 30x'd our Node parallelism
151–160 of 261 posts
Re: We 30x'd our Node parallelism
#152Earlier quoted context omitted.
> safety and correctness features You can achieve safety and correctness features for node via good lint rules and typescript/flow.
You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?
I can appreciate your suspicion of a dynamically typed language, but type script isn't really a "hack" its a superset of the language. I am not a huge fan of it, to me its kind of the worst of both worlds- if you need type safety, use something else.
However, if you haven't taken a look at Joi- https://hapi.dev/family/joi/?v=16.1.8 you probably should. To me this was a very happy medium- you can enforce your "types" at the point of ingress and egress at your api, and even do validations there, while still being able to enjoy the flexibility of type safety within your code.
Re: We 30x'd our Node parallelism
#153Earlier quoted context omitted.
> safety and correctness features You can achieve safety and correctness features for node via good lint rules and typescript/flow.
You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?
I find it really annoying how JavaScript is treated on developer forums like this one. Why is it that when the same exact things are done in a typical language it's called “tooling” but in JavaScript it's “duct tape hacks on top of hacks”?
Don't get me wrong, my favorite language is Rust, but pretending the JavaScript ecosystem is unusable doesn't make you cool. I can be extremely productive in TypeScript.
Re: We 30x'd our Node parallelism
#154The only way this makes sense to me is if they have to contend with lots of expensive parsing, event sequencing, and throttling requirements. Payment APIs, bank websites, etc can be quite byzantine. I could understand how one might code yourself into a corner with a monolithic node app and basically just say "F-it, we're doing this synchronously!" I don't even think it's a terribly bad thing to do assuming it favors…
> explore Go or Elixir
I have never seen a good argument for using golang for business logic. If you are writing the actual server then sure, use golang. If you are writing some high-speed network interconnect, use golang. Some crazy caching system, sure use golang. The public WS endpoint, use golang.
But if you need to access a DB with golang for anything more than, like, a session token, then you made the wrong choice and you need to go back and re-assess.
Elixir is in the "germination phase" and I predict massive adoption in the next 5 years. It is a truly excellent platform, every fintech company I know at least has their toe in the water. Everyone I show this video to [1] just says "well, shit."
Re: We 30x'd our Node parallelism
#155I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…
It's important to keep in mind that efficiency isn't usually particularly important for a startup. I'm sure they knew when they initially set up this system that it wasn't performant...but it was nice and quick and easy and gets the feature out the door. Why should they worry about $100k or whatever when they're funded for > $350M? Their bottleneck is engineer hours, not dollars.
Instead the rational thing to do is build something quick and dirty and optimize later, and that's exactly what they've done.
Re: We 30x'd our Node parallelism
#156I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…
I mean, that's always the thing, isn't it? If a company publishes about the problems it has, the question is whether other companies have the same problems and just hide it, or whether this company is actually worse. This comes up a lot with gitlab, for instance; remember the time they discovered they had no backups? At most companies, customers would never find out about that, so I'm not sure that them telling us about it usefully informs my view of their competence. Similarly, here, the only way we'd know about this if they didn't say anything would be poor performance, which would be... less than surprising, on a financial website, in my experience. So maybe they suck, or maybe they're equal with others but a little more open; I don't know how to tell.
Re: We 30x'd our Node parallelism
#157I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…
> When I was last doing this stuff, upwards of 80% of our time was being spent essentially just JSON.parse()'ing, and we were looking to move to protobufs to avoid that. It's only tangentially related to your question, but I can't help but ask this question: why people use JSON instead of protobufs at all? I'm mostly a client-side developer, and most of my server-side experience is in hobby projects; still, I always…
JSON existed before protobufs is really it. When I left the node world 3 years ago, protobufs were the new hot thing. Any new project should start with them over JSON imho.
There is some friction to them though, and I think a lot of it is that most tutorials and beginner books like to keep things as simple as possible, and people start their little project, it gets traction, and then they figure out they need protobufs but now its hard to introduce. In most projects, even today, it seems that its the version 2.0 that gets protobufs, v1.0 keeps JSON for simplicity, unless you have a bunch of seasoned devs involved.
Re: We 30x'd our Node parallelism
#158Earlier quoted context omitted.
>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he…
Most bugs encountered in production systems aren't type based issues. Types are more useful for developer productivity (e.g., intellisense) than any other purpose.
Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors.
I'd love to see any data or case studies that claim the opposite if you have any.
Re: We 30x'd our Node parallelism
#159$300k is $300k, but they just raised $250M last year, is this a really good use of time for their engineering team? That's a little above ~0.1% of capital.
Why wouldn't it be? You save 300k, that's an engineer salary... that's pretty much the meaning of a job, building value that's higher than your salary. This clearly took less than a year of engineer time. Seems like they got their value out of that employee.
Re: We 30x'd our Node parallelism
#160I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…
>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he…
The real killer feature is async. Since a modern web request typically spend most of the time waiting for database calls, file system requests or similar, a naively coded server in most languages can handle relatively few requests per thread, so you scale up the number of threads to something like 100 per core, and now the overhead of running and switching between these treads is limiting the performance.
Being used to Node, I was flabbergasted when writing C for Linux* . The file system commands just leave my thread hanging while the result is being generated, if I use it on a network drive it might hang for a minute before timing out, so I have to make a tread for each file system command, solely so that it can stall without bringing down the whole application.
* I have no delusions that Windows is any better, Linux is just what I have first hand experience with.