Either optimising the main app or caching with Redis/Memcached can seriously reduce the number of instances & improve the 133 req/sec per server metric as well.
Lessons learned from running GraphQL
21–30 of 64 posts
Re: Lessons learned from running GraphQL
#22Earlier quoted context omitted.
If you read between lines you'll see failures. This article is an example. Spawning 7500 servers to handle this traffic is a facepalm failure.
To be fair, looking at what kind of optimizations they did to improve the situation, it looks like GQL is not to blame but rather a pretty big disconnect between their implementation practices and understanding about what is costly and what is not costly (ie lack of mechanical sympathy). People may get by without that writing client side javascript, but backend code is not as generous.
Yes, their optimisation team did great job, including writing post about it. They had 5 months from zero to some solution deployed on prod. And they shaved numbers. Great work.
However number of servers required to serve this traffic has still very poor ratio. And if you read between lines – they maxed out their optimisation effort on it. They cache at multiple layers. To be more precise this statement that they haven't found any low hanging fruits for optimisation should raise some serious questions and their analysis from "first principles" should probably be more thorough.
Re: Lessons learned from running GraphQL
#23It was obvious to me from the beginning that GraphQL would add overhead and complexity on the backend; especially related to caching all possible permutations/views of the data. In some cases I imagine it would consume a lot of memory; wouldn't it cause a memory leak vulnerability if you allow infinite permutations to be cached by the server? On the other hand, if you only cache responses to popular requests, doesn't…
> It was obvious to me from the beginning that GraphQL would add overhead and complexity on the backend;
Did you read the article? Most of the issues weren't related to GraphQL, they were just Node issues/optimizations.
> I think it's regrettable that all the big money got behind GraphQL instead of aiming for solutions which provide resource granularity and shift decision-making to the client side.
This is the stated intent of GraphQL. Literally the reason it exists.
Re: Lessons learned from running GraphQL
#24It was obvious to me from the beginning that GraphQL would add overhead and complexity on the backend; especially related to caching all possible permutations/views of the data. In some cases I imagine it would consume a lot of memory; wouldn't it cause a memory leak vulnerability if you allow infinite permutations to be cached by the server? On the other hand, if you only cache responses to popular requests, doesn't…
> I think it's regrettable that all the big money got behind GraphQL instead of aiming for solutions which provide resource granularity and shift decision-making to the client side. Who is better placed to know what resources they want than the client?
With GraphQL the client specifies exactly what it needs. It‘s as granular as you can imagine, unlike REST.
Re: Lessons learned from running GraphQL
#25Do I read this right? 1,000,000 requests/sec across 7,500 instances is only 133 requests/second, and graphql wouldn’t typically represent the business logic or data layer. I love me some graphql, but that seems to be a very low figure. I’m curious how complex the queries are and what else these servers are doing.
I've seen similar situations where each host was achieving just 16 requests/sec. Engineers are expensive, and growing more so every year. It's hard to justify time spent to optimise rather than throwing more instances at it. The cloud has made this worse in a way, since provisioning more hosts can be done so easily. Not many engineers even have the skill to identify and resolve performance problems, so again, people…
I get it, but at this scale I think we are talking in the order of a million bucks a year. I don’t know the situation in India, but I imagine that buys a lot of engineers.
Re: Lessons learned from running GraphQL
#26>"We provision approximately 7,500 instances for 1 million requests per second." Looking at this numbers makes me think that a single instance of properly written server running on a single dedicated piece of hardware can handle this without breaking a sweat. My servers for example handle thousands of requests per second. It looks to me like one giant waste of human and hardware resources. Not very "green" approach I…
The default implementation of GraphQL has a lot of overhead in query parsing and validation alone. You can try this yourself with complex queries and simulating some load.
But it‘s an issue that can be solved.
Re: Lessons learned from running GraphQL
#27It was obvious to me from the beginning that GraphQL would add overhead and complexity on the backend; especially related to caching all possible permutations/views of the data. In some cases I imagine it would consume a lot of memory; wouldn't it cause a memory leak vulnerability if you allow infinite permutations to be cached by the server? On the other hand, if you only cache responses to popular requests, doesn't…
> The fact that GraphQL allows all these permutations in the queries is the root of the problem. It's not something which can be solved or optimized within GraphQL.
Common ways to solve that are to whitelist the allowed queries or to cache at the resolver level instead of the query level.
Re: Lessons learned from running GraphQL
#28I wonder if they feel like GraphQL was worth it, vs. normal API servers. Maybe they saved some dev time on the front end, but did that outweigh the dev time spent on building, optimizing, etc? Somehow I doubt it.
I’ve been using graphql for years. In my experience, it dramatically simplifies microservice API architecture vs “normal” API servers. Graphql is super easy to understand, easy to deploy, easy to scale and easy to grow. It’s not perfect - the lack of namespaces can be a pain, a few more standard types would be good, and mutations feel a bit under baked - but there’s much to love, and very little to dislike.
Downsides afaik are:
(1) No way to do queries, which return recursive JSON objects of arbitrary depth. (2) Not using standard JSON as a format for writing your query, instead unnecessarily making up a new querying lang, a design flaw basically. (3) More dependencies in frontend as well as backend. (4) More difficult to determin what exactly is going on in processing 1 query, ergo more difficult to fix performance problems.
Re: Lessons learned from running GraphQL
#29The big items in that list of performance issues don't seem to have anything to do with GraphQL. They seem to be related to a heavily function style using lots of immutability and the Ramda library. I'd also suspect that these choices are responsible for the GC issues due to lots of allocations for the immutable objects. I know, premature optimization and all that. But I think at the point where you're going for micr…
At around one million requests per second, imperative programming becomes affordable; and more importantly, necessary.
Re: Lessons learned from running GraphQL
#30I wonder if they feel like GraphQL was worth it, vs. normal API servers. Maybe they saved some dev time on the front end, but did that outweigh the dev time spent on building, optimizing, etc? Somehow I doubt it.
I bet REST API + HTTP caching is going to outperform the GraphQL APIs. And maybe most importantly, it’s going to be cheaper.