Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned from running GraphQL

blog.dream11engineering.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: Lessons learned from running GraphQL

#11

Post by tshaddox »

ripper1138 wrote: post #3

I wonder if they feel like GraphQL was worth it, vs. normal API servers. Maybe they saved some dev time on the front end, but did that outweigh the dev time spent on building, optimizing, etc? Somehow I doubt it.

I think it would depend mostly on the diversity of combinations of data the frontend needs. Their GraphQL implementation is essentially automating the process of frontend teams asking the backend for new endpoints or configurations of existing endpoints to deliver new combinations of data for use by frontend clients. It’s pretty easy to see that certain for frontend requirements the backend GraphQL will be worth it, and for other frontend requirements the backend work would not be worth it. In this case I’m relatively confident that they’re coming out ahead.

Re: Lessons learned from running GraphQL

#12

Post by jayceedenton »

doctor_eval wrote: post #7

Do I read this right? 1,000,000 requests/sec across 7,500 instances is only 133 requests/second, and graphql wouldn’t typically represent the business logic or data layer. I love me some graphql, but that seems to be a very low figure. I’m curious how complex the queries are and what else these servers are doing.

I've seen similar situations where each host was achieving just 16 requests/sec.

Engineers are expensive, and growing more so every year. It's hard to justify time spent to optimise rather than throwing more instances at it. The cloud has made this worse in a way, since provisioning more hosts can be done so easily.

Not many engineers even have the skill to identify and resolve performance problems, so again, people just keep adding more machines. Long term, the problem slowly builds all over the system and the bill becomes mind-boggling.

I do think that we (the software folks) don't help ourselves here. We build frameworks and tools that are still far too hard to inspect. What to watch (and how to optimise) in production, is often never considered deeply when building or documenting the hot new thing.

Re: Lessons learned from running GraphQL

#13

Post by brabel »

jjtheblunt wrote: post #4
ripper1138 wrote: post #3

I wonder if they feel like GraphQL was worth it, vs. normal API servers. Maybe they saved some dev time on the front end, but did that outweigh the dev time spent on building, optimizing, etc? Somehow I doubt it.

I wonder if they feel like Node was worth it. (seriously)

Exactly, the problems they found were mostly Node problems, not GraphQL problems.

Re: Lessons learned from running GraphQL

#14

Post by rawoke083600 »

medium.com is terrible with GQL... I can have 50 tabs open in chrome. Then I open 1 tab on medium.com and CPU spike like crazy. Turns out it's their graphql-queries at 100 miles a second. One of the API (GQL) queries fails (I'm guessing my adblocker ?) then the retry seems to be in a tight never ending loop !

Re: Lessons learned from running GraphQL

#15

Post by jopsen »

jayceedenton wrote: post #12
doctor_eval wrote: post #7

Do I read this right? 1,000,000 requests/sec across 7,500 instances is only 133 requests/second, and graphql wouldn’t typically represent the business logic or data layer. I love me some graphql, but that seems to be a very low figure. I’m curious how complex the queries are and what else these servers are doing.

I've seen similar situations where each host was achieving just 16 requests/sec. Engineers are expensive, and growing more so every year. It's hard to justify time spent to optimise rather than throwing more instances at it. The cloud has made this worse in a way, since provisioning more hosts can be done so easily. Not many engineers even have the skill to identify and resolve performance problems, so again, people…

Well, adding machines allows you to postpone fixing the problem.

It's perfectly reasonable, once the bill starts to catch up to your budget you spend time on optimizations :)

EDIT: the only solid argument against throwing machines at the problem is that: scaling something across multiple servers is hard. If you spent energy on performance, maybe you didn't have to.

Re: Lessons learned from running GraphQL

#16

Post by mirekrusin »

sibeliuss wrote: post #5
ripper1138 wrote: post #3

I wonder if they feel like GraphQL was worth it, vs. normal API servers. Maybe they saved some dev time on the front end, but did that outweigh the dev time spent on building, optimizing, etc? Somehow I doubt it.

> Somehow I doubt it I don't know. They seem to be satisfied customers, and were simply optimizing an already working pipeline in anticipation of saving money. As a side note, I haven't seen a single "We tried GraphQL and it failed us" story on HN. Not that they don't exist, of course. It's just that there doesn't seem to be much debate about its promise.

If you read between lines you'll see failures. This article is an example. Spawning 7500 servers to handle this traffic is a facepalm failure.

Re: Lessons learned from running GraphQL

#17

Post by vosper »

rawoke083600 wrote: post #14

medium.com is terrible with GQL... I can have 50 tabs open in chrome. Then I open 1 tab on medium.com and CPU spike like crazy. Turns out it's their graphql-queries at 100 miles a second. One of the API (GQL) queries fails (I'm guessing my adblocker ?) then the retry seems to be in a tight never ending loop !

This doesn’t sound like a problem due to GraphQL, though?

Re: Lessons learned from running GraphQL

#18

Post by fabian2k »

The big items in that list of performance issues don't seem to have anything to do with GraphQL. They seem to be related to a heavily function style using lots of immutability and the Ramda library. I'd also suspect that these choices are responsible for the GC issues due to lots of allocations for the immutable objects.

I know, premature optimization and all that. But I think at the point where you're going for microservices because of scaling you really should also look into the lower level issues like that from the start. You should notice that the shiny library you're using is 100x slower than just writing plain code. And you should be aware of excessive allocations in hot paths.

Re: Lessons learned from running GraphQL

#19

Post by yxhuvud »

mirekrusin wrote: post #16
sibeliuss wrote: post #5

Earlier quoted context omitted.

> Somehow I doubt it I don't know. They seem to be satisfied customers, and were simply optimizing an already working pipeline in anticipation of saving money. As a side note, I haven't seen a single "We tried GraphQL and it failed us" story on HN. Not that they don't exist, of course. It's just that there doesn't seem to be much debate about its promise.

If you read between lines you'll see failures. This article is an example. Spawning 7500 servers to handle this traffic is a facepalm failure.

To be fair, looking at what kind of optimizations they did to improve the situation, it looks like GQL is not to blame but rather a pretty big disconnect between their implementation practices and understanding about what is costly and what is not costly (ie lack of mechanical sympathy). People may get by without that writing client side javascript, but backend code is not as generous.

Re: Lessons learned from running GraphQL

#20

Post by cryptica »

It was obvious to me from the beginning that GraphQL would add overhead and complexity on the backend; especially related to caching all possible permutations/views of the data. In some cases I imagine it would consume a lot of memory; wouldn't it cause a memory leak vulnerability if you allow infinite permutations to be cached by the server? On the other hand, if you only cache responses to popular requests, doesn't that expose your servers to DDoS? An attacker could just generate a ton of unique GraphQL queries to make the servers bypass the cache and consume a ton of CPU. The fact that GraphQL allows all these permutations in the queries is the root of the problem. It's not something which can be solved or optimized within GraphQL.

I think it's regrettable that all the big money got behind GraphQL instead of aiming for solutions which provide resource granularity and shift decision-making to the client side. Who is better placed to know what resources they want than the client? A big advantage of HTTP/REST is that it either serves individual resources or a limited number of different collections of resources and it lets clients do the heavy lifting of figuring out which resources they need and how they want to combine them. Caching REST endpoints is straight forward and resilient to DDoS attacks because the variations in responses is strictly limited.

Also, it makes sense to move processing to clients when those processing costs are imperceptible to users.

Post reply on HN