Live data from Hacker News

Riot Games Messaging Service

engineering.riotgames.com

31–39 of 39 posts

Re: Riot Games Messaging Service

#31
post #30
post #27

Earlier quoted context omitted.

Not the OP, but I work on the same team. The edge servers are not clustered and share no state. We require at least 2 servers as minimum for fault tolerance.

Did the need for edge server arise from the fact the a service like https://aws.amazon.com/elasticloadbalancing/applicationloadb... didn't exist back then ?

Well we need edge servers to handle the persistent websocket connections which last through the life the player session.

Re: Riot Games Messaging Service

#32
post #31
post #30

Earlier quoted context omitted.

Did the need for edge server arise from the fact the a service like https://aws.amazon.com/elasticloadbalancing/applicationloadb... didn't exist back then ?

Well we need edge servers to handle the persistent websocket connections which last through the life the player session.

yeah the ALB provides that feature as well. I just checked. it was released in Aug 2016. So clearly before that time your setup makes a lot of sense.

I am wondering if the ALB would be the preferred method now, if you were to redo it ?

Re: Riot Games Messaging Service

#33
Do people use zeromq for this kind of work? It seems the various socket types are tailored to building such load balancing and routing architectures, but I'm not aware if zeromq is in use at this scale.

Re: Riot Games Messaging Service

#34
post #33

Do people use zeromq for this kind of work? It seems the various socket types are tailored to building such load balancing and routing architectures, but I'm not aware if zeromq is in use at this scale.

Yup, we use ZeroMQ for internal routing for this kind of thing. Each instance of our edge server (Pushpin) binds on a SUB socket to receive messages destined for external clients. This makes it possible for internal publishers (using PUB) to route among edge servers using a brokerless mesh.

Re: Riot Games Messaging Service

#35
post #27

Earlier quoted context omitted.

Looks like a load issue, a single server can only take care of so many conn, auth, etc... So having several solves the problem and can scale horizontally easily, but each edge maintains conn information, so a given user needs to be routed to that same server every time. On the other hand, all LBs share a single address and all should be replaceable by each other, they exist purely to route the user to the right edge…

Not the OP, but I work on the same team. The edge servers are not clustered and share no state. We require at least 2 servers as minimum for fault tolerance.

Thanks for replying!

So what happens to a player session when an edge server dies?

Do you have a way to rehydrate the session on a different server?

Re: Riot Games Messaging Service

#36

I'd like to see more info on their customized Erlang/OTP mnesia . michalptaszek mentioned it in the comment: http://disq.us/p/1gltwfs

Sure, few words on customizations we've built into mnesia: - we integrated it with our eureka-based service discovery mechanism, so that it can automatically cluster with other servers that are spun up in a process of cluster bootstrap/resizing. Also relaxed constraints when merging 2 identical schemas of separate clusters (when table cookies don't match, but everything else matches we still want to merge and take a union of already existing data) - we've added a bunch of auto-merge code (heavily inspired by Ulf's wonderful https://github.com/uwiger/unsplit library) in case of network partitions - we've also added support for maintaining pools of processes for each table for dirty updates (as opposed of going through mnesia_tm for every single operation, including transactions as well as dirty_asyncs)

I'm 100% aware that these changes are RMS/Riot specific and won't work in many other situations (e.g. they violate certain transaction isolation properties).

Re: Riot Games Messaging Service

#37
post #27

Earlier quoted context omitted.

Not the OP, but I work on the same team. The edge servers are not clustered and share no state. We require at least 2 servers as minimum for fault tolerance.

Thanks for replying! So what happens to a player session when an edge server dies? Do you have a way to rehydrate the session on a different server?

When the edge server dies player will reconnect to another node (load balancer will select a healthy one this time). In the same time RMS will detect that the session got lost in an abrupt way and will buffer any outgoing messages addressed to that session for a short time, just in case when player reconnects.

Re: Riot Games Messaging Service

#38
post #32
post #31

Earlier quoted context omitted.

Well we need edge servers to handle the persistent websocket connections which last through the life the player session.

yeah the ALB provides that feature as well. I just checked. it was released in Aug 2016. So clearly before that time your setup makes a lot of sense. I am wondering if the ALB would be the preferred method now, if you were to redo it ?

Right, ALB were introduced after we built RMS. We would have to re-evaluate the ALB stability/cost/scalability - but definitely something to consider.

Re: Riot Games Messaging Service

#39
post #13

Their JSON have an key called payload which saves a String rather than an Object, is there any benefits to doing this? since they also escape the " in the String

Probably that services aren't required to use JSON (since they use many different service languages, it's not unlikely that they use many different serialisation methods - JSON, protobuf, etc.)

Exactly this - each publisher can encode their own payload, including protobufs, JSON, plain text or base64'd binaries. RMS itself is completely oblivious to the format of payload used in the message.
Post reply on HN