It's a pretty question architecture that outsources feature flags to a third party. You're creating a critical path hard dependency and for most apps this is going to be hit multiple times on every request and is something you're going to want to run locally.
How LaunchDarkly Serves Over 4B Feature Flags Daily
11–20 of 20 posts
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#12It's a pretty question architecture that outsources feature flags to a third party. You're creating a critical path hard dependency and for most apps this is going to be hit multiple times on every request and is something you're going to want to run locally.
[deleted]
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#13It's a pretty question architecture that outsources feature flags to a third party. You're creating a critical path hard dependency and for most apps this is going to be hit multiple times on every request and is something you're going to want to run locally.
We've thought quite a bit about how to make this work as a service. The key to our architecture is that evaluating a feature flag for a user does not involve a remote call. We make that work by embedding a rule evaluation engine in our SDKs. When you request a flag, the user is compared against these rules (in memory) and served the appropriate variation.
We then use a streaming API to serve rule changes, so when you make a change to your dashboard, the new rules are streamed to all your backend servers within a few hundred milliseconds.
If you need even more resiliency, you can deploy a small service in your own infrastructure (https://github.com/launchdarkly/ld-relay) that allows you to persist these flag configurations in Redis.
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#14Is averaging 4.6k requests per second really that much?
I think it depends on the workload. Serving 4.6k static pages per second, cached on a CDN, is not too difficult. However, handling an analytics workload of 4.6K RPS is a little harder.
Albeit doable with Lambda and Cassandra. ;)
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#15Earlier quoted context omitted.
I think it depends on the workload. Serving 4.6k static pages per second, cached on a CDN, is not too difficult. However, handling an analytics workload of 4.6K RPS is a little harder.
Albeit doable with Lambda and Cassandra. ;)
I mentioned this briefly in the article, but we thought about doing something with Lambda + API Gateway. But doing the math, 5k RPS pushed through API Gateway is about $1500 daily just to authenticate.
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#16It's a pretty question architecture that outsources feature flags to a third party. You're creating a critical path hard dependency and for most apps this is going to be hit multiple times on every request and is something you're going to want to run locally.
We've thought quite a bit about how to make this work as a service. The key to our architecture is that evaluating a feature flag for a user does not involve a remote call. We make that work by embedding a rule evaluation engine in our SDKs. When you request a flag, the user is compared against these rules (in memory) and served the appropriate variation. We then use a streaming API to serve rule changes, so when you…
Your rules engine sounds interesting - definitely worth a closer look.
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#17Earlier quoted context omitted.
> Events are stored in DynamoDB. We use mongo for "core" data like accounts, feature flags, etc, but none of the high throughput stuff touches MongoDB. Curious, why the choice of MongoDB for anything? In my mind "core" data tends to be very structured and require true transactional updates, both of which are cornerstones of a relational database ( cough Postgres ). Sure it's possible to use MongoDB, but what was the…
We had a lot of operational experience with Mongo from our previous jobs at Atlassian, and started out storing almost everything in Mongo. As we scaled out, we migrated all of our high volume data into other stores-- analytics into DynamoDB, searchable data into ElasticSearch, etc. The last remaining piece is our core data, and there's no strong push to move it out of Mongo.
So, just to clarify, you use DynamoDB and ElasticSearch as primary datastores for those types of data and don't replicate data from Mongo into them?
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#18Earlier quoted context omitted.
We had a lot of operational experience with Mongo from our previous jobs at Atlassian, and started out storing almost everything in Mongo. As we scaled out, we migrated all of our high volume data into other stores-- analytics into DynamoDB, searchable data into ElasticSearch, etc. The last remaining piece is our core data, and there's no strong push to move it out of Mongo.
So, just to clarify, you use DynamoDB and ElasticSearch as primary datastores for those types of data and don't replicate data from Mongo into them?
That's mostly right. We don't use ES as a primary datastore, though.
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#19Are there any good resources for managing feature flags like this? I am currently using Netflix Archaius and it's working great, but I find it's easy to forget what values are set or what flags exist. Over time, it's easy to make a mistake and deploy something broken. I'm currently working on instituting a kanban policy to replace feature flags with static configurations ~1-2 months after their initial deployment where possible. But I'm thinking there's a better way to avoid this trap.
Re: How LaunchDarkly Serves Over 4B Feature Flags Daily
#20Hi! I wet & wanna masturbate with you now. We will play? Im here - http://4ouoat.skhit.in/