I am looking to design a low latency (sub-second) real time system. Clients (~600k) will feed data to Kinesis to Lambda and then to DDB and Redis for fast retrieval. Issue is Kinesis is batched so there is default delay. Can I replace Kinesis with SQS and get better performance?
Low Latency Architecture – AWS
1–9 of 9 posts
Re: Low Latency Architecture – AWS
#2If your batch settings are low enough Kinesis should be fine. SQS with long polling would also be pretty fast if you have enough workers.
Re: Low Latency Architecture – AWS
#3If your batch settings are low enough Kinesis should be fine. SQS with long polling would also be pretty fast if you have enough workers.
I am also okay to dump the whole architecture and re-architecture. Will do a POC but looking to know, if some one has designed something of this sort.
Re: Low Latency Architecture – AWS
#4I work in subsecond latency domain. Can you elaborate what do you want to achieve in the sub-second: Does the ingestion have to be subsecond? or the entire cycle of ingestion and data available into redis/DDB needs to be done in sub-second?
Re: Low Latency Architecture – AWS
#5This is one of those "build a quick PoC and find out" moments.
Also consider what happens when latency spikes; performance isn't guaranteed in this environment. At scale, edge cases become the common case.
On the DynamoDB side, I'm sure you've already seen the DynamoDB Accelerator (10x read performance) already in preview, but just in case, see https://aws.amazon.com/dynamodb/dax/
Re: Low Latency Architecture – AWS
#6I work in subsecond latency domain. Can you elaborate what do you want to achieve in the sub-second: Does the ingestion have to be subsecond? or the entire cycle of ingestion and data available into redis/DDB needs to be done in sub-second?
Entire cycle in sub-second. I agree that it depends on many factors but as per the system, we are looking subsecond from ingestion of data from client to availability of data for query.
A -> stream -> datastore (subsecond)
Re: Low Latency Architecture – AWS
#7I work in subsecond latency domain. Can you elaborate what do you want to achieve in the sub-second: Does the ingestion have to be subsecond? or the entire cycle of ingestion and data available into redis/DDB needs to be done in sub-second?
Entire cycle in sub-second. I agree that it depends on many factors but as per the system, we are looking subsecond from ingestion of data from client to availability of data for query. A -> stream -> datastore (subsecond)
I see. I would suggest you explore the following:
1. CitusDB (an offshoot of postgres with high ingestion rate, cloudflare used it)
2. Druid.io (popular in ad-tech)
3. Linkedin Pinot (uber uses it for similar use-cases)
4. memSQL (used in ad-tech)
Re: Low Latency Architecture – AWS
#8Earlier quoted context omitted.
Entire cycle in sub-second. I agree that it depends on many factors but as per the system, we are looking subsecond from ingestion of data from client to availability of data for query. A -> stream -> datastore (subsecond)
I see. I would suggest you explore the following: 1. CitusDB (an offshoot of postgres with high ingestion rate, cloudflare used it) 2. Druid.io (popular in ad-tech) 3. Linkedin Pinot (uber uses it for similar use-cases) 4. memSQL (used in ad-tech)
Thanks for the pointers but I am planning to revolve more around AWS ecosystem so that I have less ops burden of managing servers + regionalization becomes easy.
Re: Low Latency Architecture – AWS
#9How about using Pubnub or Pusher for this use case?