Questioning the Lambda Architecture • Hacker News Board

Re: Questioning the Lambda Architecture

#11

Post by vicpara » Thu, Jul 03, 2014, 9:44 AM UTC

The ideas exposed in the article aren't new. Now we just use all these hipsterish technologies that we hope to magically solve our problems by just sticking one's output into another's input. If we think about what happens in a single machine while processing data we have had exactly the same problems for decades. How do you process 2 CDs worth of data when you only have a 486 with 4/8/16Mb of ram ?

- Historically, the data rarely (never) fitted into memory and was at least 100x larger than it.

- If we want to have it for the long run we need to store it on disk. Smarter, dumber, compact or verbose .. you have to do it.

- We have to make sure we spend the little CPU time we have on processing data not jiggling with it. Map-Reduce jobs takes ages to initialize and burn CPU just to read and write to file partitions.

- If you have a long processing pipeline there are two major concepts that we use: buffers and pumps. Files, cache, DBs act as buffers. Kafka is a essentially a pump with a buffer;

- When you process data, depending on what you compute, you need or need not multiple passes through the data. ML and AI most of the time needs such things. Descriptive stats with some smart math tricks avoids two passes. This variable number of passes is the party pooper in stream analytics. In cryptography they solved the problem by breaking down the stream into blocks of equal size. That makes sense for raw data because it is being assemble back using some buffers and pumps at some upper layers. Data wise, mathematically and statistically wise, it doesn't make sense to randomly split data into chunks and apply your choice of algos.

- I still don't understand why so many of us rely on out-of-the-box solutions instead trying to solve the problems, they have specifically, on their own. Why wouldn't a developer stick his java code directly into the pipeline to suck data from Kafka and do his bespoke magic. It will be super fast because it is very specific and does exactly one single job. Yes, there will be maintenance time but all solutions require that time. Instead of debugging apache hadoop/spark/Tez code you debug your own.

What is mentioned above just scratches the surface of the knobs and tuning points of a processing pipeline. These are decisions we need to take and expecting fast-food solutions to do it for us are completely unrealistic expectations.

Re: Questioning the Lambda Architecture

#12

Post by boredandroid » Thu, Jul 03, 2014, 4:14 PM UTC

ntoshev wrote: post #10

There seems to be a fundamental trade-off between latency and throughput, with stream processors optimizing for latency and batch processors optimizing for throughput. The post says stream processors are actually used for reprocessing in production at LinkedIn, and simplifying things might well worth it being slower, but do they know how much slower this is than using Hadoop?

This is an excellent point, there IS a fundamental tradeoff between latency and throughput. Computers are much better a processing larger chunks of data linearly rather than small bits of data. However the question to ask is how much data do you have to batch together to get good throughput? There are diminishing returns and you stop getting much benefit after about 1MB in my experience. So this line of reasoning will not get you to additional latency of more than a few seconds, it definitely won't get you to a 24 hour batch data cycle.

Actually because we think this tradeoff is important we allow consumers from Kafka to specify how much data they want accumulated on the server before the server should complete their fetch request. The default is 1 byte, which means respond as soon as there is anything new for me, which optimizes latency. However setting this to a higher number will reduce round-trips and optimize throughput by avoiding lots of small fetches. This allows you to trade a small amount of latency for throughput. (In either case you can bound the waiting with a timeout so the latency is never worse than some maximum delay even if sufficient data hasn't arrived).

In any case when doing re-processing, as described in this article, there will always be lots of data accumulated and you will always fetch chunks of the maximum size you have configured (say 1MB). So in the reprocessing case you always get the "batch"-like throughput.

Re: Questioning the Lambda Architecture

#13

Post by mythbuster » Thu, Jul 03, 2014, 8:42 PM UTC

ntoshev wrote: post #10

There seems to be a fundamental trade-off between latency and throughput, with stream processors optimizing for latency and batch processors optimizing for throughput. The post says stream processors are actually used for reprocessing in production at LinkedIn, and simplifying things might well worth it being slower, but do they know how much slower this is than using Hadoop?

FWIW most of LinkedIn's batch processing still runs on Hadoop. Stream versus batch is not just about throughput versus latency. It is about what you do with the extra latency budget to increase your throughput. e.g. If you reorganize your data to make your batch process go 10x faster because you used a better join algorithm, then its worth doing things in batch. Otherwise there isn't much difference.