11–20 of 21 posts
Pretty interesting to see this actually happen and see where Google ended up with it.
The most scalable, fastest reaction time and simplest log processing tool is awk.
> It also guarantees exactly-once semantics for application pipelines to process logs in the form of event bundles. Well, it's got my attention.
Let me take this opportunity to say I'm appalled all research at Google has been rolled into "AI." Ubiq has nothing to do with machine learning, but you can't access it without wading through a swath of Google-branded AI marketing material. The domain research.google.com now redirects to ai.google. If you want to search for any research publications by Google employees, you will be searching on a domain that first an…
30% and 1/3 not the exact real numbers, just showing a concept.
The most scalable, fastest reaction time and simplest log processing tool is awk.
Ok, I'll bite. Most scalable, you say. I have 250TB of Django logs, how do you recommend I use awk to process them to determine the 99th percentile response time that's faster than using SparkSQL?
Where are your logs stored? Is that a distributed storage? Will SparkSQL not eat all of the bandwidth of it’s ethernet interfaces?
Yeah; sure.
The most scalable, fastest reaction time and simplest log processing tool is awk.
Ok, I'll bite. Most scalable, you say. I have 250TB of Django logs, how do you recommend I use awk to process them to determine the 99th percentile response time that's faster than using SparkSQL?
Earlier quoted context omitted.
Ok, I'll bite. Most scalable, you say. I have 250TB of Django logs, how do you recommend I use awk to process them to determine the 99th percentile response time that's faster than using SparkSQL?
SparkSQL doesn’t support gz. Are your logs splittable on a file-by-file basis or are they in gz format? Where are your logs stored? Is that a distributed storage? Will SparkSQL not eat all of the bandwidth of it’s ethernet interfaces? Yeah; sure.
The most scalable, fastest reaction time and simplest log processing tool is awk.
Ok, I'll bite. Most scalable, you say. I have 250TB of Django logs, how do you recommend I use awk to process them to determine the 99th percentile response time that's faster than using SparkSQL?
Earlier quoted context omitted.
Ok, I'll bite. Most scalable, you say. I have 250TB of Django logs, how do you recommend I use awk to process them to determine the 99th percentile response time that's faster than using SparkSQL?
SparkSQL doesn’t support gz. Are your logs splittable on a file-by-file basis or are they in gz format? Where are your logs stored? Is that a distributed storage? Will SparkSQL not eat all of the bandwidth of it’s ethernet interfaces? Yeah; sure.
Yes it does. Source - use Spark SQL routinely. You're right that multiple small Gzipped files are not an ideal input source as it'll create a bunch of small tasks, but Spark definitely does support GZ.
Earlier quoted context omitted.
SparkSQL doesn’t support gz. Are your logs splittable on a file-by-file basis or are they in gz format? Where are your logs stored? Is that a distributed storage? Will SparkSQL not eat all of the bandwidth of it’s ethernet interfaces? Yeah; sure.
It's not so much that SparkSQL doesn't support gz as that gz is slow because you can't parallelize the reads. Regardless, parquet format in hdfs so yarn can allocate containers local to the chunk to be processed. Scales nicely.