Performance in Big Data Land: Every CPU cycle matters
eng.localytics.com
Performance in Big Data Land: Every CPU cycle matters
1–10 of 54 posts
Re: Performance in Big Data Land: Every CPU cycle matters
#2Am I misunderstanding something? If one CPU cycle accounts for 27 seconds, then the savings of 10 seconds suggest we saved one half of a CPU cycle per iteration? Or do the queries not touch every row?
Optimizing data types and minimizing locks seem like general optimization tips, I was hoping for more advanced techniques for 100B rows.
Re: Performance in Big Data Land: Every CPU cycle matters
#3I might suggest a new definition for "Big Data" - Data, whose size is greater than fits in one machine's memory.
Re: Performance in Big Data Land: Every CPU cycle matters
#4I might suggest a new definition for "Big Data" - Data, whose size is greater than fits in one machine's memory.
Some propose an even stronger definition: when the indices for accessing said data no longer fits in one machine's memory.
Re: Performance in Big Data Land: Every CPU cycle matters
#5[deleted]
Re: Performance in Big Data Land: Every CPU cycle matters
#6Am I misunderstanding something? If one CPU cycle accounts for 27 seconds, then the savings of 10 seconds suggest we saved one half of a CPU cycle per iteration? Or do the queries not touch every row? Optimizing data types and minimizing locks seem like general optimization tips, I was hoping for more advanced techniques for 100B rows.
One CPU cycle per row saves 27 seconds of one CPU's time. That 10 seconds was saved on every CPU in the cluster. So if there were 50 CPUs, that's 500 seconds, or ~20 cycles per record, by the original calculation.
In reality, the change in data type probably optimized disk access more than it did number of CPU cycles. That can often be more of a bottleneck.
Re: Performance in Big Data Land: Every CPU cycle matters
#7I might suggest a new definition for "Big Data" - Data, whose size is greater than fits in one machine's memory.
Obligatory half-joke:
The most commonly used definition deduced from reading online articles is, in terms of size
"More data than naively fits into the memory of my (midrange) laptop using a high overhead platform"
or in terms of speed
"More data per second than can be handled using the same naive database code we used in the 90s for our website's comment section"
Re: Performance in Big Data Land: Every CPU cycle matters
#8Totally agree with the "Every CPU cycle matters". It might be more easier to save cpu cycle by saving I/O, utilizing data locality (with in datacenter racks) or even better serialization (binary, columnar or indexed).
Reducing locking and using shorter data type seem inadequate for the "Big Data" scene.
Re: Performance in Big Data Land: Every CPU cycle matters
#9Then why big data land is dominated by JVM-based frameworks?
Re: Performance in Big Data Land: Every CPU cycle matters
#10I hope later posts in this series explore Linux perf_events or flame graphs, which is the origin of the (unattributed) background image (http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html). :)