31–40 of 76 posts
This is off-topic, but I can't help myself: Slack, Hive, Presto, Spark, Sqooper, Kafka, Secor, Thrift, Parquet. I sometimes can't tell the difference between real Silicon Valley product names and parodies. I'm starting to miss the days when it was all just letters and numbers.
I give up. Which one's the real one?
We are implementing a very similar architecture, and have decided to use Avro for schema validation / serialization, rather than Parquet. Does anyone have experience with both that can talk to their strengths / weaknesses?
This is off-topic, but I can't help myself: Slack, Hive, Presto, Spark, Sqooper, Kafka, Secor, Thrift, Parquet. I sometimes can't tell the difference between real Silicon Valley product names and parodies. I'm starting to miss the days when it was all just letters and numbers.
Seems like a pretty typical set of problems. Dependency conflicts hard. Schema evolution hard. Upgrades hard. The big data space still feels like an overengineered, fractured, buggy mess to me. I was hoping spark would simplify the user experience but it's as much of a clusterf*ck as anything else. How hard can fast, reliable distributed computation and storage for petabytes of data be? He said ironically.
IMO one major problem is integration between different projects. Like you said, its a hard problem, and any solution typically depends on many many different open source projects because of the scope of challenges. All of those projects go forward without much coordination between the teams because they're open source. Then we end up in this fun, fun clusterfuck.
We are implementing a very similar architecture, and have decided to use Avro for schema validation / serialization, rather than Parquet. Does anyone have experience with both that can talk to their strengths / weaknesses?
We're actually having a debate now as we're starting to process larger datasets as to whether or not we should keep everything on S3 or start using HDFS w/ Hive. I'm curious if you guys considered HDFS and why you decided to go strictly with S3, and additionally, are there any issues you encounter with S3.
We've considered HDFS, but we really liked the idea of having compute only clusters and have our data kept completely separate. Clusters failure happen and having data on S3 makes us worry less if a cluster goes down. Just spin up a new one and you're good to go. There is a bit of more latency when using S3 compared to HDFS, but it's not bad and the benefits overcame that. We do have a couple of jobs that store some…
Earlier quoted context omitted.
IMO one major problem is integration between different projects. Like you said, its a hard problem, and any solution typically depends on many many different open source projects because of the scope of challenges. All of those projects go forward without much coordination between the teams because they're open source. Then we end up in this fun, fun clusterfuck.
There is some sort of hope, Apache Arrow is (in my opinion) a step in the right direction - A common In-Memory data layer for storage and data analysis systems? Yes please. It's important to start thinking about how all these big data storage/analytics tools can bridge the gap between themselves, hopefully projects like Apache Arrow will help... As long as there is adoption.
Earlier quoted context omitted.
IMO one major problem is integration between different projects. Like you said, its a hard problem, and any solution typically depends on many many different open source projects because of the scope of challenges. All of those projects go forward without much coordination between the teams because they're open source. Then we end up in this fun, fun clusterfuck.
There is some sort of hope, Apache Arrow is (in my opinion) a step in the right direction - A common In-Memory data layer for storage and data analysis systems? Yes please. It's important to start thinking about how all these big data storage/analytics tools can bridge the gap between themselves, hopefully projects like Apache Arrow will help... As long as there is adoption.
In general, I've learned to be skeptical of any new big data solution. Hadoop and hive are clumsy but as someone on my team said "they've found and fixed the tens of thousands of bugs".
It seems to take five years before any significant new solution is stable and reliable enough to be used on large, complex workloads.
Which makes me really uncertain how we get out of this situation. Maybe something like arrow is a silver bullet that fixes everything with minimal complexity and thus few bugs. But I'm skeptical.