You can try Qubole [0]. The pricing is a small percentage of what you pay to the cloud provider, so it's predictable to an extent.
Disclosure: I work here.
Ask HN: What are the alternatives of hosting Apache Spark?
11–18 of 18 posts
Re: Ask HN: What are the alternatives of hosting Apache Spark?
#12If your spark jobs are mostly batch workloads, that can tolerate moderately infrequent failures and restarts, try using google dataproc with preemptible vms or amazon emr using spot instances. Depending on your use case, you might spend many times less than you would using regular VMs. Many instances that are several dollars an hour on AWS can be used for a fraction of the price. Its also fairly easy to automate the…
I find the EMR markup to be substantial; if I weren't working in a corporation, I would stand up my own spark clusters, e.g. using spark-ec2
Re: Ask HN: What are the alternatives of hosting Apache Spark?
#13We're using Spark on EMR with Data Pipeline to do ETL and to run Scheduled Jobs. Data pipelines terminates the cluster once ETL or job gets completed, helps us a lot to save cost.
Re: Ask HN: What are the alternatives of hosting Apache Spark?
#14If your spark jobs are mostly batch workloads, that can tolerate moderately infrequent failures and restarts, try using google dataproc with preemptible vms or amazon emr using spot instances. Depending on your use case, you might spend many times less than you would using regular VMs. Many instances that are several dollars an hour on AWS can be used for a fraction of the price. Its also fairly easy to automate the…
I find the EMR markup to be substantial; if I weren't working in a corporation, I would stand up my own spark clusters, e.g. using spark-ec2
It is, but you can run the bare minimum number of core nodes (3 I think?) and use spot instances for any others.
At a previous job, we just built our own ec2 image that ran spark in standalone mode for ephemeral spark clusters, and it was wonderful and cheap. And the clusters launched very fast compared to EMR.
Re: Ask HN: What are the alternatives of hosting Apache Spark?
#15Re: Ask HN: What are the alternatives of hosting Apache Spark?
#16what part of it is unpredictable? I guess if you know how much VMs or EC2 you're planning on using its the same pricing model as Dataproc or EMR
Re: Ask HN: What are the alternatives of hosting Apache Spark?
#17Google Dataproc - very good and very soon they will release kubernetes as the manager instead of yarn.
"very soon they will release kubernetes as the manager instead of yarn"
Do you have more details on this?
Re: Ask HN: What are the alternatives of hosting Apache Spark?
#18Google Dataproc - very good and very soon they will release kubernetes as the manager instead of yarn.
"very soon they will release kubernetes as the manager instead of yarn" Do you have more details on this?