Ask HN: What are the alternatives of hosting Apache Spark?

Check out Azure Databricks

You could give AWS EMR a shot, it probably doesn't offer as much as databricks but should have consistent pricing.

Run Spark on a managed Kubernetes like GKE? There is experimental support for using Kubernetes as the cluster manager.

https://apache-spark-on-k8s.github.io/userdocs/index.html

If your spark jobs are mostly batch workloads, that can tolerate moderately infrequent failures and restarts, try using google dataproc with preemptible vms or amazon emr using spot instances.

Depending on your use case, you might spend many times less than you would using regular VMs. Many instances that are several dollars an hour on AWS can be used for a fraction of the price.

Its also fairly easy to automate the region selection and bid (on AWS that is, not sure about gcloud).

If you need streaming, obviously this might not be the way to go.

So I've been knee deep in glue for the past month. Great service but we really need more examples and an easier dev flow. I wasn't able to get consistent results between juypter and dev endpoint as i was with the glue running the script directly. So i spent a lot of wasted time waiting for glue to run my job that eventually failed and then getting a less than helpful error message. Now that i have my jobs running and orchestrated with step functions it's a thing of beauty but should have taken 1/10 of the time if i had good examples and a proper environment.