Judging from the code samples they showed during the keynote, I'd guess that Google Cloud Dataflow is based on (or an extension of, or a public version of...) FlumeJava, described in this PLDI 2010 paper: http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F...
Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
21–30 of 33 posts
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#22It looks like an attempt to respond to AWS kinesis which was released last year. The monitoring stuff seems to be about the software they got when they bought stackdriver.
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#23Earlier quoted context omitted.
I believe it's Kinesis: http://aws.amazon.com/kinesis/
+1. It's Kinesis.
DataFlow is not Kinesis. It's more like Kinesis plus Esper plus BigQuery and you still wouldn't have one set of queries to run against streaming and batch data like you do with DataFlow.
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#24This is one space where Google really excels. We're in the AWS ecosystem, and the database offerings are really subpar. DynamoDB, which I originally expected to be somewhat comparable to MongoDB, is an incredibly frustrating (and expensive) product to use. AWS Data Pipeline is extremely confusing and very expensive as well. AWS's offerings really lag behind Google's offerings (like BigQuery) in this space. Hopefully…
If you are in AWS there is also the four RDS (Oracle, MySQL, PostresSQL, SQL Server) options as well as RedShift. Also the best thing about AWS is that there are so many third party choices e.g. MongoLab, MongoHQ, Instaclustr, Cloudant. Databases is not the area I would be choosing Google for.
I think App Engine's Datastore is generally an under-appreciated gem. Possibly because you have to use App Engine to use it without sacrificing performance, maybe because it's not easy enough to use if all you have is some JavaScript + JSON and don't want or know how to write Python/Java/Go.
But it's actually the only generally available product I know of that solves all the hard problems (availability, partition tolerance, some - but well defined - consistency with cross entity transactions) with zero hassle for you.
If you read through http://aphyr.com/tags/Jepsen, you get some appreciation for how hard this is to pull off without running into operational nightmares (massive data loss, split brains, etc).
Disclaimer: I work for Google, though not on Datastore.
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#25Earlier quoted context omitted.
We've had good luck with DynamoDB, but it could be that it just fits our use case very well. What sort of frustration were you running into? (Honestly interested to avoid trouble down the line)
Most recently: hot hash key. DynamoDB uses the object to be persisted's hash key to route it to the right data cluster. We're a SaaS company with lots of tiny customers and a few very large customers. We need to keep an index to show a specific customer only their data. That means the index for our largest customers gets hit a lot. The problem with this structure is we have to pay as if all of our customers were as p…
Hot keys are going to be the same with Mongo. The issue sounds more like that you're using a single key per customer than anything else.
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#26Earlier quoted context omitted.
What type of monitoring? We use Stackdriver, which Google just recently bought.
I probably will end up building the bare minimum to meet my needs and moving on tbh. It was basically a monitoring/metrics system to merge how I handle the monitoring of crons, work queue, system metrics, analytics, etc. into a single service. Right now, I'm stuck using 3. Sure, I could just build something to merge it together ... but at that point, I'm halfway to building my own.
Hi, I am developer and hacker and wanted to see if i can offer help here. The reason, wanted to see what would be the typical needs and use cases and learn from the experience.
I can be contacted on sid4it@gmail.com
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#27Earlier quoted context omitted.
What type of monitoring? We use Stackdriver, which Google just recently bought.
I probably will end up building the bare minimum to meet my needs and moving on tbh. It was basically a monitoring/metrics system to merge how I handle the monitoring of crons, work queue, system metrics, analytics, etc. into a single service. Right now, I'm stuck using 3. Sure, I could just build something to merge it together ... but at that point, I'm halfway to building my own.
I was about to do the same thing. App Engine sorely lacks those features currently, so I'm very excited for this (assuming it has good support for App Engine in addition to Compute Engine which I saw in the keynote).
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#28The streaming data stuff looks extremely interesting. My main concern is around cost, unfortunately many of these things are great if you've got a massive data problem but not particularly worth it if you've got much smaller data.
I'm in a rather awkward phase of having small enough data that I don't need "Scale to 1000 machines!", I want just one or a few machines occasionally but managed for me (turn on, run code, shut off). Tutum works very well for this, but I'd like to use more of the ecosystem available at Google or AWS (pay-per-usage datastorage, for example). GCE is pretty decent, but a bit awkward, although the new docker support helps (but I've had problems getting it even working).
Maybe this is my magic bullet :)
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#29The streaming data stuff looks extremely interesting. My main concern is around cost, unfortunately many of these things are great if you've got a massive data problem but not particularly worth it if you've got much smaller data. I'm in a rather awkward phase of having small enough data that I don't need "Scale to 1000 machines!", I want just one or a few machines occasionally but managed for me (turn on, run code,…
I'm using MITs StarCluster to quickly spin up a bunch of AWS Spot Instances, run some calculations and shut them down again.
http://star.mit.edu/cluster/ http://www.youtube.com/watch?v=2Ym7epCYnSk
Re: Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products
#30Earlier quoted context omitted.
Most recently: hot hash key. DynamoDB uses the object to be persisted's hash key to route it to the right data cluster. We're a SaaS company with lots of tiny customers and a few very large customers. We need to keep an index to show a specific customer only their data. That means the index for our largest customers gets hit a lot. The problem with this structure is we have to pay as if all of our customers were as p…
Hot keys are going to be the same with Mongo. The issue sounds more like that you're using a single key per customer than anything else.
Yeah, I think all nosql db's will have that issue if you have extremely unbalanced sharding. This is an application level fault and should be solved there.
But, the thing that is extra bad about dynamo db is how they are paying for 10x higher provisioning as a stop gap, and still getting throttled. That sucks.
FWIW we're using dynamo db and we love it. Pro tip: setup dynamic-dynamodb and let it autoscale for you in realtime. http://aws.amazon.com/blogs/aws/auto-scale-dynamodb-with-dyn...