Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Picks Diane Greene to Expand Its Cloud Business

nytimes.com

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Google Picks Diane Greene to Expand Its Cloud Business

#81

Post by josephjacks »

wstrange wrote: post #74
yeukhon wrote: post #59

Earlier quoted context omitted.

Because technology evolves too quickly to do cross-platform, cross-cloud solution and the demand for getting work done now is more demanding than ever. Tried to do it for two cloud platforms + on-premise, you will literally want to kill yourself for supporting your code three-way. Do it right once for one platform, and then next step. AWS isn't going away any time soon. But seriously, you don't just click buttons all…

Why not Kubernetes? It run's on all major cloud platforms, VM environments and bare metal. Supported by every major player except Amazon (for obvious reasons, why would they want to support something that makes it easy to migrate away from them?)

Kubernetes is supported on AWS by various companies in the industry, including my own, Kismatic (the enterprise Kubernetes company). We have been pushing for the team at ECS to adopt K8S as a standard framework on which to run containerized workloads. As Kubernetes gets adopted as a container cluster management standard for orchestrating and running microservice-oriented apps, all the cloud providers will need to support it based on customer demand, in the same way they are all supporting Docker due to the demand.

Re: Google Picks Diane Greene to Expand Its Cloud Business

#82

Post by nulltype »

vgt wrote: post #49
nulltype wrote: post #35

Earlier quoted context omitted.

In the ones they mentioned, GCE vs EC2, GCS vs S3, and BigQuery vs Redshift, they seem pretty comparable to me.

I like to think that BigQuery is far far ahead of Redshift in terms of performance + scale + manageability + cost (but i may be biased).

I don't have direct experience with Redshift, but I have heard some stuff. BigQuery has really variable performance for us though, not sure how it compares to Redshift.

Re: Google Picks Diane Greene to Expand Its Cloud Business

#83

Post by herriojr »

boundlessdreamz wrote: post #73
alexatkeplar wrote: post #48

Earlier quoted context omitted.

I think this sentence [1] helps to explain the difference: > When you create a subscription, the system establishes a sync point. That is, your subscriber is guaranteed to receive any message published after this point. [1] https://cloud.google.com/pubsub/subscriber With Kafka or Kinesis, I can write events to a stream/topic completely independently of any consumer. I can then bring as many consumers online as I want…

I'm not familiar with Kafka 1. Can you direct the consumer to a point in stream? (ideally time based i.e messages from 16 Nov UTC) 2. Can old events be auto removed defined by rules?

I haven't played with kafka in a while, but basically,

1. each group id represents a point in the stream that a consumer is processing off of. You could technically have multiple processes consuming off of a single group id.

2. there was a configuration on time to keep things there as well as space if I remember correctly, but basically, there has to be. There's a pretty hard limit on what all you can store on disk.

edit: changed consumer id to group id. If you want more info, feel free to ping me about the ecosystem

Re: Google Picks Diane Greene to Expand Its Cloud Business

#84

Post by alexatkeplar »

jganetsk wrote: post #80
alexatkeplar wrote: post #48

Earlier quoted context omitted.

I think this sentence [1] helps to explain the difference: > When you create a subscription, the system establishes a sync point. That is, your subscriber is guaranteed to receive any message published after this point. [1] https://cloud.google.com/pubsub/subscriber With Kafka or Kinesis, I can write events to a stream/topic completely independently of any consumer. I can then bring as many consumers online as I want…

Cloud Pub/Sub engineer here. You can create as many consumers as you want. You can create them offline and bring them up and down whenever you want. Each consumer will receive a full copy of the stream, starting with its sync point (subscriber creation). Each message is delivered, and redelivered, to each consumer until that consumer acks that message. If I understand your point correctly, the only expectation we hav…

From your comment it sounds like you haven't used Kinesis or Kafka yourself - rather than take my word for it, I'd suggest your team give both of those platforms a serious try-out to really understand the capability gaps. I'd be surprised if a lot of your [prospective] customers weren't asking for these kinds of unified log capabilities in Cloud Pub/Sub.

Re: Google Picks Diane Greene to Expand Its Cloud Business

#85

Post by devsquid »

shawn-butler wrote: post #79
devsquid wrote: post #11

Earlier quoted context omitted.

Their managed SQL service doesn't support Postgresql and POSTGIS, its the only thing stopping me from move over to GCN.

Trust me, it won't be the only thing. It's just the first roadblock.

Oh? I have found the GCN to be quite impressive. Their prices are very cheap and i've found their international offerings to be next to none. What issues have you had?

Re: Google Picks Diane Greene to Expand Its Cloud Business

#86

Post by nigelk »

mikecb wrote: post #24
nigelk wrote: post #21

Earlier quoted context omitted.

There are some great aspects to Google Compute, but I hope this change results in catch up on features needed for corporate adoption compared to AWS. The lack of a billing API and the lack of centralized management are really painful if you're trying to adopt it across an organization.

There is a billing API: https://cloud.google.com/billing/ There have been some recent introductions of account and key management that help too.

Excellent, thanks. Hopefully we'll see tools like Cloudability support it then.

I'll check it out again, really the frustrating management aspect was the lack of org oversight over multiple projects started within your domain.

Re: Google Picks Diane Greene to Expand Its Cloud Business

#87

Post by SEJeff »

yeukhon wrote: post #78
wstrange wrote: post #77

Earlier quoted context omitted.

> is Google going to make their environment so compatible with other cloud platform? Yes. Google's strategy with Kubernetes is to commoditize the cloud - making them all functionally interchangeable. Write to Kubernetes, and your app runs on AWS, GCE, Azure, etc... They are betting that they can deliver raw CPU cycles, network bandwidth, lower latency etc. - better/faster/cheaper than their competitors.

I am not familiar with GCE so I can be absolutely wrong, but my way of thinking is that any time anyone says something can run the open source version anywhere, true, but when it comes to offering a paid service, making some technology native to the platform, means there can be difference such as customized APIs or customized features which may never get backport into the community/open source version.

That is very clearly not how they are working with Kubernetes. They are doing it entirely in the open and even recently hired one of their huge community contributors (shoutout to Kelsey Hightower) to the team. If they were doing an "open core" version, my money would be on Kelsey speaking out against it. Besides, given their goal of making the cloud a commodity, they can't do what you say.

Re: Google Picks Diane Greene to Expand Its Cloud Business

#88

Post by univalent »

x0x0 wrote: post #6
univalent wrote: post #5

An easy area of improvement is documentation, samples etc. They are seriously lagging behind AWS and Azure. Becomes hard to choose Google because the cost of learning is higher.

I dunno, I've found aws documentation to be atrocious. For example, I've had a very hard time figuring out how to do things like reattaching ebs volumes to ec2 spot instances. Maybe it's because I don't use aws much, but I get frustrated and have a hard time understanding how simple tasks need to be so complex. I wonder how effective the Microsoft style API lockin strategy will be for aws. My personal guess is very e…

Yes, it isn't perfect but they have more samples for every language, tutorials etc. They also have quite a few training videos etc. online.

Re: Google Picks Diane Greene to Expand Its Cloud Business

#89

Post by jganetsk »

alexatkeplar wrote: post #84
jganetsk wrote: post #80

Earlier quoted context omitted.

Cloud Pub/Sub engineer here. You can create as many consumers as you want. You can create them offline and bring them up and down whenever you want. Each consumer will receive a full copy of the stream, starting with its sync point (subscriber creation). Each message is delivered, and redelivered, to each consumer until that consumer acks that message. If I understand your point correctly, the only expectation we hav…

From your comment it sounds like you haven't used Kinesis or Kafka yourself - rather than take my word for it, I'd suggest your team give both of those platforms a serious try-out to really understand the capability gaps. I'd be surprised if a lot of your [prospective] customers weren't asking for these kinds of unified log capabilities in Cloud Pub/Sub.

We hear you.

Let me see if I'm understanding the criticism: when creating a consumer, the sync point of a new consumer really should start from the very beginning of the topic, at a predictable explicit start point, rather than at the current end of the topic. This makes a lot of sense, and yes, there is a disconnect between the models. We think the capabilities you are talking about are great and those use cases are important. All I can say is keep your eyes open.

We went with defaults from Google's internal use of Pub/Sub, which is older than the public release of Kinesis and Kafka. Internal use involves an approach where topics and consumers are very long-lived. Topics are high throughput, in terms of bytes published per unit time. Retaining all messages and starting consumers from the very beginning wasn't a sensible default; our focus was more centered on making sure that, once topics and consumers were set up, consumers could keep up over time.

One example use case to help illustrate this thinking is doing real-time sentiment analysis on tweets: https://www.youtube.com/watch?v=O3mfuc-syTI

In the work described by that video, they were essentially publishing tweets in real time into a Cloud Pub/Sub topic, thus making an "all tweets on Twitter in realtime" topic. This is a great example of a topic where producers and consumers are completely decoupled from each other. It doesn't necessarily make sense to retain all tweets forever by default (although there certainly are use cases for that). There are plenty of use cases where a consumer might want to say "ok, please start retaining all tweets made from here on out" rather than starting from a specific tweet.

Re: Google Picks Diane Greene to Expand Its Cloud Business

#90

Post by alexatkeplar »

jganetsk wrote: post #89
alexatkeplar wrote: post #84

Earlier quoted context omitted.

From your comment it sounds like you haven't used Kinesis or Kafka yourself - rather than take my word for it, I'd suggest your team give both of those platforms a serious try-out to really understand the capability gaps. I'd be surprised if a lot of your [prospective] customers weren't asking for these kinds of unified log capabilities in Cloud Pub/Sub.

We hear you. Let me see if I'm understanding the criticism: when creating a consumer, the sync point of a new consumer really should start from the very beginning of the topic, at a predictable explicit start point, rather than at the current end of the topic. This makes a lot of sense, and yes, there is a disconnect between the models. We think the capabilities you are talking about are great and those use cases are…

Thanks for the detailed explanation jganetsk.

> when creating a consumer, the sync point of a new consumer really should start from the very beginning of the topic, at a predictable explicit start point, rather than at the current end of the topic

I'll talk about Kinesis because that's the technology we use more at Snowplow. When creating a Kinesis consumer, I can specify whether I want to start reading from a) TRIM_HORIZON (which is the earliest events in the stream which haven't yet been expired aka "trimmed"), b) LATEST which is the Cloud Pub/Sub capability, c) AT_SEQUENCE_NUMBER {x} which means from the event in the stream with the given offset ID or d) AFTER_SEQUENCE_NUMBER {x} which is the event immediately after c).

Kinesis streams or Kafka topics don't themselves care about the progress of any individual consumer - consumers are responsible for tracking their own position in the stream via sequence numbers / offset IDs.

> It doesn't necessarily make sense to retain all tweets forever by default (although there certainly are use cases for that)

Completely agree. I think a good point of distinction between pub/sub systems and unified log is: use pub/sub when the messages are a means-to-an-end (which is feeding one or more downstream apps); use unified log when the events are an end-in-themselves (i.e. you would still want to preserve the events even if there were no consumers live).

Anyway, I could talk about this stuff all day :-) - if you'd like to chat further, my details are in my profile!

Post reply on HN