Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS is not a dumb pipe

matt-rickard.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: AWS is not a dumb pipe

#81

Post by glogla »

zX41ZdbW wrote: post #79
glogla wrote: post #78

Earlier quoted context omitted.

Oh wow, I missed that. I guess I heard about the CTE somewhere and understood dbt-clickhouse not having ephemeral materialization as CH still not supporting it. I am still somewhat salty about not being able to do GROUP BY 1, 2, 3 :) > You are right that ClickHouse requires attention and skills to get the best performance. However, that includes fixed, low-latency use cases like real-time marketing that Snowflake sim…

> GROUP BY 1, 2, 3 It is not ANSI SQL. Nevertheless, it is already available in ClickHouse as well under `enable_positional_arguments` setting and we are considering making it enabled by default.

Ooooh.

Is there a way to turn unquoted column and table names case insensitive too? Thats like the second biggest annoyance.

Re: AWS is not a dumb pipe

#82

Post by hodgesrm »

glogla wrote: post #68
hodgesrm wrote: post #66

Earlier quoted context omitted.

> Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower, not a leader. Redshift was the first cloud data warehouse-as-a-service in the Amazon cloud. Every data warehouse since then has built on that concept. Speaking of innovation, Snowflake depends on object storage, which Amazon basically invented.

Yes, and credit where credit's due, with Redshift Amazon changed analytics industry forever and they changed whole IT space forever with S3. But since then, there's not much innovation in analytics space from them, is there?

Not so sure about that: https://aws.amazon.com/ground-station/. Enabling data collection from the rest of the solar system seems pretty cutting edge to me.

Re: AWS is not a dumb pipe

#83

Post by ljm »

acdha wrote: post #15
ljm wrote: post #7

Earlier quoted context omitted.

OpenSearch is a massively inferior offering compared to Elasticsearch too. It became outdated the moment it was forked, the documentation is lacking, and since you'll end up looking up ES docs and forgetting to switch to version 7.10, you'll get a nice reminder of everything new that has been added that you can't actually use. The only thing it has going for it is that it's managed and you're already on AWS, so you d…

> The only thing it has going for it is that it's managed and you're already on AWS, so you don't need to spend months working up a contract with a new vendor and doing the security audit dance. That's a pretty substantial win for many projects, similar to how popular RDS is while giving up the ability to get updates as fast as if you run your own servers. People pay a lot for stability and reduced staffing requireme…

It's functionally inferior, because if you go into it expecting modern elasticsearch you'll be sorely disappointed. It's only now that it's been renamed to OpenSearch that this becomes less of an issue, as the two technologies have completely diverged.

Anything AWS decides to run a managed version of will automatically have an advantage over the non-AWS equivalent but that's not a property of the technology itself really, it's just that you've already done all the paperwork and bureaucracy to use AWS in your org.

I mean, the word we're not using here but ought to is vendor lock-in. I don't really need the patronising remark about recognising other's needs either.

Re: AWS is not a dumb pipe

#84

Post by acdha »

ljm wrote: post #83
acdha wrote: post #15

Earlier quoted context omitted.

> The only thing it has going for it is that it's managed and you're already on AWS, so you don't need to spend months working up a contract with a new vendor and doing the security audit dance. That's a pretty substantial win for many projects, similar to how popular RDS is while giving up the ability to get updates as fast as if you run your own servers. People pay a lot for stability and reduced staffing requireme…

It's functionally inferior, because if you go into it expecting modern elasticsearch you'll be sorely disappointed. It's only now that it's been renamed to OpenSearch that this becomes less of an issue, as the two technologies have completely diverged. Anything AWS decides to run a managed version of will automatically have an advantage over the non-AWS equivalent but that's not a property of the technology itself re…

> It's functionally inferior, because if you go into it expecting modern elasticsearch you'll be sorely disappointed.

Or, for a large number of people, won't notice the difference. There are some important questions about how ElasticSearch's license & community works, and whether you're risking lock-in by using a managed service, but detail-free hyperbole contributes noise but not value to that discussion.

For example, what are the features you think no user of ElasticSearch could live without which are in ElasticSearch after 7.10 but not OpenSearch 1.1? What percentage of users depend on those features? How concerned are you about vendor lock-in with ElasticSearch's unilateral control of the project's roadmap? Do you think there's more or less risk from an open source project you can run anywhere which, you fear, will not be updated as frequently or from one which has a history of backwards-incompatible changes requiring you to stay current or fall out of support by popular clients?

Re: AWS is not a dumb pipe

#85

Post by ljm »

acdha wrote: post #84
ljm wrote: post #83

Earlier quoted context omitted.

It's functionally inferior, because if you go into it expecting modern elasticsearch you'll be sorely disappointed. It's only now that it's been renamed to OpenSearch that this becomes less of an issue, as the two technologies have completely diverged. Anything AWS decides to run a managed version of will automatically have an advantage over the non-AWS equivalent but that's not a property of the technology itself re…

> It's functionally inferior, because if you go into it expecting modern elasticsearch you'll be sorely disappointed. Or, for a large number of people, won't notice the difference. There are some important questions about how ElasticSearch's license & community works, and whether you're risking lock-in by using a managed service, but detail-free hyperbole contributes noise but not value to that discussion. For exampl…

These are all questions about the merits of either technology.

We've already identified that the merit of a managed AWS (but really any cloud provider) solution is that you've already done your due diligence for that provider, and that alone far outweighs whatever other merit you might consider.

Some of the other stuff is what I might consider if I had to make a case for approving a new vendor, but this is an overwhelming barrage of rhetorical questions that makes it difficult to have a reasoned conversation about.

I mean, come on... what percentage of users depend on those features? Unilateral control over a product roadmap? Am I supposed to actually know these things when stating my opinion that OpenSearch is lame in comparison to the original ElasticSearch, based on my anecdotal experience of dealing with the two? Do you have those answers yourself?

As to the risk, a fair question. There's currently a threat that ES client libraries will reject a connection to OS (and I recall this has already been done for some languages). Not the end of the world but the only people who have lost out from the licensing spat you've alluded to are the users. The risks of onboarding Elastic as a new vendor or self-hosting are well known and part of the usual discussion of trade-offs one will have.

Re: AWS is not a dumb pipe

#86

Post by acdha »

ljm wrote: post #85
acdha wrote: post #84

Earlier quoted context omitted.

> It's functionally inferior, because if you go into it expecting modern elasticsearch you'll be sorely disappointed. Or, for a large number of people, won't notice the difference. There are some important questions about how ElasticSearch's license & community works, and whether you're risking lock-in by using a managed service, but detail-free hyperbole contributes noise but not value to that discussion. For exampl…

These are all questions about the merits of either technology. We've already identified that the merit of a managed AWS (but really any cloud provider) solution is that you've already done your due diligence for that provider, and that alone far outweighs whatever other merit you might consider. Some of the other stuff is what I might consider if I had to make a case for approving a new vendor, but this is an overwhe…

You were making some absolute assertions, so yes, I would expect you to have some actual examples based on real experience. Asking you what homework you based that on wasn’t rhetoric but simply a question because I know multiple projects using OpenSearch who’ve had no problems other than Elastic sabotaging certain clients. It would be useful to know, for example, if there was a certain class of functionality or performance challenge where the difference is significant.

Re: AWS is not a dumb pipe

#87

Post by zX41ZdbW »

glogla wrote: post #81
zX41ZdbW wrote: post #79

Earlier quoted context omitted.

> GROUP BY 1, 2, 3 It is not ANSI SQL. Nevertheless, it is already available in ClickHouse as well under `enable_positional_arguments` setting and we are considering making it enabled by default.

Ooooh. Is there a way to turn unquoted column and table names case insensitive too? Thats like the second biggest annoyance.

Not yet. I will add it to the short list.

Re: AWS is not a dumb pipe

#88

Post by oneplane »

deanCommie wrote: post #56
xg15 wrote: post #46

Earlier quoted context omitted.

Ok, but by that logic it would be impossible to make any kind of analysis about AWS (or other cloud providers) in the sense that the author attempted to make. In the end I believe, the question is about lock-in and control. In a true "dumb pipe" offering, you could just take your software elsewhere and run essentially the same code without changes. Meanwhile, if you use various proprietary APIs, this is not the case.…

I think "lock-in" is a terrible term. You can avoid all lock-in by ignoring all sorts of proprietary features of any vendor, and leave ENDLESS benefits and optimizations on the table for the hypothetical likelyhood that you might one day have to "free yourself". Conversely, if you embrace those things wholeheartedly, and at some point find yourself wanting to break the relationship and do a full egress migration, eve…

I'd expand on that a little: 'lock-in' is not always what people think it is. Having all your 'stuff' in a binary database of an unknown format that only the vendor can read/write would be a lock-in, or if it's a known format but you don't have access to it.

If you have access to all the details of your systems you can always 'go elsewhere' but it becomes a matter of cost, knowledge and time. The balance is how much you want to spend on those, and what it gives you in return.

As an anecdote: I and the teams I manage avoid CDKs like the plague, they end up a version of IaC self-lock-in that makes it nearly impossible to integrate between vendors. Using tools like Terraform or perhaps Ansible or SaltStack (or Idem) still means you write your abstractions against a specific provider, but the code and data formats are standard enough that you can retain and re-use your knowledge and skills and depending on how much layering you apply you can reasonably easily migrate wherever you want to (from an infra perspective). Contents (unique data, that is) are always a different issue, even with full access you'll be thinking about egress cost. All of those issues are of course mostly annoying at scale, but that's also one of the reasons you'd go to AWS: for scaling options.

Re: AWS is not a dumb pipe

#89

Post by oneplane »

dilyevsky wrote: post #62
oneplane wrote: post #49

Earlier quoted context omitted.

You need SGs because not everything in the world lives inside a K8S cluster or flows through a CNI. And you need IAM because if you have 10 users and 10 resources but not every user is allowed to perform all actions on all resources you need policies. Policies apply (generally) on roles, actions and resources. IAM that only works on some random resources but not on others are pointless, unless you have a very small s…

I’m sure you can find a feature that minio/ceph/swift don’t support out of the box just as i can probably find something that’s very easy to do do/add in ceph/seaweedfs but will be on the back burner of your cloud tam for a few quarters/years or do it at a cost that can’t be beat by any volume commits. At the end of the day we are payed to solve business problems with set constraints so statements like “you need s3 b…

Unless you are in the business of reinventing wheels and building knowledge silos that will never transfer, none of what you wrote applies in general commercial business.

Re: AWS is not a dumb pipe

#90

Post by oneplane »

Scarbutt wrote: post #64
oneplane wrote: post #30

Earlier quoted context omitted.

Commodity in usage, yes, but not on the service side. Doing S3 properly is really hard, having a service with a 'compatible API' doesn't make it the same as S3. Same goes for EKS (Yes, it looks and smells like a normal API server) and RDS. If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'. That sai…

If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'. Must cheap providers have crappy networks compared to AWS.

That's true. And sometimes you get an odd mix of good network quality but crappy features, or the other way around and that isn't amazing to work with either.
Post reply on HN