Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS is not a dumb pipe

matt-rickard.com

Post reply on HN
Print view View original

61–70 of 90 posts

Re: AWS is not a dumb pipe

#61

Post by dimgl »

rglover wrote: post #23

AWS would be a far superior product if it trended toward being a dumb pipe. The "all the things" approach that AWS has taken has led to a lot of great ideas being poorly executed, leading to a mess and poor engineering. E.g., the new (V3) of the JavaScript SDK is such a clusterf*ck I almost can't believe they had the stones to ship it.

Wow... I had not touched the AWS SDK with JavaScript in a long while. This is actually very surprising. It looks significantly different for what seems like a marginal improvement from V2.

Re: AWS is not a dumb pipe

#62

Post by dilyevsky »

oneplane wrote: post #49
dilyevsky wrote: post #37

Earlier quoted context omitted.

Doing EB-scale s3 is really hard, throwing a few hundred terrabytes (or even couple petabytes) into s3 compatible storage on top of k8s is not that hard (getting easier every day). There’s numerous options available today even with enterprise support > The most AWS-native thing (or two things) that make most other offerings not work out all that great is the integrated IAM and Security Groups SGs are easily covered b…

You need SGs because not everything in the world lives inside a K8S cluster or flows through a CNI. And you need IAM because if you have 10 users and 10 resources but not every user is allowed to perform all actions on all resources you need policies. Policies apply (generally) on roles, actions and resources. IAM that only works on some random resources but not on others are pointless, unless you have a very small s…

I’m sure you can find a feature that minio/ceph/swift don’t support out of the box just as i can probably find something that’s very easy to do do/add in ceph/seaweedfs but will be on the back burner of your cloud tam for a few quarters/years or do it at a cost that can’t be beat by any volume commits. At the end of the day we are payed to solve business problems with set constraints so statements like “you need s3 bc everything else is hard” are kind of myopic

Re: AWS is not a dumb pipe

#63

Post by vm2196 »

rckrd wrote: post #9
FridgeSeal wrote: post #8

Earlier quoted context omitted.

I was just about to add a comment about this too. Also saying they “missed the Snowflake opportunity” doesn’t make sense either: they have Redshift? They don’t need to acquire Snowflake, they’re already the incumbent in the field.

Author here. They missed the Snowflake opportunity by having the wrong architecture for Redshift (decoupled storage and compute). They shifted to the Snowflake model in 2019, but the damage might already be done. For other the other services I listed, the main differentiators are mainly plugins/extensibility and developer experience.

I don't know what you are smoking here but let's take MongoDB vs DocumentDB or Elastic Cloud. Mongo built a 30B$ business around Atlas that is growing at a staggering rate. DocumentDB has a tiny share. Same with Elastic.

Also what the hell are you talking about with RedShift. Decoupled Compute and Storage is a far superior architecture. Here is an expert from snowflake.com - " "Snowflake was founded on the belief that tying compute and storage together is not an effective approach for limitless, seamless scaling."

Re: AWS is not a dumb pipe

#64

Post by Scarbutt »

oneplane wrote: post #30
gkop wrote: post #27

Earlier quoted context omitted.

Huh? Yes Aurora is highly proprietary, but S3 is a commodity in many use cases.

Commodity in usage, yes, but not on the service side. Doing S3 properly is really hard, having a service with a 'compatible API' doesn't make it the same as S3. Same goes for EKS (Yes, it looks and smells like a normal API server) and RDS. If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'. That sai…

If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'.

Must cheap providers have crappy networks compared to AWS.

Re: AWS is not a dumb pipe

#65

Post by anothernewdude »

To their own failure. It's a dumb pipe with lots of bad microtransactions and traps added.

The best features of AWS are the dumb ones. EC2, ECS, EKS, S3, Lambda.

The truly horrible ones are ones like Athena etc. where you are better of hosting your own solution on the dumb features. No self-hosted data services fails queries because the query has taken too long, and charges you for the privilege.

Re: AWS is not a dumb pipe

#66

Post by hodgesrm »

glogla wrote: post #33
FridgeSeal wrote: post #13

Earlier quoted context omitted.

Isn’t this a bit of a moot point now though? If you’re going to pay someone for a data warehouse, you can pay AWS some money, or you can pay Snowflake a heinous amount of money, for something not significantly better? Mongo, ES and Redis all come with open source versions, which shifts the appeal back away from the AWS offerings, especially with things like Elastic Cloud for Kubernetes which has made our ES cluster b…

Snowflake is technically far superior to Redshift. The performance and features are somewhere else. Even if you pay more in operating cost (which is not always the case, especially if you count Redshift DBA salary), you get to do what you actually need to do much faster instead of fighting the platform. Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower, not a leader.…

> Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower, not a leader.

Redshift was the first cloud data warehouse-as-a-service in the Amazon cloud. Every data warehouse since then has built on that concept. Speaking of innovation, Snowflake depends on object storage, which Amazon basically invented.

Re: AWS is not a dumb pipe

#67

Post by thejosh »

djbusby wrote: post #47
greiskul wrote: post #45

Earlier quoted context omitted.

I hate this practice that some companies do of testing their sql database usage... By using another database. For postgres, if you are using the jvm, you can use the amazing test containers library to test reasonably fast, with a postgres database that you can set up to be the same version as you use in production. Any database vendor should either support having their database used quickly in test like these, or at…

For PG you can just initdb in a tmpfs.

With elixir and ecto it wraps the whole thing in a transaction so it runs Async across multiple tests.

Re: AWS is not a dumb pipe

#68

Post by glogla »

hodgesrm wrote: post #66
glogla wrote: post #33

Earlier quoted context omitted.

Snowflake is technically far superior to Redshift. The performance and features are somewhere else. Even if you pay more in operating cost (which is not always the case, especially if you count Redshift DBA salary), you get to do what you actually need to do much faster instead of fighting the platform. Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower, not a leader.…

> Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower, not a leader. Redshift was the first cloud data warehouse-as-a-service in the Amazon cloud. Every data warehouse since then has built on that concept. Speaking of innovation, Snowflake depends on object storage, which Amazon basically invented.

Yes, and credit where credit's due, with Redshift Amazon changed analytics industry forever and they changed whole IT space forever with S3.

But since then, there's not much innovation in analytics space from them, is there?

Re: AWS is not a dumb pipe

#69

Post by glogla »

FridgeSeal wrote: post #53
glogla wrote: post #33

Earlier quoted context omitted.

Snowflake is technically far superior to Redshift. The performance and features are somewhere else. Even if you pay more in operating cost (which is not always the case, especially if you count Redshift DBA salary), you get to do what you actually need to do much faster instead of fighting the platform. Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower, not a leader.…

> Snowflake is technically far superior to Redshift. The performance and features are somewhere else From the discussions I’ve had about this before, I think I’m in the minority when I say I’m categorically unimpressed by Snowflakes performance. Add that to the hideous cost, and the worlds most aggressive sales/account management team and I’ve less than zero desire to ever deal with them again. > Snowflake is also te…

I think that depends on your target and especially scope.

ClickHouse demonstrates this well - it is incredibly fast and powerful, but also very limited. It has its own dialect of SQL incompatible with anything else. It doesn't even have a traditional query planner so you have to be expert to write fast queries. It has no update, no merge, no CTEs. To get most out of it, you have to think about data sorting, data types (is this LowCardinality String or just String?). And once you reach scale where single node can handle it (which is much more than single node Redshift could handle) you manage a stateful cluster which is a pain compared to storage/compute separated clusters like Presto/Trino over S3.

So ClickHouse is very good if it is in hands of dedicated team that is willing to learn CH in and out and is using it for a particular purpose. But you can't really setup ClickHouse, dump data into it and then send 100 random Data Analysts to "go forth and make reports". The barrier of entry and list of things that are different in ClickHouse is just too long.

Snowflake is exactly at the opposite end of "how much knowledge / how much of a dedicated team do I need to make use of this". Dumping data in and giving other teams access is something you can do easily. Scaling isn't a problem, users interfering with each other isn't a problem (you can give them their own compute that suspends when not in use), all the typical "data features" are there, backup is handled via time travel once they delete something they shouldn't, they don't have to understand data types or what is the best partitioning and sorting for the querying they are planning to do. It even has fancy web UI with charts and CSV upload.

You do pay a pretty penny for that, but there are many companies with a lot of money and many data problems to solve, so paying more for infrastructure and enabling a many of less technically skilled users (who are likely SMEs) is worth it.

Redshift sits somewhere in the middle. You still need DBAs, you still somewhat need to think about data layout, you still need to worry about managing backups and workload management where one using can slow down everyone else, etc. It feels like worst of all worlds.

In onprem world, Trino behaves more like Snowflake than ClickHouse, which has its uses - again if youre looking to build a general platform instead of dedicated application. Might be worth it even if you never get milisecond latency queries out of it.

Re: AWS is not a dumb pipe

#70

Post by pjmlp »

We are back into the timesharing days, and newer generations are finally getting that cloud platforms have replaced UNIX with hypervisors, language runtimes and cloud vendor specific APIs/services .

Most businesses don't care about dumb pipes, at the end of the day how the accounting comes out is what matters.

So many wasted projects trying to make the whole stack technology neutral and then it never changes during its lifetime, with added maintenance complexity to ease migration to something else that never got used.

Post reply on HN