Live data from Hacker News

AWS is not a dumb pipe

matt-rickard.com

71–80 of 90 posts

Re: AWS is not a dumb pipe

#71
post #10
post #7

Earlier quoted context omitted.

OpenSearch is a massively inferior offering compared to Elasticsearch too. It became outdated the moment it was forked, the documentation is lacking, and since you'll end up looking up ES docs and forgetting to switch to version 7.10, you'll get a nice reminder of everything new that has been added that you can't actually use. The only thing it has going for it is that it's managed and you're already on AWS, so you d…

> so you don't need to spend months working up a contract with a new vendor and doing the security audit dance. That's a very big moat. Many decision makers are risk-averse w.r.t to infrastructure vendors and don't mind paying (or making someone else pay) a premium for that. The only thing that changed in the saying "no one was fire for choosing IBM" is the name.

> The only thing that changed in the saying "no one was fire for choosing IBM" is the name.

Yes. IBM, then Oracle, then AWS. Wonder who's next.

Re: AWS is not a dumb pipe

#72
Nothing is a dumb pipe on today's internet. Remember "the network interprets censorship as damage and routes around it"? Yeah, that went out like JNCO jeans. Today's network interprets information unpleasant to the surveillance-industrial complex as noise and filters it out. And the information it does allow, it is very interested in surveilling and price-tiering. Expecting anything to be a dumb pipe in today's world is a pipe dream.

Re: AWS is not a dumb pipe

#73
post #23

AWS would be a far superior product if it trended toward being a dumb pipe. The "all the things" approach that AWS has taken has led to a lot of great ideas being poorly executed, leading to a mess and poor engineering. E.g., the new (V3) of the JavaScript SDK is such a clusterf*ck I almost can't believe they had the stones to ship it.

Like a sibling comment I’d also like to know what was bad about the SDK. It seems.. alright? That alone is good enough for me considering its implementation is autogenerated from something else.

It also has tree-shakeable imports which v2 doesn’t have, and a very extensible looking middleware architecture which I’ve yet to need but good to know that it’s there if I do. The auto-generated docs leave something to be desired, but are passable, with a few extra clicks here and there.

Re: AWS is not a dumb pipe

#74
post #69

Earlier quoted context omitted.

> Snowflake is technically far superior to Redshift. The performance and features are somewhere else From the discussions I’ve had about this before, I think I’m in the minority when I say I’m categorically unimpressed by Snowflakes performance. Add that to the hideous cost, and the worlds most aggressive sales/account management team and I’ve less than zero desire to ever deal with them again. > Snowflake is also te…

I think that depends on your target and especially scope. ClickHouse demonstrates this well - it is incredibly fast and powerful, but also very limited. It has its own dialect of SQL incompatible with anything else. It doesn't even have a traditional query planner so you have to be expert to write fast queries. It has no update, no merge, no CTEs. To get most out of it, you have to think about data sorting, data type…

> ClickHouse demonstrates this well - it is incredibly fast and powerful, but also very limited.

Your examples are not entirely correct. ClickHouse introduced CTEs in early 2021. I use them constantly. ClickHouse SQL is turning into a superset of standard SQL at least for queries. Most SELECT syntax, including window functions, just works. ClickHouse does have updates; but they are asynchronous. This too is being fixed. Synchronous DELETEs will be available this year, UPDATE is next. One big issue for my money is distributed joins. They still require a lot of reasoning about data locality.

You are right that ClickHouse requires attention and skills to get the best performance. However, that includes fixed, low-latency use cases like real-time marketing that Snowflake simply does not handle. ClickHouse and Snowflake aren't interchangeable for these use cases.

Re: AWS is not a dumb pipe

#75
post #23

AWS would be a far superior product if it trended toward being a dumb pipe. The "all the things" approach that AWS has taken has led to a lot of great ideas being poorly executed, leading to a mess and poor engineering. E.g., the new (V3) of the JavaScript SDK is such a clusterf*ck I almost can't believe they had the stones to ship it.

> AWS would be a far superior product if it trended toward being a dumb pipe.

"Superior" by which measure? AWS is making money hand over fist. Companies use AWS because they think it's better than alternatives.

Re: AWS is not a dumb pipe

#76
post #9

Earlier quoted context omitted.

I was just about to add a comment about this too. Also saying they “missed the Snowflake opportunity” doesn’t make sense either: they have Redshift? They don’t need to acquire Snowflake, they’re already the incumbent in the field.

Author here. They missed the Snowflake opportunity by having the wrong architecture for Redshift (decoupled storage and compute). They shifted to the Snowflake model in 2019, but the damage might already be done. For other the other services I listed, the main differentiators are mainly plugins/extensibility and developer experience.

Are you saying AWS does not make money off the infrastructure Snowflake consumes? Snowflake consumes a lot of compute, plus there are lots of other SaaS services that integrate with it and also need compute, networking, etc.

It's my understanding the margins on compute at least are pretty good. So it seems as if AWS wins either way.

Re: AWS is not a dumb pipe

#77

Earlier quoted context omitted.

Yeah, I would be really careful about relying on something like Aurora and planning to migrate to Cloud SQL or another vendor's managed PostgreSQL. Apps couple very tightly to implementation quirks of their database, and even the tiniest difference between implementations is going to cause problems when you migrate. (For example, we used to test our Postgres app with SQLite. We found that each query engine treats "WH…

I hate this practice that some companies do of testing their sql database usage... By using another database. For postgres, if you are using the jvm, you can use the amazing test containers library to test reasonably fast, with a postgres database that you can set up to be the same version as you use in production. Any database vendor should either support having their database used quickly in test like these, or at…

Which library are you referring to?

I'm new to jvm programming, and was wondering about this as I'm used to rails where the test db is spun up on your behalf and is usually the same db

Re: AWS is not a dumb pipe

#78
post #69

Earlier quoted context omitted.

I think that depends on your target and especially scope. ClickHouse demonstrates this well - it is incredibly fast and powerful, but also very limited. It has its own dialect of SQL incompatible with anything else. It doesn't even have a traditional query planner so you have to be expert to write fast queries. It has no update, no merge, no CTEs. To get most out of it, you have to think about data sorting, data type…

> ClickHouse demonstrates this well - it is incredibly fast and powerful, but also very limited. Your examples are not entirely correct. ClickHouse introduced CTEs in early 2021. I use them constantly. ClickHouse SQL is turning into a superset of standard SQL at least for queries. Most SELECT syntax, including window functions, just works. ClickHouse does have updates; but they are asynchronous. This too is being fix…

Oh wow, I missed that. I guess I heard about the CTE somewhere and understood dbt-clickhouse not having ephemeral materialization as CH still not supporting it.

I am still somewhat salty about not being able to do GROUP BY 1, 2, 3 :)

> You are right that ClickHouse requires attention and skills to get the best performance. However, that includes fixed, low-latency use cases like real-time marketing that Snowflake simply does not handle. ClickHouse and Snowflake aren't interchangeable for these use cases.

Yup, that was my point.

Re: AWS is not a dumb pipe

#79
post #78

Earlier quoted context omitted.

> ClickHouse demonstrates this well - it is incredibly fast and powerful, but also very limited. Your examples are not entirely correct. ClickHouse introduced CTEs in early 2021. I use them constantly. ClickHouse SQL is turning into a superset of standard SQL at least for queries. Most SELECT syntax, including window functions, just works. ClickHouse does have updates; but they are asynchronous. This too is being fix…

Oh wow, I missed that. I guess I heard about the CTE somewhere and understood dbt-clickhouse not having ephemeral materialization as CH still not supporting it. I am still somewhat salty about not being able to do GROUP BY 1, 2, 3 :) > You are right that ClickHouse requires attention and skills to get the best performance. However, that includes fixed, low-latency use cases like real-time marketing that Snowflake sim…

> GROUP BY 1, 2, 3

It is not ANSI SQL.

Nevertheless, it is already available in ClickHouse as well under `enable_positional_arguments` setting and we are considering making it enabled by default.

Re: AWS is not a dumb pipe

#80
post #23

AWS would be a far superior product if it trended toward being a dumb pipe. The "all the things" approach that AWS has taken has led to a lot of great ideas being poorly executed, leading to a mess and poor engineering. E.g., the new (V3) of the JavaScript SDK is such a clusterf*ck I almost can't believe they had the stones to ship it.

What’s not working for you with the V3 JS SDK?

A few irks:

- Auto-generated Typescript documentation that's unintelligible (Lord Jesus help me).

- Weird naming of imported functions (e.g., appending "Command" to the end of imports).

- The need to create instances of requests (commands) and then "send" them to the API vs. just passing an object of options. The irk here is that it breaks the affordances around API design.

For example, using S3 used to look like...

---

import AWS from 'aws-sdk';

const s3 = new AWS.S3({ ... });

s3.putObject({

  Bucket: 'some-bucket',

  Body: 'data buffer',

  Key: 'file-name',
}).promise().then((response) => { // Handle URL here. });

---

Now, it looks like...

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const client = new S3Client(config);

const command = new PutObjectCommand(input);

const response = await client.send(command);

---

This example is taken straight from docs (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clien...). config and input are not described anywhere near this example, meaning you have to go hunt for them.

---

The gold standard I reference for API design is Stripe. They have just as complex an API as anyone, and yet, they don't go out of their way to overcomplicate things. For example, this is stupid simple and obvious:

const stripe = require('stripe')('sk_test_4eC39HqLyjWDarjtT1zdp7dc');

// `source` is obtained with Stripe.js; see https://stripe.com/docs/payments/accept-a-payment-charges#we...

const charge = await stripe.charges.create({ amount: 2000, currency: 'usd', source: 'tok_amex', description: 'My First Test Charge (created for API docs)', });

---

Not only is the API descriptive of what it's doing by design, they had the wherewithal to explain where the missing piece can be found. Contrast that with Amazon where they give you a half-assed example and you have to go on a goose chase to find the params/options for the request. Why?!

The above changes look like a wank or job protection, not any form of well-considered improvement. Couple that with turning the docs into a maze (as opposed to just saying "here's the function, here are the required/optional params, this is what a response looks like) and you realize that the person responsible for this just greenlit it without any question as to "why are we doing this?"

Yes, I'm being an asshole about it. But this is now going to inform API design for other people as it's representative of "what the big guys do." Rinse and repeat that thinking for a decade or two and everything turns into a f*cking Rube Goldberg machine for absolutely zero reason.

Post reply on HN