31–40 of 40 posts
Ditch the cloud. Go buy a bunch of fast computers. You’d be surprised at how fast modern computers are.
Re-learn the ancient art of running your own systems…. careful, all the naysayers will tell you that no company can run their own systems, to try would be crazy!
Earlier quoted context omitted.
I agree that some GCP services are better than others. I’ve never used Pub/Sub or Cloud Run, but have been quite happy with BigQuery and GKE.
BigQuery has more footguns than GKE in my experience, but that’s perhaps because I have a lot more experience with GKE and know how to avoid those footguns. To me at least it’s understandable enough to say More Nodes is More Money but completely non-straightforward to say that this query I wrote is going to scan the data in a new and expensive way. Am I doing it wrong?
Note that this is not the default! :-)
Earlier quoted context omitted.
At the beginning of my career, I incurred some hundreds of dollars in losses with BigQuery and Google Cloud Function. The problem with these services is that they are easy and intuitive enough for a beginner to use, but a nightmare to maintain.
That's nothing. Wait until you incur $150k of Lambda costs in a day!
The Cloud is where Moore’s Law goes to die. Ditch the cloud. Go buy a bunch of fast computers. You’d be surprised at how fast modern computers are. Re-learn the ancient art of running your own systems…. careful, all the naysayers will tell you that no company can run their own systems, to try would be crazy!
BigQuery has two completely separate pricing models: on-demand, and slot-based. The two models bill on completely separate things:
* On-demand pricing bills per GB of data processed by your query.
* Slot-based (or editions) pricing allocates your project a number of CPU/memory slots, and you pay per CPU-second.
You can find the costs each query would have by looking at the total_bytes_billed and total_slot_ms columns in the INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION table, and multiplying those values by the slot-ms cost (total_slot_ms * 0.01111) and the bytes-billed cost (total_bytes_billed * 0.0000059). Then you can go through your queries and allocate them to either on-demand or slot-based pricing, depending on which is cheaper.
Usually slot-based is cheaper, but queries that do a lot of internal joins can have really huge CPU costs but cost very little in on-demand if they're not reading a lot of bytes.
Somewhat annoyingly, these billing models are configured at a per-project level; you can't switch between the two of them in a single project. Fortunately, you can query tables from other projects easily.
We've recently been struggling with BigQuery and various other GCP services (i.e. CloudRun and Pub sub) as it feels like utilizing these services can feel like a minefield of gotcha's. With their documentation, and limits/quotas being spread all over the place. It's given us more problems than solutions thus far; albeit that could very well be our fault. Has anybody else had this experience? Or are we just doing it w…
But I do agree, there are some gotchas. PubSub examples: Duplicated messages, shitty DLQ implementation (in my opinion), some developers had improper error handling which lead to to infinite resends of messages (because they nacked message on error), etc..
However, I think the scaling and setup weighs up for all of that. You just need to specify a topic and subscription, and then you don't really have to care about resources or scaling at all, and that is SUPER nice. Also, PubSub is stupidly cheap in comparison to any other similar product, at least that I know of.
Earlier quoted context omitted.
That's nothing. Wait until you incur $150k of Lambda costs in a day!
Invocation loop?
It was originally estimated like 10k or something per test which was approved at the time (had like 3 level of management all down my neck for getting it out, hence using lambda originally).
We did deliver, just needed one more sprint to rewrite it as a distributed system on servers. ;) Moved to like 20 machines w/ 128gb of ram that we could spin up as needed (testing millions of events a second, system in NodeJS!)
1. Google Big Query is super powerful, but it's straightforward to shoot yourself in the foot and end up with a huge bill. I would agree that Big Query shines for advanced users. For regular, it's better to use tech that can't be misused by business analytics that just try to get their job done with BI and SQL.
2. I wish a lot of tooling provided better defaults and suggestions to use it efficiently.