Well this was an embarrasing read.
Spending $5k to learn how database indexes work
61–70 of 209 posts
Re: Spending $5k to learn how database indexes work
#62The real answer here is cost limiting. I don't want my cloud provider to keep working at the cost of an order of magnitude higher bill than I was expecting because of a bug in my code. I want to be able to set a billing limit and have them degrade or stop there service if I exceed the limit. AFAIK AWS doesn't have that. They do have the ability to send me alerts if my bill is unexpectedly high, but they still keep wo…
That's why we use DigitalOcean a lot in Africa. You know upfront how much you will spend.
Re: Spending $5k to learn how database indexes work
#63Don’t use DB providers that charge for rows/data scanned. Use Amazon RDS or Google Cloud SQL or just install it yourself on a VM. Pay for CPU, memory, and storage instead.
Rows returned model works really well for certain data loads (where all data customers use is customer-keyed).... This model also scales DOWN really well .. while still providing good scalable availability... That said, I DO agree with the sentiment of paying for a set performance level (clu, memory, storage), to provide predictable pricing.. obviously these guys were bit by the scaling capability. I do a lot of pet…
Disclosure: I used to work on Google Cloud.
This is why BigQuery offers both models and lets you control the caps [1].
Buying fixed compute is effectively buying a throughput cap. Hard Quotas provide a similar function, but aren't a useful budgeting tool if you can't set them yourself.
"Serverless" without limits is basically "infinite throughput, infinite budget" (though App Engine had quotas since day 1 and then budgets once charging was added). The default quotas give you some of that budget / throughput capping, but again if you can't lower them they might not help you.
Either way, BQ won't drop ingestion or storage though because almost nobody wants their data deleted. As a provider, implementing strict budgets is impossible without having a fairly complex policy "if over $X/second stop all activity, oh except let me still do admin work, like adding indexes? Over $Y/second delete everything". I think having user adjustable quotas and throughput caps per "dimension" makes more sense but it puts the burden on the user and no provider offers good enough user control over quota.
tl;dr: true budgets are hard to do, but every provider should strive to offer better quota/throughput controls.
[1] https://cloud.google.com/bigquery/pricing
[2] https://cloud.google.com/bigquery/docs/reservations-workload...
Re: Spending $5k to learn how database indexes work
#64Earlier quoted context omitted.
I have no problems with a developer doing a 5k oopsie with things like card processing or an area that has a legitimate potential for direct monetary losses (such as payment processing where a bug could allow customers to order goods without actually paying). I have a problem with whoever looked at and said "nah, let's go with a cloud-based solution that charges per-query and gives us an essentially infinite financia…
It's not so clear cut. What's the cost of losing the entire on-prem database? Do you trust a company who hired a developer who didn't know about indexes to hire a rock solid DBA? And how much does that DBA cost?
HA/clustered/replicated DB setups are not rocket science. Backups are not rocket science. Losing an on-prem database irrevocably never happened for me in 20 years.
Re: Spending $5k to learn how database indexes work
#65Earlier quoted context omitted.
Paid products can still do all the same datamining that advertisers do, and there are markets for buying/selling that info (eg. Palantir). The truth is that all forms of monetization are inherently at-odds with the user's interests. Paying for an app doesn't magically make this friction go away, and it certainly doesn't reduce the incentive for developers to abuse your trust.
> there are markets for buying/selling that info (eg. Palantir) A major reason for these markets is that the information can be used for advertising or marketing targeting. In a world where the majority of products/services are paid and the amount of advertising is significantly reduced there will be much less demand for this information, leading to lower prices and even lower payoff from selling this information, no…
Palantir's biggest customers are government entities and market researchers, not advertisers. In a world where advertising has been significantly reduced, their products would become more valuable, since regular analytics would become inaccessible. Compared to the data these corporate aggregators collect, ad fingerprinting seems trivial.
The biggest crux of this, though, is the fact that both of these monetization schemes are destructive. Paying for software simply doesn't make sense in most cases, as there aren't that many people who are developing novel solutions these days. That's the exact reason why advertising is so popular: the market knows that relying on your conscious contribution is unsustainable, so why should you believe otherwise?
Re: Spending $5k to learn how database indexes work
#66Earlier quoted context omitted.
I have no problems with a developer doing a 5k oopsie with things like card processing or an area that has a legitimate potential for direct monetary losses (such as payment processing where a bug could allow customers to order goods without actually paying). I have a problem with whoever looked at and said "nah, let's go with a cloud-based solution that charges per-query and gives us an essentially infinite financia…
It's not so clear cut. What's the cost of losing the entire on-prem database? Do you trust a company who hired a developer who didn't know about indexes to hire a rock solid DBA? And how much does that DBA cost?
You can test backups, that doesn't require much expertise, only effort.
Re: Spending $5k to learn how database indexes work
#67Don’t use DB providers that charge for rows/data scanned. Use Amazon RDS or Google Cloud SQL or just install it yourself on a VM. Pay for CPU, memory, and storage instead.
Rows returned model works really well for certain data loads (where all data customers use is customer-keyed).... This model also scales DOWN really well .. while still providing good scalable availability... That said, I DO agree with the sentiment of paying for a set performance level (clu, memory, storage), to provide predictable pricing.. obviously these guys were bit by the scaling capability. I do a lot of pet…
[deleted]
Re: Spending $5k to learn how database indexes work
#68Let me understand please. These people are selling a commercial product and their team has no idea whatsoever of what an index is? And this is news?
It sounds a bit more like they were confused by the automatic index creation for foreign keys they expected to be there. So they probably knew they'd need an index, just assumed this was implicit in the foreign key.
Re: Spending $5k to learn how database indexes work
#69Earlier quoted context omitted.
Which really is a head scratcher. You'd figure especially as a startup seeing a 5k oopsie isn't really as acceptable. Mistakes do happen and I don't mean any shade to this particular person (they'll never make this mistake again) but as an industry the aggregate consequence of this is you have a lot of waste and stupid choices that then have to be cleaned up when more knowledgeable (read highly paid) people are intro…
The best technical people aren't always the best to start a business. The goal is to make money not have perfect code.
Cool, and if this was a case of bike shedding over something that hadn't just cost that early stage startup 5 grand I'd agree with you.
However regurgitating a platitude that everyone, including myself, learned when we tried getting our first business off the ground doesn't add much value here.
Had this been a Database with 10million rows it would have cost them 50k, and this is incredibly basic programming knowledge.
Basic proficiency is a far cry from worrying about best technical talent and not a particularly egregious ask.
Re: Spending $5k to learn how database indexes work
#70Earlier quoted context omitted.
Rows returned model works really well for certain data loads (where all data customers use is customer-keyed).... This model also scales DOWN really well .. while still providing good scalable availability... That said, I DO agree with the sentiment of paying for a set performance level (clu, memory, storage), to provide predictable pricing.. obviously these guys were bit by the scaling capability. I do a lot of pet…
Disclosure: I used to work on Google Cloud. This is why BigQuery offers both models and lets you control the caps [1]. Buying fixed compute is effectively buying a throughput cap. Hard Quotas provide a similar function, but aren't a useful budgeting tool if you can't set them yourself. "Serverless" without limits is basically "infinite throughput, infinite budget" (though App Engine had quotas since day 1 and then bu…
[deleted]