191–200 of 226 posts
Earlier quoted context omitted.
Nope; it was used every time we needed to look up whether someone was a subscriber, which was cached for some things but not all of them.
Self-replying to add: This wasn’t a particularly frequent occurrence.
Earlier quoted context omitted.
Self-replying to add: This wasn’t a particularly frequent occurrence.
What sort of events would trigger this lookup if it was infrequent?
Just because the table scan is under some threshold doesn't automatically make it better. If a table scan takes 250ms vs 0.01 for a indexed lookup, you're still gonna have to justify to me why making silicon work that damn hard is worth even the electrical use. Are you inserting and deleting so many rows that maintaining the index is prohibitive? Do you have space concerns, and are not able to keep the index in memor…
For small enough tables doing a full scan will be faster than an index. This is also true for regular non-database applications like checking if an item is present in a small collection: it's faster to do a linear scan over a small vector than it is to do a lookup in a hash table or b-tree. With a linear scan (whether it's for data on disk, or a data structure in memory) you don't have to do hashing or branching (exc…
https://doc.rust-lang.org/std/collections/struct.BTreeMap.ht...
Earlier quoted context omitted.
Indices are far more likely to cause a cliff than to remove one, IME. Full table scans are linear and grow linearly. Indices are crazy fast until one day they're suddenly not.
Can you clarify? I don't think I've ever seen such a cliff where the database is using the same query plan? There's sometimes small incremental slow downs as you pass btree depth thresholds and have to add more, but tbh even that's not usually noticeable. I have seen plenty of instances where the database decides an index is no longer the right access,but gets that wrong, and the result is terrible.. but I see that a…
Even this kind of "narrow" slowdown can be substantial. Well, I don't know specifically that it's depth thresholds, but that kind of behaviour - the table passes a certain size and it's suddenly taking 30% longer to insert/read than it did when it was 2% smaller.
> I have seen plenty of instances where the database decides an index is no longer the right access,but gets that wrong, and the result is terrible.. but I see that as a different issue.
IMO it's the same issue, at least from the perspective of the poor chump on call.
Earlier quoted context omitted.
So no primary key or uniqueness constraints?
Unlikely, given Reddit's past schema design. One table of "things", and then another table of attributes of those things in a entity,key,value format. https://kevin.burke.dev/kevin/reddits-database-has-two-table...
Earlier quoted context omitted.
So no primary key or uniqueness constraints?
Unlikely, given Reddit's past schema design. One table of "things", and then another table of attributes of those things in a entity,key,value format. https://kevin.burke.dev/kevin/reddits-database-has-two-table...
According to that post, in 2010, they had about 10 million users. At a conservative 10 fields per user, you're looking at 100 million records.
I'm a bit skeptical that they table scanned 100 million records anytime they wanted to access a user's piece of data back in 2010.
Earlier quoted context omitted.
Are you just trying to create employment for yourself? This is amazingly bad advice. I literally get hired to fix these setups by people that take this approach. Yes, it's good to periodically review query plans. But designing and planning for data retrieval is part of a design process. And it's easy. Even Django since 2003 allowed you to define indexes inside your models. Got a 100m record table and you're adding a…
It sounds like he's thinking of it from the DBA perspective, where they have to react to sudden changes in behavior from devs with little or no warning - since the devs don't talk to them. DBAs doing proactive index creation when they don't know what the devs are doing is indeed futile. The devs, however, should definitely be doing proactive database design/planning whenever they do anything, since they can (and will…
I can indeed imagine life being a lot different if you're on the receiving end of an unknown barrage of queries.
Earlier quoted context omitted.
Can you clarify? I don't think I've ever seen such a cliff where the database is using the same query plan? There's sometimes small incremental slow downs as you pass btree depth thresholds and have to add more, but tbh even that's not usually noticeable. I have seen plenty of instances where the database decides an index is no longer the right access,but gets that wrong, and the result is terrible.. but I see that a…
> There's sometimes small incremental slow downs as you pass btree depth thresholds and have to add more, but tbh even that's not usually noticeable. Even this kind of "narrow" slowdown can be substantial. Well, I don't know specifically that it's depth thresholds, but that kind of behaviour - the table passes a certain size and it's suddenly taking 30% longer to insert/read than it did when it was 2% smaller. > I ha…
This has usually been a memory problem for us - our hot dataset no longer fits in it.
Earlier quoted context omitted.
W.U.C.T Works until critical threshold. In enterprise software I noticed software tends to work in a WUCT'ed up manner. Things may slow down over time, but no one complains about it because "the software is just slow", then suddenly one day you hit the timeout component of some higher layer and then the software is completely and utterly broke.
Everything has a critical threshold. Even your perfect db schema will still eventually run out of disk. There are no magic solutions that work on all scale levels.