Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cases where full scans are better than indexes

jefftk.com

Post reply on HN
Print view View original

221–226 of 226 posts

Re: Cases where full scans are better than indexes

#221

Post by grog454 »

Aeolun wrote: post #185
jefftk wrote: post #144

Earlier quoted context omitted.

(author) If you're already using a relational database you should almost certainly set up indexes on your table ids and foreign keys. But that's pretty different from the examples in the post! I'm not anti-index, I'm anti-"if you ever have a full table scan in production you're doing it wrong".

‘If you ever have a non-deliberate full table scan in production you are doing it wrong’ then?

If you ever have non-deliberate anything in production there's a non zero chance you're doing it wrong.

Re: Cases where full scans are better than indexes

#222

Post by TylerE »

Aeolun wrote: post #186
TylerE wrote: post #130

Earlier quoted context omitted.

Now document the two words, run the test suite to verify no-breakagem commit to source control, and push to production. Suddenly those two words cost a lot more than $0.06, and that's IF everything goes smoothly.

This falls under the “if we need to test that indexes still work on our mysql/postgres we have bigger problems” header.

Depending on your IO pattern, adding an index can make writes quite a bit slower.

Re: Cases where full scans are better than indexes

#223

Post by chlorion »

pkulak wrote: post #212
chlorion wrote: post #193

Earlier quoted context omitted.

The Rust BTreeMap implementation is an example of this as well. It does a linear scan through the nodes instead of a binary search because it's actually faster in this case! https://doc.rust-lang.org/std/collections/struct.BTreeMap.ht...

Did we both just watch the latest Crust of Rust, or did you just know that? If the latter, I’m impressed!

>Did we both just watch the latest Crust of Rust

Yes that's exactly how I knew about this!

Re: Cases where full scans are better than indexes

#224

Post by anarazel »

wiredfool wrote: post #213
anarazel wrote: post #170

Earlier quoted context omitted.

> My recollection in postgres anyway is that low cardinality indexes aren't useful, because it doesn't take into account which side of the 1/99% you're on when determining to use the index. It does take that into account. Demo: =# CREATE TABLE low_cardinality AS SELECT generate_series(-10, 1000000)

Guess they've improved -- That used to be a thing. Looks like it was probably in the PG13 changes to the statistics and btree indexes that did that, though it's hard to tell exactly.

It worked like this for much longer. The oldest version I have around is 9.2 and it behaves the same. You probably are thinking of a somewhat more complex scenario...

Re: Cases where full scans are better than indexes

#225

Post by chlorion »

fulafel wrote: post #203
chlorion wrote: post #193

Earlier quoted context omitted.

The Rust BTreeMap implementation is an example of this as well. It does a linear scan through the nodes instead of a binary search because it's actually faster in this case! https://doc.rust-lang.org/std/collections/struct.BTreeMap.ht...

Interesting. But the doc doesn't go as far as to say it's faster, just fast - any measurements for eg string keys?

I can't seem to find any benchmarks for this but Gankra did a long write up about Rust's btree implementation!

I did not read this entire thing so it's very possible it contains a benchmark somewhere but I couldn't find any when doing a quick skim.

https://faultlore.com/blah/rust-btree-case/

Re: Cases where full scans are better than indexes

#226

Post by jefftk »

jefftk wrote: post #142
doodlesdev wrote: post #74

Earlier quoted context omitted.

Have you tried out using ripgrep [0] or fzf [1]? 200ms is quite a lot for history search, I'd expect search to be 20ms or less. In fact, it should be instantaneous as it should all fit in memory really. I've been using fzf and the searches are all faster than my monitor can display frames (so less than 16ms), although I only have 10k entries on my history as I clear it up once in a while. [0]: https://github.com/Burn…

Trying now, "cat > /dev/null" takes ~150ms, so I suspect it's related to keeping my history in Google Drive (with local mirroring). If it gets annoyingly slows I'll stop doing that and use a different syncing system.

Actually, I was an idiot and misreading these numbers. It's 0.020s and 0.015s, which are 20ms and 15ms, not 200ms and 150ms.
Post reply on HN