Searching 20 GB/sec: Systems Engineering Before Algorithms
blog.scalyr.com
Searching 20 GB/sec: Systems Engineering Before Algorithms
1–10 of 87 posts
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#2modern processors are really, really fast at simple, straight-line operations
Exactly.
Especially given the overhead of the network: for most apps, if the user notices anything else, you've screwed up.
KISUFI [kiss-you-fee]: Keep it simple, you fantastic individual!
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#3This comes up in my work modestly frequently, generally with a slightly different scenario for the tradeoff between "complicated and considered" versus "cheap and dirty but gets the job done."
e.g. We could spend 3 weeks using feature vectors and backtesting against prior data to figure out what signals accounts which are likely to churn send (for the purpose of proactively identifying them and attempting to derisk them, like by having the customer success team help them out directly)... or we could write a two-line if statement based on peoples' intuitive understandings of what unsuccessful accounts look like. (No login in 30 days and doesn't have the $STICKY_FEATURE enabled? Churn risk!)
The chief benefit of the second approach is that it actually ships, 100% of the time, whereas priorities change and an emergency develops and suddenly 75% complete analysis gets thrown in a git repo and forgotten about. Actually shipping is a very useful feature to have.
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#4modern processors are really, really fast at simple, straight-line operations Exactly. Especially given the overhead of the network: for most apps, if the user notices anything else, you've screwed up. KISUFI [kiss-you-fee]: Keep it simple, you fantastic individual!
Canadian version of KISS?
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#5modern processors are really, really fast at simple, straight-line operations Exactly. Especially given the overhead of the network: for most apps, if the user notices anything else, you've screwed up. KISUFI [kiss-you-fee]: Keep it simple, you fantastic individual!
Canadian version of KISS?
Or the PG version of what I really mean. ;)
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#6Simple is an algorithm more or less. Simple and in-memory is something I've used in a number of cases. The other benefit of simple is reliability. 20-30 years ago this type of solution wasn't really possible but with today's 64 bit CPU's, tons of ram, (and if you have to ) SSD's a lot of what used to require cleverness can now be done with simple.
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#7I did a similar thing when holding a forum in Redis - generating a keyword index for searching took up 3x the amount of space of the raw text and, whilst faster for single whole word searches, fetching each article and grep'ing it in Ruby was plenty fast enough for my needs. Plus no overheads of index maintenance (new posts, expiring posts) and search limits ("only posts by X", "last 7 days") etc.
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#8Always.
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#9Isn't this answered quite simply by "when they get the job done to your satisfaction?"
Re: Searching 20 GB/sec: Systems Engineering Before Algorithms
#10This comes up in my work modestly frequently, generally with a slightly different scenario for the tradeoff between "complicated and considered" versus "cheap and dirty but gets the job done." e.g. We could spend 3 weeks using feature vectors and backtesting against prior data to figure out what signals accounts which are likely to churn send (for the purpose of proactively identifying them and attempting to derisk t…
I get the feeling this is more "era-defining" than that...
What hit me was the "Processors are so fast now we can Brute force grep over 100GB in a second".
We are entering a world where 20TB on a magnetic disk is viable, but randomly accessing that data could take months to extract. So how we store data on disks will become vitally important to how we use the data - not unlike tape drives of pre-1980s era where rewinding to the front of the tape cost you several minutes of (expensive) waittime.
Imagine a scenario where these guys design how to stream logs to the disk to maximise streaming reads, then optimise reading that out to SSD thence to RAM and L2 and so forth. All designed to drive text past a regex running at bazillions of times a second.
Lets call it the New Brute Force, where its just as much effort to get out the door as elegant algorithms, but it is much much much simpler. And of course, sells more hardware.
Expect to see Intel Inside wrapped around the New Brute Force any time soon :-)
Edit: And they used Java ! I was expecting low-level C optimisations all over the place