Searching 20 GB/sec: Systems Engineering Before Algorithms

You have to code for Windows just so. Dear POSIX people in general, there is more to cross platform coding than being able to build it with cygwin.

Every grep I've used on Windows handles wildcards, so it must be doing the expansion itself. The right way to do this sort of thing on Windows is to call FindFirstFile/FindNextFile, which is a cross between readdir and stat in that each call returns not only the name of one directory entry that matches the pattern (as per readdir+fnmatch) but also its metadata (as per stat). So, if you were going to call readdir (to get the names) and then stat each name (to get the metadata), you should really be doing all of it in just this one loop so that each bit of data is retrieved just the once.

But this is exactly what POSIX programmers would seemingly never, ever do. Probably because doing it that way would involve calling a Win32 function. Well, more fool them, because getting this data from FindFirstFile/FindNextFile seems pretty quick, and getting it any other way seems comparatively slow.

I cobbled together a native Win32 port of the_silver_searcher a while ago and it was about twice as fast as the stock EXE thanks to retrieving all the file data in one pass. In this case the readdir wrapper just needed to fill in the file type in the dirent struct; luckily it seems that some POSIX-style systems do this, so there was already code to handle this and it just needed the right #define adding. (I have absolutely no idea why MingW32 doesn't do this already.)

Prior to switching to the_silver_searcher, I used to use grep; the grep I usually used is the GNU-Win32 one (http://gnuwin32.sourceforge.net/packages/grep.htm), and it looks to call readdir to get the names, and then stat to get each name's details. I checked that just now, and after all those years I can finally imagine why it's so slow.

GNU-Win32 find is really slow, too.

http://www.altdevblogaday.com/2013/02/22/latency-mitigation-...

HN discussion: https://news.ycombinator.com/item?id=5265513

I would just like to throw in here if you are going to be featurizing users to do churn prediction, I would highly recommend doing a whole data pipeline and do profit curves.

A data driven approach is only as good as the algorithms used.

If you used profit curves to augment churn prediction (churn prediction is a pre req), you can figure out how much a user is actually worth bringing back. This also allows you to figure out an optimal budget to profit ratio on a cost of a user.

That being said, use it only if you have enough data to be accurate. And also only use it for the right reasons. Done routinely, I believe it can be a great investment.

And on using Java. It's actually pretty fast, and certainly fast enough for a prototype. Once you have the stack build, you can benchmark and optimize the bits that needs to be optimized, usually this means C or handcrafted ASM.

Where they do need pass messages between threads they use the famous LMAX Disruptor which keeps data copying and blocking to a minimum.