How to speed up massive data analysis by eliminating disk seeks
petewarden.typepad.com
How to speed up massive data analysis by eliminating disk seeks
1–10 of 28 posts
Re: How to speed up massive data analysis by eliminating disk seeks
#2I'm certain I'm re-inventing the wheel with this approach, but I obviously haven't been researching in the right places, since I hadn't run across this approach before I cobbled it together.
I'm expecting an education on what I'm missing from the HN community!
Re: How to speed up massive data analysis by eliminating disk seeks
#3I'm certain I'm re-inventing the wheel with this approach, but I obviously haven't been researching in the right places, since I hadn't run across this approach before I cobbled it together. I'm expecting an education on what I'm missing from the HN community!
I got the impression from some of what I was reading in the run up to Y2K that this sort of approach was common in the mainframe world.
Obviously, many sites had lots of data (often a lot stored on tape) and limited main memory (especially back in the days in which it was core).
You might start here: http://en.wikipedia.org/wiki/Mainframe_sort_merge ("It is very frequently used; often the most commonly used application program in a mainframe shop.")
Show the old dogs that new dogs can learn old tricks ^_^.
Re: How to speed up massive data analysis by eliminating disk seeks
#4I'm certain I'm re-inventing the wheel with this approach, but I obviously haven't been researching in the right places, since I hadn't run across this approach before I cobbled it together. I'm expecting an education on what I'm missing from the HN community!
is http://en.wikipedia.org/wiki/Merge_sort#Merge_sorting_tape_d... what you were remembering? that's what immediately sprung to my mind on reading the article, but i don't know if you're expecting something more specific.
Re: How to speed up massive data analysis by eliminating disk seeks
#5I'm certain I'm re-inventing the wheel with this approach, but I obviously haven't been researching in the right places, since I hadn't run across this approach before I cobbled it together. I'm expecting an education on what I'm missing from the HN community!
[deleted]
Re: How to speed up massive data analysis by eliminating disk seeks
#6This is basically a map reduce. You should look at hadoop as you start doing more complicated stuff.
Re: How to speed up massive data analysis by eliminating disk seeks
#7This is basically a map reduce. You should look at hadoop as you start doing more complicated stuff.
but for hadoop, you need several boxes and the admin overhead of setting it up and administrating it. if you have a large iron box, this approach works well as I have done this style many times.
Re: How to speed up massive data analysis by eliminating disk seeks
#8[deleted]
Re: How to speed up massive data analysis by eliminating disk seeks
#9This is basically a map reduce. You should look at hadoop as you start doing more complicated stuff.
The sort mode is map-reduce style, but the overall idea was to put data sequentially on disk before you process it.
Re: How to speed up massive data analysis by eliminating disk seeks
#10Random access is always slower than linear reading. Even if you aren't going to disk, you can avoid blowing out the processor cache and having to get from main system memory. Ram is a cache, and L2 is a cache, &c. What you are doing is pretty normal "old-school" unix programming.