111–120 of 235 posts
I would urge people doing analytics to take a look at kdb+ from kx. Unless you have ridiculously large amounts of data(> 200 TB), I can bet that you would be better off with kdb. The only downside is that it costs a lot of money which is a pity.
On a related note, I find it amusing that people seem to think that "Big Data == Hadoop". In actuality, there are plenty of other approaches to scaling out clusters to handle large jobs, including MPI and OpenMP, as well as BSP (Bulk Synchronous Parallel)[3] frameworks. [1]: http://en.wikipedia.org/wiki/Message_Passing_Interface [2]: http://en.wikipedia.org/wiki/OpenMP [3]: http://en.wikipedia.org/wiki/Bulk_synchrono…
Any suggestions for libraries or just use basic numpy/scipy and implement the algorithms?
A local python script is great, but what if it takes 2 or 3 hours to run? Now you need to set up a server to run python scripts. What if the data is generated somewhere that would have high locality to a hadoop cluster? Now you need to pull that data down to your laptop to run your job. What if there are a dozen people running similar jobs? Now your python script server is a highly stressed single point of failure. What if the data is growing 100% month-over-month? Your python scripts are going in the trash soon since they were not written in a way that can be easily translated to map-reduce and hadoop-sized workloads are inevitable.
The next step up is a centralized database, but in my experience running your own (large, highly used) database is a whole lot harder than just throwing files on S3 and spinning up hadoop clusters on EC2 if you have people that can write pig jobs.
A solution like elastic map reduce removes a lot of practical problems such as data distribution, resource management, and system operations beyond the fact that it makes it possible to easily run jobs over terabytes of data at a time.
http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-t...
Earlier quoted context omitted.
Yes I've worked for a couple of outfits that did this over the years. It's usually down to who sold them the solution though. It's worst if the salesperson is external. The funniest has to be the engineering company which I used to work with (not for thank fuck) back in the 90's (when 100Gb was big data!). They managed to bag 2x full 42U racks full of HP N-Class HPUX UNIX kit, PDU systems and separate disk arrays for…
When in the 90's? I'd expect it would have taken the full 2x 42U racks to keep the whole dataset in memory at one time. Not to say it's required, but that doesn't sound totally crazy for the time period. Also, having an order of magnitude more hardware than you ought to need is totally understandable when hardware capability is increasing by an order of magnitude faster than hardware replacement cycles. It's when you…
I had a Sun 1000E [1] for 3 years as a workstation from 1996. 8x 50MHz CPU, 2Gb of RAM, 16Gb disk array. It was made in'93 and was approx 12U for the unit and array. Things weren't that big in the 90's. 80's yes...
99% of the cases I have seen where people have been working with tables that are in the 5+ TB range for analysis, there is some obvious way to compress the data that they have overlooked. Most analysts find some way to aggregate a dataset once, then do actual work on that aggregated dataset, rather than the raw data. In geospatial analytics, for example, a trillion records can be aggregated down to census blocks/bloc…
The point of the article that resonates with me is how frequently a technology that is poorly fit with a problem domain is selected because of conventional wisdom rather than data. Related, it is remarkable how we developers routinely cite Knuth's advice about premature optimization to justify our decision when the shoe fits, and then turn around and flatly ignore the advice when it doesn't fit. Selecting Hadoop befo…
It is especially bad when people uses Knuth's advice to forgo the conception part where you think about what technologies you will use and why.
While I couldn't agree more with the general point of the article, I have some small additional comments. Just as a bit of background, I think that Chris would very much agree that I am not the intended recipient of this advice, and so my comments probably aren't keeping in the spirit of the article. I've spent the last 10+ years exclusively in very large HPC environments where the average size of the problem set is…
I work in particle astrophysics data processing and management, mostly with gamma ray events for Fermi gamma ray space-telescope. Our total data size after five years is somewhere around 2 PB, but the data set usually used for analysis runs about 200 TB, not really that large. AFAIK, none of the 100+ collaborators use Hadoop. Mostly they use software I develop to run batch jobs in parallel on one of several clusters.…