31–40 of 47 posts
Earlier quoted context omitted.
Counter-question: Which startup has a actual data table with over 3 billion rows?
We have just crossed 2 billion items in our datastore. While not 3 billion yet, I expect that to happen later this year. Too bad Redshift can't handle JSON files: Converting everything will be annoying.
The article mentions this briefly, but it should be emphasized: parallel loading from S3 is MUCH faster. This weekend I loaded 2 billion rows from S3 both ways: - From a single gzipped object: 4 hours 42 minutes - From 2000 gzipped slices of 1M rows each: 17 minutes (Loading from gzipped files is considerably faster, in addition to saving S3 charges.) The article notes that choice of distribution key is critical. I'd…
Earlier quoted context omitted.
Which off-the-shell RDBMS can handle queries over 3 billion rows?
In 2007 I worked for a firm with a 4 billion row join table in PostgreSQL. Might've been 7 or 8, I don't recall which. It ran on a quad core server with 16Gb of RAM. Joins going through this table took about 2-3 seconds to complete.
Earlier quoted context omitted.
In 2007 I worked for a firm with a 4 billion row join table in PostgreSQL. Might've been 7 or 8, I don't recall which. It ran on a quad core server with 16Gb of RAM. Joins going through this table took about 2-3 seconds to complete.
But I suspect the join must have been over an indexed column, so it did not touched 4bln rows, otherwise 2-3 seconds would be hard to believe. The group by query in the article must access all 3bln rows, which makes a huge difference.
I remember it well, because I was trying to explain why having tens of gigabytes of indexes wouldn't help them much if they only had 16Gb of RAM.
In terms of group-by performance, it depends a lot on the kind of data and how it's stored. For example, taking a sum on a columnar store is quite amenable to parallel solutions and a lot of databases will do that way.
Earlier quoted context omitted.
I think part of the issue why so many people have gone with Hive is that good, production-ready column stores are expensive. Redshift is posed to change that. If you're shopping in this space, Infobright is also worth checking out. And even for moderate data sizes (10+ GB per table), row store DBs tend to become painful. This is especially true when you need to support ad-hoc reporting queries, since the usual techni…
Dimensional modeling (I'm a fan of Kimball's approach) mitigates these problems quite well while still offering very flexible ad-hoc reporting. Works great on a row-based RDBMS, even better on columnar. http://en.wikipedia.org/wiki/Dimensional_modeling http://www.amazon.com/Data-Warehouse-Toolkit-Complete-Dimens... Redshift is indeed a solid product but all these comparisons against Hive are surprising, as that's not…
At my university, standard normalisation was taught in the "databases" course. OLAP was mentioned as part of the "advanced databases" course.
The database course at that time blew about half its time on building PHP applications to talk to the database. I hate to second guess my professors, but I can't help but feel that a more productive use of the time would have been to teach normalised OLTP in the first half, and dimensionally modelled OLAP in the second half. Better yet, to divide them into two courses and spend some time talking about database history ("here's why network and hierarchical databases sucked") and maybe some introduction to how query planners work.
Earlier quoted context omitted.
No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB. How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an un…
Seriously can you and your ilk just please stop. It's so exhausting to hear how much smarter you are and if we just educated ourselves we would realise the error of our ways. People who choose the technologies aren't stupid or masochistic. They understand their use case and the fact is that there are plenty of situations where SQL is suboptimal.
Sometimes with technologies going through the Gartner Hype Cycle people choose the incorrect one, because of the buzz, the glamour around it.
NoSQL is most definately in vouge, quite rightly, too many people often use heavy RBDMS when they are not required.
But too many people perhaps are too quick to dismiss the regular database without actually understanding it.
Any suggestion to avoid hype of technology, question your use cases fully is in my mind a good suggestion.
Earlier quoted context omitted.
No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB. How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an un…
Which off-the-shell RDBMS can handle queries over 3 billion rows?
SQL Server can
Oracle can
Postgres can
Even MySQL can (!)
The limitations are almost always in the hardware, not the software.
If you're looking at column based systems, you can look at Greenplum (does both row and column-based storage), InfiniDB (MySQL based), and all sorts of expensive but very fast appliance options like Netezza, Teradata, etc.
Earlier quoted context omitted.
Dimensional modeling (I'm a fan of Kimball's approach) mitigates these problems quite well while still offering very flexible ad-hoc reporting. Works great on a row-based RDBMS, even better on columnar. http://en.wikipedia.org/wiki/Dimensional_modeling http://www.amazon.com/Data-Warehouse-Toolkit-Complete-Dimens... Redshift is indeed a solid product but all these comparisons against Hive are surprising, as that's not…
I realised a few years ago that pretty much every database course taught only teaches OLTP. OLAP never really gets a lookin. At my university, standard normalisation was taught in the "databases" course. OLAP was mentioned as part of the "advanced databases" course. The database course at that time blew about half its time on building PHP applications to talk to the database. I hate to second guess my professors, but…
Earlier quoted context omitted.
I realised a few years ago that pretty much every database course taught only teaches OLTP. OLAP never really gets a lookin. At my university, standard normalisation was taught in the "databases" course. OLAP was mentioned as part of the "advanced databases" course. The database course at that time blew about half its time on building PHP applications to talk to the database. I hate to second guess my professors, but…
Do you know of any resource that talks about the same topic as your comment, but in more detail?