Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
1–10 of 18 posts
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#2Hi, Álvaro here, from 8Kdata, the company behind ToroDB.
Please let us know if you have any questions or comments about ToroDB. We will be happy to answer them :)
Enjoy!
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#3Impressive performance, even compared against a Mongo shard with 3 nodes!
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#4One of our engineer's recently floated up the idea of using ToroDB Stampede for replacing our MongoDB -> PostgreSQL ETL, since it's merely a set of Python scripts parsing JSON into tables.
Have you seen use cases of Stampede being implemented on existing databases/schemas, rather than an entirely new DB?
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#5One of our engineer's recently floated up the idea of using ToroDB Stampede for replacing our MongoDB -> PostgreSQL ETL, since it's merely a set of Python scripts parsing JSON into tables. Have you seen use cases of Stampede being implemented on existing databases/schemas, rather than an entirely new DB?
Hi kasano.
One of the most relevant use cases of ToroDB Stampede is precisely what you say, replacing MongoDB to PostgreSQL ETLs, where you have to design the schema, solve data type conflicts, maybe flatten or discard data, etc. And other problems like real-time replication, managing HA, etc. Stampede addresses all these problems.
While most people may want to use an empty, dedicated database for Stampede, it is not required to do so. Stampede will generate the tables under a schema name that matches the collection name. So as long as there are no name conflicts, you can happily have ToroDB-generated tables alongside your own. Needless to say, this gives you the ability to JOIN information from different data sources.
So sure, go and give it a try! :)
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#6It's so funny to hear sometimes that noSQL is better that SQL databases and see how much more powerful tools can be built using old good relational engines.
Also, this story a year ago story https://www.linkedin.com/pulse/mongodb-32-now-powered-postgr... was simply epic.
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#7Hi, Álvaro here, from 8Kdata, the company behind ToroDB. Please let us know if you have any questions or comments about ToroDB. We will be happy to answer them :) Enjoy!
How do you deal with optional fields in documents? do you modify the table schema on the run?
If there's a larg-ish number of optional fields, but each document has only or a few of them, would it create a sparse table with lots of columns? Did you find any problem in these scenarios?
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#8Hi, Álvaro here, from 8Kdata, the company behind ToroDB. Please let us know if you have any questions or comments about ToroDB. We will be happy to answer them :) Enjoy!
How do you deal with optional fields in documents? do you modify the table schema on the run? If there's a larg-ish number of optional fields, but each document has only or a few of them, would it create a sparse table with lots of columns? Did you find any problem in these scenarios?
So basically yes. ToroDB creates columns and tables on the fly. All the added columns are nullable, so the ALTER TABLE ADD COLUMN is an almost free operation in PostgreSQL.
Sure, sparse tables are created. This is not a problem since nulls in PostgreSQL are quite cheap (they require no or a few bytes of storage per record).
Even if there is a high cardinality of optional fields, we have not seen in real cases that the number of columns goes beyond a few hundred. And that's perfectly manageable by PostgreSQL :)
There might be some pathological, degraded use cases. But we have found none of them on real datasets.
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#9It's so funny to hear sometimes that noSQL is better that SQL databases and see how much more powerful tools can be built using old good relational engines. Also, this story a year ago story https://www.linkedin.com/pulse/mongodb-32-now-powered-postgr... was simply epic.
Another ToroDB developer here! What you say is one of the things we always repeat: is it really necesary to reinvent the wheel? relational databases have been here since several decades and they have improved from then!
Re: Show HN: ToroDB Stampede: Automagical MongoDB to PostgreSQL, 100x Faster Queries
#10It's so funny to hear sometimes that noSQL is better that SQL databases and see how much more powerful tools can be built using old good relational engines. Also, this story a year ago story https://www.linkedin.com/pulse/mongodb-32-now-powered-postgr... was simply epic.
Indeed, MongoDB BI connector v1 was based on PostgreSQL. However, they used 9.4's foreign data wrappers and since those could not push down query clauses, the connector itself introduced a significant performance degradation (on top of MongoDB's current performance difference when compared to Stampede/PostgreSQL).