Stripe: Open Source • Hacker News Board

Re: Stripe: Open Source

#71

Post by ahachete » Fri, Sep 04, 2015, 1:59 PM UTC

sandGorgon wrote: post #70

ahachete wrote: post #55

Earlier quoted context omitted.

Or use ToroDB ( http://www.torodb.com ) that speaks the MongoDB protocol but stores the data relationally, in PostgreSQL. Directly. No ETL, no mosql required :) (ToroDB developer here)

hey - this looks interesting. Quick question: why did you build it in Java rather than as a Postgresql extension - like mongres [1] or mogloid [2] ? [1] https://github.com/umitanuki/mongres [2] https://github.com/JerrySievert/mongolike

There are many reasons. The first is that Java is in our opinion the language with the biggest and greatest ecosystem of open source libraries on top of which we can build our software. And this is saving many man-years for us.

I contrast, PostgreSQL extensions need C, which doesn't have these advantages and becomes less productive.

But even more importantly: we don't want to tie it, as much as possible, to PostgreSQL. Maybe, tomorrow, ToroDB would run on other RDBMs :)

If run locally, there's barely performance hit. And it also enables you to run it remotely.

Re: Stripe: Open Source

#72

Post by ahachete » Fri, Sep 04, 2015, 2:04 PM UTC

Alex3917 wrote: post #56

ahachete wrote: post #55

Earlier quoted context omitted.

Or use ToroDB ( http://www.torodb.com ) that speaks the MongoDB protocol but stores the data relationally, in PostgreSQL. Directly. No ETL, no mosql required :) (ToroDB developer here)

I just checked out the page. This seems like an interesting project, but how would rewriting unstructured documents into relational form be faster to query than just retrieving the whole document as jsonb or whatever?

Pulling a document together back from its parts cannot be faster than jsonb or whatever. But it can be orders of magnitude faster if you are pulling a collection out of a query and you avoid scanning the whole thing. Let me elaborate.

When you search in MongoDB (or PostgreSQL with a single table with a jsonb column) the executor needs to visit every single document, every single key, to try to see whether that documents matches the query criteria. This means that in the absence of indexes (or inability to use them, as it happens in MongoDB, where they are not used under many circumstances) the whole db would be scan to answer every single query.

I contrast, ToroDB effectively is "partitioning" the data "by type" (being the type the shape of the documents). See this explanation for more detail: https://github.com/torodb/torodb/wiki/How-ToroDB-stores-json.... To answer a given query, ToroDB only reads those tables with potential documents involved in the result, which usually is a small % of the while database. And this, of course, leads to significantly less I/O and queries that can run up to orders of magnitude faster.

Re: Stripe: Open Source

#73

Post by aerovistae » Fri, Sep 04, 2015, 3:27 PM UTC

Cogito wrote: post #58

aerovistae wrote: post #45

Earlier quoted context omitted.

didn't get a chance to read what he wrote :(

If you're not aware, you can turn on 'showdead' in your preferences to see hidden comments. My preferred way to browse HN :)

LMFAO that was so worth it, thank you for telling me about this. That's hysterical.

Re: Stripe: Open Source

#74

Post by [deleted] » Fri, Sep 04, 2015, 6:58 PM UTC

aerovistae wrote: post #73

Cogito wrote: post #58

Earlier quoted context omitted.

If you're not aware, you can turn on 'showdead' in your preferences to see hidden comments. My preferred way to browse HN :)

LMFAO that was so worth it, thank you for telling me about this. That's hysterical.

[deleted]