Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Startups should use a relational database

raycmorgan.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: Startups should use a relational database

#61

Post by sixbrx »

microtonal wrote: post #31
taspeotis wrote: post #24

Earlier quoted context omitted.

I had a quick Google and MSSQL [1], DB2 [2] and Oracle [3] all support indexing XML. [1] http://technet.microsoft.com/en-us/library/ms191497.aspx [2] http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.j... [3] http://docs.oracle.com/cd/B28359_01/appdev.111/b28369/xdb_in...

It's not about indexing paths you indicate, XML databases are about indexing the whole structure (remember, XML is not relational, but are graphs). Besides that they provide XPath/XQuery processing and optimization. You can query large XML documents or sets of documents, like you'd query them with e.g. XQilla. Of course, it is possible to implement all of this on top of existing database technology. E.g. Oracle's Ber…

Oracle for one does allow full document xml indexing (via XMLIndex) which is trivial to implement, also a rather full featured XQuery integrated with SQL.

Re: Startups should use a relational database

#62

Post by raycmorgan »

VLM wrote: post #55

There can only be one DB much like the LOTR can only have one ring. Why? Thats the only area the linked article falls down on. Its a pretty good article other than that. So you properly normalized your entire system, customer billing transaction records all the way up to article tags. Then article tags gets too huge. So next version looks at RDBMS and Redis, and the next version after that only looks at Redis. Custom…

Thank you for your comment. I agree that there is nothing wrong with using a plurality of systems when needed. Your example of moving from one to another is great! Start with a simple system, and once you find bottlenecks, optimize with specialized stores.

Re: Startups should use a relational database

#63

Post by VLM »

RyanZAG wrote: post #58
VLM wrote: post #54

Earlier quoted context omitted.

Total disagree. The most important job of a startup at startup time is probing the market. If you have a billion customers each with a million records you are the Google Maps location thingy and are not a startup anymore and a relational solution may, or may not, work. If you have ten users the ideal database is probably ten interns and some whiteboards. I'm not kidding. I observe there is a common claim brought up e…

Fake it till you make it is why you'd use a quick and dirty mongodb with json dumped straight from your clients, though. Maybe we have different ideas on the speed of development on mysql vs mongodb though.

Also probably different outlooks on which letters to emphasize in CRUD, and probably different assumptions about what "all" startups are doing with a database, anyway.

Re: Startups should use a relational database

#64

Post by pkolaczk »

gaius wrote: post #48
pkolaczk wrote: post #36

Earlier quoted context omitted.

And you provided no arguments at all. Sure, there are things like multi-master replication in RDBMS world, but I yet have to see a scalable system that utilizes it and at the same time is fully ACID compliant (which rules out async replication). We tried multimaster HA replication once, but the write throughput was terrible.

There is no possible argument to make. There are a million - probably more - highly available RDBMS systems all over the world handling real money and real goods and services, 24/7. You can argue if you wish that tables are not a good way to store particular data structures, fine. But the claim that RDBMSs are unreliable is just ludicrous and had been for 30 years.

First, there is difference between reliability and availability. IMHO availability is a subset of reliability.

Not saying they are unreliable per se, but making them really highly available is much, much harder than some NoSQL stores designed for HA, and the solutions are much more complex, usually beyond the point of being able to prove their correctness. It will cost you lots of effort, money and hardware. And to make it network-partition-tolerant, you'll have to give up ACID anyway, so one of the main advantage of RDBMSes over NoSQL stores goes away. IMHO not worth the trouble. In fact, most of the RDBMS systems operating in banks and insurance institutions I've seen were not even fully ACID. They were AD + eventually C and very relaxed I. You really don't need full ACID to handle money, it is just a convenient model for programmers.

I can see systems based on RDBMSes, even the most expensive ones, claiming 7 nines availability on paper do actually fail and sometimes in a totally weird ways, that fixing the mess takes too long. I know some of the companies migrated to NoSQL stores exactly because of this reason - an expensive RDBMS cluster failing after hardware accident while another NoSQL cluster still operating fine in the same datacenter, despite networking problems. I've seen that simply way too often happening to big names, including a few commercial banks and telecoms in Poland, to believe in marketing of HA RDBMS store. Sure, all of them recovered (sometimes after minutes and in one case after a week) and none lost any data, therefore I'm not saying RDBMSes are unreliable ;)

This is exactly a similar story as with scalability. Can RDBMSes be scaled? Yes, they can. But it is expensive, hard and requires very careful application design. It does not work automagically by "I'll simply normalize and throw my queries at it".

Re: Startups should use a relational database

#65

Post by pkolaczk »

coolsunglasses wrote: post #43
pkolaczk wrote: post #33

Earlier quoted context omitted.

For the startup I once worked for, it mattered much more than we had thought at the beginning. The investors were smart enough to notice we had some considerable periods of downtime. Additionally, once we got first million of users (not really that much and nowhere near the scale of Google or FB) we ran into performance problems which couldn't be easily solved just by indexing, optimizing queries or adding more hardw…

First million users. Come on. 99% of your audience is never going to have that problem. Especially if they spend their early days fucking with a Cassandra cluster instead of talking to customers. And it should be noted, you made it anyway . When you make it by the skin of your teeth, that means you probably timed it right. Preempting a problem far-ahead of time in startups means time and effort was wasted, especially…

Additionally, when you do a startup and it succeeds it is already too late to redesign your app completely to make it scale. Changing the RDBMS in the middle of the game and migrating data is risky and would cost you probably more than using a properly scalable database right from the beginning.

Additionally once you know you need to scale, your competition will see your product. Knowing the idea works, if they start from scratch, but using a better database for the job, they can easily put you out of the business, because instead of competing and adding new feauters, you'll be busy fighting scalability problems.

Re: Startups should use a relational database

#66

Post by robconery »

marcosdumay wrote: post #52
robconery wrote: post #9

One thing that could likely get you fired rather quickly is running analytics on your live transactional system. Yes, your business needs to make decisions based on data, this is not terribly new. To think that you only have one data store is a bit short-sighted. Many businesses (including startups) have moved to using document stores for high read environments and scraping nightly drops to their backend analytics sy…

What I don't get about those discussions is why people assume databases that can ONLY do map/reduce are a good thing? SQL is perfectly able to express map/reduce operations, and they main relational databases handle it almost as well as it's possible.

Completely agree here. Your live transactional system needs to export reportable data - this can come in the form of a DB backup/download (which I've done many times) so you can run queries locally or by pushing CSVs using a cron job.

You can use a SQL query for this or a simple map/reduce - either way my argument is to focus on transactional design that works best for your system - don't conflate reporting needs with your live system (i.e. BA queries).

Re: Startups should use a relational database

#67

Post by robconery »

nostromo wrote: post #12
robconery wrote: post #9

One thing that could likely get you fired rather quickly is running analytics on your live transactional system. Yes, your business needs to make decisions based on data, this is not terribly new. To think that you only have one data store is a bit short-sighted. Many businesses (including startups) have moved to using document stores for high read environments and scraping nightly drops to their backend analytics sy…

The disconnect between your comment and the article is the term "startup" now means giant companies like Airbnb and tiny two person companies that haven't yet created an MVP. I think this article is targeted at the latter: pre-MVP and just post-MVP. For those startups, having two databases with one dedicated to a backend analytics system reeks of premature optimization.

True - but the backend analytics system can be Excel :) which it often is. In fact in my "two person startup" (that was alive happily for 5 years) this is exactly what I did :). I do take your point, however. I still think it's wise to not have an admin backend that runs rollups on your live system (which we've all done).

Re: Startups should use a relational database

#68

Post by robbles »

AznHisoka wrote: post #50
HorizonXP wrote: post #19

Serious question: what are NoSQL databases really good for? I'm only really used to relational DBs, and I'm unclear about which problems a NoSQL database is useful for.

I have been using redis as a write though cache(not just a read cache). So I used to have 1000 concurrent writes to my Postgres db. Obviously this had a negative impact on performance even with modern specs(64gb ram, 8 cores). I realized that a lot of those writes we're updating the same column for the same row but in different threads. So I basically use redis to buffer these queries and perform 1 single write inste…

What's the algorithm you use for consolidating those repeated writes? I've thought about doing something like this in the past, but was put off by the worry that there might be edge cases where data would be lost (e.g. insufficient writes to trigger a write through, many concurrent writes, etc.)

Re: Startups should use a relational database

#69

Post by coolsunglasses »

pkolaczk wrote: post #46
coolsunglasses wrote: post #43

Earlier quoted context omitted.

First million users. Come on. 99% of your audience is never going to have that problem. Especially if they spend their early days fucking with a Cassandra cluster instead of talking to customers. And it should be noted, you made it anyway . When you make it by the skin of your teeth, that means you probably timed it right. Preempting a problem far-ahead of time in startups means time and effort was wasted, especially…

A million registered users is nothing for an MMORPG. You can get it pretty quickly even in a national-level game, without going global. A completely different story is keeping those users active and earning on them. The problem here is - you need to sustain a pretty massive load, yet only a few % of that load brings you revenue. And things like being out of service for even 10 minutes during peak hours (and peak load…

A million registered users is $15,000,000 a month if you're charging $15 a month, which was standard for a long time.

Are you fucking serious? A product that fails with a million users has a monetization problem not a technological one.

Re: Startups should use a relational database

#70

Post by AznHisoka »

robbles wrote: post #68
AznHisoka wrote: post #50

Earlier quoted context omitted.

I have been using redis as a write though cache(not just a read cache). So I used to have 1000 concurrent writes to my Postgres db. Obviously this had a negative impact on performance even with modern specs(64gb ram, 8 cores). I realized that a lot of those writes we're updating the same column for the same row but in different threads. So I basically use redis to buffer these queries and perform 1 single write inste…

What's the algorithm you use for consolidating those repeated writes? I've thought about doing something like this in the past, but was put off by the worry that there might be edge cases where data would be lost (e.g. insufficient writes to trigger a write through, many concurrent writes, etc.)

So I actually store the rows as serialized JSON objects in my redis cache, and when a write occurs, I update the JSON representation in the cache (if it's not in the cache, I read it from the DB and store it there). After X hours, I have a daemon that goes through all expired keys, deserializes the JSON, and executes a transaction, updating 500 objects at a time. To prevent possible race conditions, you need a mutex to allow only 1 thread to modify the cache for that row.

Each object has a field that keeps track of what fields were updated, so I can construct an update query from that.

When I read a row from the DB, I first check if it's in the write through cache, if so I retrieve it there.

Of course, this means that all writes go through my cache. There can't be any other process that updates that table that avoids writing to the cache, or else we have database inconsistency.

This improved the load in my DB server by more than ten-fold, as well as the indexing time to my ElasticSearch server.

Post reply on HN