Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Startups should use a relational database

raycmorgan.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: Startups should use a relational database

#71

Post by robbles »

AznHisoka wrote: post #70
robbles wrote: post #68

Earlier quoted context omitted.

What's the algorithm you use for consolidating those repeated writes? I've thought about doing something like this in the past, but was put off by the worry that there might be edge cases where data would be lost (e.g. insufficient writes to trigger a write through, many concurrent writes, etc.)

So I actually store the rows as serialized JSON objects in my redis cache, and when a write occurs, I update the JSON representation in the cache (if it's not in the cache, I read it from the DB and store it there). After X hours, I have a daemon that goes through all expired keys, deserializes the JSON, and executes a transaction, updating 500 objects at a time. To prevent possible race conditions, you need a mutex…

Sounds pretty effective to me. Thanks for sharing this.

Possibly stupid question - how do you retrieve all the expired keys? I was under the impression that once keys expired, they were effectively invisible to clients?

Re: Startups should use a relational database

#72

Post by AznHisoka »

robbles wrote: post #71
AznHisoka wrote: post #70

Earlier quoted context omitted.

So I actually store the rows as serialized JSON objects in my redis cache, and when a write occurs, I update the JSON representation in the cache (if it's not in the cache, I read it from the DB and store it there). After X hours, I have a daemon that goes through all expired keys, deserializes the JSON, and executes a transaction, updating 500 objects at a time. To prevent possible race conditions, you need a mutex…

Sounds pretty effective to me. Thanks for sharing this. Possibly stupid question - how do you retrieve all the expired keys? I was under the impression that once keys expired, they were effectively invisible to clients?

So I actually set a soft time to expire as a value for the key. My daemon reads all objects < this time, and deletes those keys itself. The hard time to expire is set to a time much further than this to prevent the scenario of the daemon exiting for some reason, and memory overflowing.

Re: Startups should use a relational database

#73

Post by taspeotis »

microtonal wrote: post #31
taspeotis wrote: post #24

Earlier quoted context omitted.

I had a quick Google and MSSQL [1], DB2 [2] and Oracle [3] all support indexing XML. [1] http://technet.microsoft.com/en-us/library/ms191497.aspx [2] http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.j... [3] http://docs.oracle.com/cd/B28359_01/appdev.111/b28369/xdb_in...

It's not about indexing paths you indicate, XML databases are about indexing the whole structure (remember, XML is not relational, but are graphs). Besides that they provide XPath/XQuery processing and optimization. You can query large XML documents or sets of documents, like you'd query them with e.g. XQilla. Of course, it is possible to implement all of this on top of existing database technology. E.g. Oracle's Ber…

> It's not about indexing paths you indicate, XML databases are about indexing the whole structure

From my first link, MSSQL does both:

    The primary XML index indexes all tags, values, and paths within the XML instances in an XML column
    ...
    To enhance search performance, you can create secondary XML indexes. A primary XML index must first exist before you can create secondary indexes
And it supports some reasonable subset of XQuery [1].

[1] http://technet.microsoft.com/en-us/library/ms187854.aspx

Re: Startups should use a relational database

#74

Post by memracom »

stiff wrote: post #45
lmm wrote: post #40

The unhappy truth is that for many startups, relational integrity and transaction safety are simply not very valuable. Customers of an early-stage startup are by definition willing to take a risk on whatever they're getting from that startup. So simply not thinking about these problems - accepting that occasionally a partial write will happen, or two writes will collide, or a migration will not quite work correctly a…

This is just a very bad excuse for doing ridiculously shitty systems that stay shitty way after the startup phase. If you at all bother to understand your problem domain, and are not just a monkey at the typewriter, writing down a good domain model and adding the constraints is not going to decrease development speed, quite to the contrary, it's going to increase it, web developers often spend whole workdays just tra…

Why do you call it "software engineering"? Do they try to follow SEMAT? SWEBOK? If not then they probably are just software hackers and do not deserve to be called software engineers. A real software engineer is someone who follows rigourous engineering processes, does not reinvent the wheel and who does their work by applying the lessons of those who have gone before. It requires mental discipline to do this, in particular because you have to resist the urge to reinvent the wheel.

Reinventing the wheel is great fun. Without constraints you can appear to be amazingly productive. But it is all a fantasy because without constraints and a rigorous process you are probably piling up technical debt faster than anything else.

Re: Startups should use a relational database

#75

Post by waps »

VLM wrote: post #57
nostromo wrote: post #12

Earlier quoted context omitted.

The disconnect between your comment and the article is the term "startup" now means giant companies like Airbnb and tiny two person companies that haven't yet created an MVP. I think this article is targeted at the latter: pre-MVP and just post-MVP. For those startups, having two databases with one dedicated to a backend analytics system reeks of premature optimization.

"having two databases with one dedicated to a backend analytics system reeks of premature optimization." Its free software, you don't have to pay for two instances of Oracle. One thing that will quickly kill a biz is combining the functions of PROD and DEV/TEST. Making the DEV/TEST box the DEV/TEST/REPORTS box is not a big deal, and you can't run a (real) biz without a DEV/TEST box.

Having run the technical side of ~5 "small" businesses now (no more than $1.5M revenue), I disagree.

Eventually, nothing can match the performance of storing binary blobs on a cluster. But that only becomes worthwhile if you database is significantly larger than a terabyte. And I'm only talking about the operational "core" database, not your "data warehouse" (the log dumping ground, which should be split off when your database gets to be a few dozen gigs).

Meanwhile, mysql has big advantages :

1) can do basic optimization with "ALTER TABLE", even (mostly) live.

2) you can mix PROD and DEV/TEST (though obviously you need to use good judgement). Obviously you should also have a DEV/TEST instance for actual testing. Sometimes you want to run a test quickly against PROD though. Adding a slave, having it sync and then running against the slave is a joy.

3) creating reports is quick, customizable and everything you want.

4) It's "idiot-friendly". Employees can ramp up to the structure in a mysql db in 2 weeks flat. Try that with custom document stores.

5) It's typesafe and relational safe (if correctly designed), with the advantages that brings : significantly less weirdness in the database.

6) Phpmyadmin. Mysql workbench. Django. Php ...

I'm even going to argue that the GP's argument, that running analytics on PROD can get you fired, is not just wrong, it's actually an advantage of using mysql. (And the open source SAP database can run "live" analytics. You just can't believe how great that is for dashboards)

Re: Startups should use a relational database

#76

Post by pkolaczk »

coolsunglasses wrote: post #69
pkolaczk wrote: post #46

Earlier quoted context omitted.

A million registered users is nothing for an MMORPG. You can get it pretty quickly even in a national-level game, without going global. A completely different story is keeping those users active and earning on them. The problem here is - you need to sustain a pretty massive load, yet only a few % of that load brings you revenue. And things like being out of service for even 10 minutes during peak hours (and peak load…

A million registered users is $15,000,000 a month if you're charging $15 a month, which was standard for a long time. Are you fucking serious? A product that fails with a million users has a monetization problem not a technological one.

Not in the freemium model. In the freemium model it is easy to get many registered users because you're giving the basic version of the game for free and charging for extras. And if 10% of users pay for some extras at all, you may consider yourself very lucky. Typical paying users share is 2-5%. Another problem is 1 million of those users don't play all the time (we had only 5% of them logged at a time), however it is possible to get most of them active for a short time by organizing special events / competitions for them. Therefore you need to have capacity for handling that 1 million of users for a very short period of time, therefore you need to scale, but you are not going to make money from them constantly.

Re: Startups should use a relational database

#77

Post by twic »

nl wrote: post #30
HorizonXP wrote: post #19

Serious question: what are NoSQL databases really good for? I'm only really used to relational DBs, and I'm unclear about which problems a NoSQL database is useful for.

which problems a NoSQL database is useful for It depends in which NoSQL database. Depending on your problem, you can probably find a NoSQL DB optimised for it. It will often be unclear if that NoSQL DB is actually better than a relational database until you try it. Examples: High write throughput: Cassandra Simple key-value: Redis Text search: Solr/Elastic Search etc..

Redis does more than simple key-value - rather than just reading and writing values, since values can be complex types like lists and dictionaries, it can insert into them, append to them, etc. It's still key-value, it's just not simple key-value!

That said, i would hesitate to describe Redis as a database at all. A key characteristic of databases is that they store every write in a durable way. Redis can checkpoint its state periodically, but as i understand it, it either can't or typically isn't used to safely keep every write. Redis is something in between a database and memcached. I doubt there's ever a situation where you have to choose between PostgreSQL/Cassandra/CouchDB and Redis; Redis is something you would use in addition to a database.

As for text search - RDBMSs have full text search, and at least in the case of PostregSQL, it seems pretty good - see slide 49 in http://es.slideshare.net/billkarwin/full-text-search-in-post... from 2009. You might not want to be leaning on your database for text search when you're at scale (for operational reasons more than performance ones), but it's a plausible way to start.

Re: Startups should use a relational database

#78

Post by lmm »

stiff wrote: post #45
lmm wrote: post #40

The unhappy truth is that for many startups, relational integrity and transaction safety are simply not very valuable. Customers of an early-stage startup are by definition willing to take a risk on whatever they're getting from that startup. So simply not thinking about these problems - accepting that occasionally a partial write will happen, or two writes will collide, or a migration will not quite work correctly a…

This is just a very bad excuse for doing ridiculously shitty systems that stay shitty way after the startup phase. If you at all bother to understand your problem domain, and are not just a monkey at the typewriter, writing down a good domain model and adding the constraints is not going to decrease development speed, quite to the contrary, it's going to increase it, web developers often spend whole workdays just tra…

If you understand the problem well enough to write a good domain model then you're basically already there - which is to say, you can only know you're modelling the right domain after you've found product/market fit. But to be able to find that out you need some level of product, which often means having working code before you fully understand what your product really is. At that stage, a schemaless database is very helpful; working out your domain model in detail and typing it out twice would be a waste of time when it's likely to change quickly.

Re: Startups should use a relational database

#79

Post by memracom »

memracom wrote: post #74
stiff wrote: post #45

Earlier quoted context omitted.

This is just a very bad excuse for doing ridiculously shitty systems that stay shitty way after the startup phase. If you at all bother to understand your problem domain, and are not just a monkey at the typewriter, writing down a good domain model and adding the constraints is not going to decrease development speed, quite to the contrary, it's going to increase it, web developers often spend whole workdays just tra…

Why do you call it "software engineering"? Do they try to follow SEMAT? SWEBOK? If not then they probably are just software hackers and do not deserve to be called software engineers. A real software engineer is someone who follows rigourous engineering processes, does not reinvent the wheel and who does their work by applying the lessons of those who have gone before. It requires mental discipline to do this, in par…

And what is wrong with pointing out that "software engineering" is not used to talk about engineering?
Post reply on HN