Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Startup Mistakes: Choice of Datastore

stavros.io

Post reply on HN
Print view View original

91–100 of 119 posts

Re: Startup Mistakes: Choice of Datastore

#91

Post by alkonaut »

maephisto wrote: post #18
elvinyung wrote: post #4

I feel like MongoDB/NoSQL is a horse that's been beaten so much in the last few years that no one is actually making that choice nowadays. Hasn't everyone already learned to stick with Postgres?

MongoDB/NoSQL is deployed and used at large scale by companies like Facebook, Ebay and many others. That's quite far from "no one is actually making that choice nowadays".

> companies like Facebook, Ebay

They didn't chose NoSQL, they were forced to. I'm fairly convinced they started with relational stores. If a company or product grows to a point where relational data doesn't work, that's a problem you want to have.

The mistake is either thinking you need to design for facebook scale from the beginning OR thinknig that you can cut time in a startup by not having to bother with those pesky schemas that just slow you down.

Re: Startup Mistakes: Choice of Datastore

#92

Post by BukhariH »

StavrosK wrote: post #13
ohstopitu wrote: post #11

Earlier quoted context omitted.

I think that there are various reasons to pick DBs. Blindly picking a SQL DB (mysql/postgres etc.) is quite expensive from the get-go (a production ready mysql/postgres would cost ~30$/m). Mongo costs ~$10/m (MongoDB Atlas), Google's Datastore is Pay as you Go (so your initial cost is close to $0 till you get paying customers), AWS's DynamoDB is similarly priced as well. Sure, sadly all those noSQL solutions get real…

Hmm, how are you deploying things that a database costs $30/m? I just put it on the same server as the application worker and split it off if I need more performance later.

I'm personally a fan of: https://www.heroku.com/postgres

Since it's free below 10K rows and only $9 for 10M. And, the dataclips feature always comes in handy.

Re: Startup Mistakes: Choice of Datastore

#93

Post by geophile »

robinwassen wrote: post #85
edneypitta wrote: post #67

Earlier quoted context omitted.

>something that can accommodate changing requirements, i.e., a relational database Wouldn't NoSQL be better suited for this scenario? Genuinely curious.

RMDBs often has great schema migration tools which usually are no problem to use on small datasets. NoSQL often rely on handling difference in the data model in the application layer, which can become messy and cumbersome if you switch a lot of requirements.

Well, they rely on doing pretty much everything in the application layer, which can become messy and cumbersome. Relational databases are the way they are for a reason. NoSQL is often selected, not because they address requirements better, but because developers are not even aware of the problems that relational databases solved years ago.

Re: Startup Mistakes: Choice of Datastore

#94

Post by StavrosK »

wonderous wrote: post #61

@StavrosK: (aka the author) - what is the TLDR of the article? Ask since there appear to be a number of users including myself that appear to not get the intent of your article. — Meta-comment: Feel like if the poster is self-identifying as the author when posting the link, it’s verified via say email/domain, an HN username has been ID’d in the past as the author, etc. — it should be automatically obvious in post, co…

It's what I explicitly call out as a TLDR in the article itself: "Think before you pick a database. If you insist on not thinking, pick PostgreSQL. Trust me."

Basically, "Postgres is a better default".

Re: Startup Mistakes: Choice of Datastore

#95

Post by mamcx »

edneypitta wrote: post #67
geophile wrote: post #46

Earlier quoted context omitted.

Yes and no. Yes, choose carefully at the beginning, and use relational. More specifically, use Postgres. But choose the right things to worry about at the beginning. So worry about something that can accommodate changing requirements, i.e., a relational database. Don't worry about scalability. You will be very, very lucky to ever have that problem. Worry about it then.

>something that can accommodate changing requirements, i.e., a relational database Wouldn't NoSQL be better suited for this scenario? Genuinely curious.

Exist the myth RDBMS are not flexible (because it have schemas). To the contrary, the relational model is very flexible, and allow to model everything you could want. Even model a NoSql :)

And the Relational model is fairly simple. You could learn anything you need in minutes and just remember in add a few index here and there. For the small amount of things you need to do, is incredible how many features a RDBMS give you for free (like transactions, some basic axioms, and a query engine).

----

I don't remember the original quote, but is alike:

"Novices worry for code, Experts focus on data structures".

Properly chose your data-structures, schemas and data-layout will have a huge net impact in your code. That is why a well modeled database will perform well and allow to easily code against it.

This is the hard part, and most novices like to "defer" it.

In my times, we start designing the app first on the DB layer, including the queries, reports, etc. Now most start with the front-end or back-end... Imaging the are focusing in the abstract logic (because some can't imagine the datastore is part of it!) and ignore or reject the concept of learn what are the datastore capabilities.

That is like ignoring the documentation about arrays, building a layer on top of it, rejecting the idea of use arrays as full, and then wondering why his code performs bad and re-creating, badly, what it already have.

---

For fast prototypes, sqlite (stored in RAM) could be good. In the early stages I erase the DB in each run of the code (after the initial DB design, tweaking it). I continue to do that as far as possible, and only worry about migrations and all that when start shipping to customers.

Also, not be afraid of build several copies of the tables - like experiments- (customerA, customerB, etc), use views and peek on your DB documentation.

Re: Startup Mistakes: Choice of Datastore

#96

Post by mamcx »

davidgh wrote: post #82
mamcx wrote: post #16

Earlier quoted context omitted.

BIG ERROR. Like, MONUMENTAL. That something (if chose very wrong) will totally derail you progress and will cost a lot of fix it later. Is incredible. Nobody remember that the most cost effective way to fix a problem is in the early stages? And, yes, the best overall primary datastore, like 90% of the cases, is a RDBMS. Very few actually need to use something else.

If your goal is to build a sustainable business you must address two questions: Will I provide value people are willing to pay for? Will I be able to find those people profitably? Both answers must be yes. You might be able to answer one of the questions without an MVP but very unlikely to answer both without. So you need to ship something. It would be clearly stupid to not consider your technology choices at all in…

>Spending a massive amount of time on getting it “just right” will most likely teach you that you’ve developed the perfect solution to a problem no one cares about.

And this is the reason a RDBMS is totally better. You NOT need to "spend a massive amount of time". With this:

1- You get a totally proven and mature tech 2- With massive tolling support and documentation 3- With the capabilities to model even most NoSql structures. 4- With a query engine that will outperform most developers. And more flexible that most NoSql give 5- With enough scalability that you will ever need for your MVP and beyond 6- And with the ability to plug specialized storage like column stores, time stores and more without complicating your code duplicating efforts.

---

P.D: I have been part of several projects that drink the NoSql koolaid, and be part of the weeks-long coding efforts that could be summarized as:

"This could have been a one line sql code/or index ..."

My last one, rebuilding core aspects of a RDBMS and still building features... dedicating large part of dev time is really "Spending a massive amount of time on getting it “just right”"

Re: Startup Mistakes: Choice of Datastore

#97

Post by StavrosK »

logicallee wrote: post #90
StavrosK wrote: post #32

Earlier quoted context omitted.

Don't use SQLite for production, for all the love I have for it, the client libraries are usually locking accesses and don't work properly with concurrent reads/writes.

have you ever had an issue with that? There are only 7 billion people, most of whom don't need your database updated faster than their ping time... seems like a non-issue.

It's an issue when two people try to write something at the same time and one of them gets a crash because acquiring the lock failed.

Re: Startup Mistakes: Choice of Datastore

#98

Post by vidarh »

edneypitta wrote: post #67
geophile wrote: post #46

Earlier quoted context omitted.

Yes and no. Yes, choose carefully at the beginning, and use relational. More specifically, use Postgres. But choose the right things to worry about at the beginning. So worry about something that can accommodate changing requirements, i.e., a relational database. Don't worry about scalability. You will be very, very lucky to ever have that problem. Worry about it then.

>something that can accommodate changing requirements, i.e., a relational database Wouldn't NoSQL be better suited for this scenario? Genuinely curious.

Postgres is one of the best NoSQL datastores you'll find, until/unless you get to the point where scaling a single Postgres database becomes a problem.

But the point is 1) you're unlikely to ever get there. 2) if you're lucky enough to get there, if you start with your data in a structured database and gradually e.g. move things to JSON blobs in your Postgres DB, then move the things that really, seriously needs it off into a more easily shardable NoSQL datastore, it's a far easier direction to go in than the opposite.

Recovering structure when you've tossed it out the window is a massive pain.

And even a lot of the time when you end up needings things like e.g. ElasticSearch for search or something like that, it may still be better to keep the canonical data store in Postgres and stream changes to a secondary store to scale reads. Or add caching.

I'm not suggesting there are no cases where NoSQL datastores can't be the right choice from the beginning. But if you don't have a very compelling reason to, it's probably not.

Re: Startup Mistakes: Choice of Datastore

#99

Post by Kpourdeilami »

elvinyung wrote: post #86
Kpourdeilami wrote: post #66

Earlier quoted context omitted.

NoSQL is the most useful when you're dealing with event data which aren't the main part of the application. We use MongoDB for analytics on our API endpoints because storing that kind of data in postgres will add unnecessary bloat to it

Hi Kash. Genuinely curious, any other reason to choose MongoDB for analytics? It seems like a poor choice if you want expressive queries for data science. Why not just directly `COPY` your log files to something like Redshift?

Hey! For convenience, mostly. We used to depend on Mixapanel for analytics and MongoDB was a good drop-in replacement with only 20-30 extra lines of code. I agree that Redshift would be a better choice for complicated queries afterwards but the only queries we run on the event data is looking up 2 events with the same "training_id" and calculate the time difference between their timestamps for billing purposes.

The "important" events for data science are still stored on postgres (i.e. logins for checking if a login was malicious)

Re: Startup Mistakes: Choice of Datastore

#100

Post by vidarh »

mamcx wrote: post #16
onion2k wrote: post #2

Just pick something and build an MVP with it. Then get on with the hard part of finding paying customers. You can fix the bad tech decisions later. Without customers it won't matter which database you used before your startup failed.

BIG ERROR. Like, MONUMENTAL. That something (if chose very wrong) will totally derail you progress and will cost a lot of fix it later. Is incredible. Nobody remember that the most cost effective way to fix a problem is in the early stages? And, yes, the best overall primary datastore, like 90% of the cases, is a RDBMS. Very few actually need to use something else.

Consider that for a startup money now is worth a lot more than money even 6-12 months from now, unless you're failing anyway.

Let's say the valuation increases 10-fold between the MVP and the next round. Any $1 in investment I can defer from the first round to the second round then costs shareholders a tenth as much dilution. So even if I end up spending far more money, I may come out ahead, as long as the choices won't hold us back until then.

This doesn't take into account that I might not even be able to raise enough for a more expensive solution, so I might not have a choice.

So a lot of the time the right choice is what costs the least amount right now even if you know it will cause costly re-engineering down the road if you're successful.

For a mature business the tradeoffs are different. If you know your growth rates will be modest, then getting it right from the start matters a lot more.

Post reply on HN