Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Common data model mistakes made by startups

metabase.com

Post reply on HN
Print view View original

61–70 of 137 posts

Re: Common data model mistakes made by startups

#61

Post by Tabular-Iceberg »

tomnipotent wrote: post #54
pyrophane wrote: post #6

I think the biggest mistake some startups make wrt their data model is not really thinking about it at all. The data model winds up being the byproduct of all the features they've implemented and the framework and the libraries they've used, rather than something that was deliberately designed.

I'm not aware of a single project, ever, that has gotten their data model right up front and not had to iterate on it countless times as it grew/evolved. Except maybe NASA. Even the best early data models fail after years of updates and evolution.

This is the rationalisation I get every time when I tell companies that their data model is a mess. Never mind that neither I nor the parent said anything about doing it up front.

Of course they have to iterate, the problem is that there is no deliberate effort anywhere, it’s just piling more crap on top of old crap and deluding themselves that they are some kind of lean, agile visionaries because of it.

Re: Common data model mistakes made by startups

#62

Post by SkyPuncher »

tomnipotent wrote: post #54
pyrophane wrote: post #6

I think the biggest mistake some startups make wrt their data model is not really thinking about it at all. The data model winds up being the byproduct of all the features they've implemented and the framework and the libraries they've used, rather than something that was deliberately designed.

I'm not aware of a single project, ever, that has gotten their data model right up front and not had to iterate on it countless times as it grew/evolved. Except maybe NASA. Even the best early data models fail after years of updates and evolution.

My rule is "make it easy for us to fix our mistakes".

Even when we've spent a bunch of time planning out data, but we still got a lot of things wrong in hindsight. The reality is we didn't know enough about our product direction to make any truly informed decisions.

In general, poor decisions seem to stem from working in ambiguity about product, rather than poor technical decisions.

Re: Common data model mistakes made by startups

#63

Post by SkyPuncher »

duped wrote: post #33
pyrophane wrote: post #6

I think the biggest mistake some startups make wrt their data model is not really thinking about it at all. The data model winds up being the byproduct of all the features they've implemented and the framework and the libraries they've used, rather than something that was deliberately designed.

Practically speaking the data model creates very little value. If your startup is trying to make money, features are more important than design for a good stretch. There comes a time to refactor and fix your architecture but it's usually not at the beginning. You can design a data model if you don't know what you're building. And no startup really knows what they're building.

"The next series is the time to fix your mistakes from this one".

No matter what, startups break as they grow. You will need to fix things. Just make sure they're not sooo bad that you can't do it in a timely/affordable way.

Re: Common data model mistakes made by startups

#64

Post by Tabular-Iceberg »

duped wrote: post #33
pyrophane wrote: post #6

I think the biggest mistake some startups make wrt their data model is not really thinking about it at all. The data model winds up being the byproduct of all the features they've implemented and the framework and the libraries they've used, rather than something that was deliberately designed.

Practically speaking the data model creates very little value. If your startup is trying to make money, features are more important than design for a good stretch. There comes a time to refactor and fix your architecture but it's usually not at the beginning. You can design a data model if you don't know what you're building. And no startup really knows what they're building.

> Practically speaking the data model creates very little value.

That can be said about any cost centre, but you don’t have to drag managers kicking and screaming to get them to buy fire insurance.

Practically what it does is allow the company to keep up velocity and not be distracted putting out fires everywhere.

Of course building features is the team’s entire reason for existing. But there is no advantage to defer refactoring to some later date. The longer you wait the more painful it gets.

Chances are the time never comes, once progress stalls and the company isn’t out of business yet someone will have the brilliant idea to rewrite everything from scratch, which is just lighting money on fire with extra steps.

Re: Common data model mistakes made by startups

#65

Post by jiggawatts »

beachy wrote: post #31
pyrophane wrote: post #6

I think the biggest mistake some startups make wrt their data model is not really thinking about it at all. The data model winds up being the byproduct of all the features they've implemented and the framework and the libraries they've used, rather than something that was deliberately designed.

At the other end of the scale is a data model designed for extreme extensibility. If you ever hear anyone bragging that their data model is entirely metadata driven, and can be used to model anything - without changing the database - that's a huge red flag, as is looking in and seeing tables called "element", "business object" and the like. Unfortunately, for most serious Enterprise systems, a degree of flexibility i…

One underlying reason for this is that DBMS systems have an unnecessary source of complexity: They have a separate Data Manipulation Language (DML) and a Data Description Language (DDL). They really ought to be unified, but few (any?) mainstream SQL databases are homoiconic in this way.

E.g.: It should be possible to take a query definition, request its columns ("schema only" execution), and then insert or merge the columns into a table definition somewhere. Something like:

    SELECT SCHEMA( SELECT * FROM "blah" ) 
    INTO "tablename"
When a black market is formed, it's a sign that there is an unmet demand. When you see the exact same "wrong" design pattern turn up over and over, it's a sign that the underlying system isn't meeting the needs of the developers.

Re: Common data model mistakes made by startups

#66

Post by edgyquant »

Pxtl wrote: post #13

How do you reconcile the first bullet point (polluting data with test data) vs Test In Production being the modern trend? Those sound irreconcilable.

Doesn’t this mean beta testing in prod? Development, at least everywhere I’ve worked, takes place on a separate db. For instance where I work atm we copy prod data to a staging db every couple of months and develop/test new features there before rolling them out. Any data coming from the beta test, in prod, is not really test data it is prod data and I don’t see why you’d want to remove it.

Re: Common data model mistakes made by startups

#67

Post by snowwrestler »

jhgb wrote: post #60
inopinatus wrote: post #51

Earlier quoted context omitted.

Most order forms are snapshots of data at the instant of their lodgement, since they are sales contracts. It is a rookie blunder to link them relationally to master data for products and PII &c. The record of an order is not intrinsically PII and thereby subject to rights of erasure. It may well be equally unlawful in some jurisdictions to irrevocably destroy it entire, it being necessary for accounting or tax audit,…

> It is a rookie blunder to link them relationally to master data for products and PII &c. Is it always? If that data is immutable, for example?

How are you going to satisfy data compliance, which may require the deletion of PII upon request or expiration, if your PII data is immutable?

Re: Common data model mistakes made by startups

#68

Post by laichzeit0 »

Tabular-Iceberg wrote: post #61
tomnipotent wrote: post #54

Earlier quoted context omitted.

I'm not aware of a single project, ever, that has gotten their data model right up front and not had to iterate on it countless times as it grew/evolved. Except maybe NASA. Even the best early data models fail after years of updates and evolution.

This is the rationalisation I get every time when I tell companies that their data model is a mess. Never mind that neither I nor the parent said anything about doing it up front. Of course they have to iterate, the problem is that there is no deliberate effort anywhere, it’s just piling more crap on top of old crap and deluding themselves that they are some kind of lean, agile visionaries because of it.

I think this is largely a consequence of microservices. What is the "data model" here? You're thinking database, to a microservice that's a repository implementation detail.

Re: Common data model mistakes made by startups

#70

Post by Ericson2314 »

Tabular-Iceberg wrote: post #61
tomnipotent wrote: post #54

Earlier quoted context omitted.

I'm not aware of a single project, ever, that has gotten their data model right up front and not had to iterate on it countless times as it grew/evolved. Except maybe NASA. Even the best early data models fail after years of updates and evolution.

This is the rationalisation I get every time when I tell companies that their data model is a mess. Never mind that neither I nor the parent said anything about doing it up front. Of course they have to iterate, the problem is that there is no deliberate effort anywhere, it’s just piling more crap on top of old crap and deluding themselves that they are some kind of lean, agile visionaries because of it.

Yes absolutely. From day 0 everyone should already know how to do data modeling and perform migrations. If you don't and call yourself and engineer, you have no business starting a company.
Post reply on HN