Live data from Hacker News

Common data model mistakes made by startups

metabase.com

111–120 of 137 posts

Re: Common data model mistakes made by startups

#111
post #33

Earlier quoted context omitted.

Practically speaking the data model creates very little value. If your startup is trying to make money, features are more important than design for a good stretch. There comes a time to refactor and fix your architecture but it's usually not at the beginning. You can design a data model if you don't know what you're building. And no startup really knows what they're building.

> Practically speaking the data model creates very little value. That can be said about any cost centre, but you don’t have to drag managers kicking and screaming to get them to buy fire insurance. Practically what it does is allow the company to keep up velocity and not be distracted putting out fires everywhere. Of course building features is the team’s entire reason for existing. But there is no advantage to defer…

I worked at a startup that rewrote major parts of their product three times before the first Series A check cleared (while I worked there at least, I think they had another rewrite before my time). The company is alive and well because they didn't waste time and money on pondering architecture to solve a very difficult class of problems when they didn't know which problems were worth money yet.

They did do some smart things working around such known-unknowns, like operate as a consultancy for several years while building out the tech stack that would ultimately become the product catalog. That way they didn't have enormous risk associated with rewrites since all users were internal and zero projects actually needed feature or ABI with the stack.

The problems they had when I left were obvious, but the data model wasn't one of them. I'll stand by what I said above (ignoring the typo) - you can't specify a data model for a problem you don't know. And no startup really knows what problems they are going to solve when they start.

Re: Common data model mistakes made by startups

#112
post #98

Earlier quoted context omitted.

> - Not anticipating backfill needs. Bugs in logging and analytics stacks happen, so it's important to plan for backfills. Without a plan, backfills can be major fire drills or impossible. This matches my experience. Building tools that allow you to rebuild some or all of a dataset with minimal headache make any individual task much easier. Both in terms of safety, and in terms of things like branching/dev environmen…

what's the relation to bugs in logging and analytics? I'm not sure I see it also, is there a good resource on how to backfill?

For example, your app logs clicks on the "submit" button, but there's a bug in your UX and the button is clickable/tappable multiple times while the form is being processed, instead of being disabled while being processed. Some users are tap-happy and will tap many times thus counting for multiple submissions. If that's how you count actions in your dashboards it will overcount.

In terms of resources, I'm not aware of a one-size-fits-all approach... the most basic would be to define upfront what the playbook is for making backfills, and testing it once in a while if you don't get the natural opportunity to do it.

Re: Common data model mistakes made by startups

#113
post #107

Earlier quoted context omitted.

Then it isn’t master data anymore; it’s just one field of a record of a commercial document. This is taking a long way around to the same point. (Presumably no-one is trying to reference-count GC their RDBMS. If so, I wish them all the luck in the world.)

I'm not quite sure what "master data" means in English (a non-native language to me) but Wikipedia tells me that it's "data about the business entities that provide context for business transactions" (and lists examples that sound relevant for this situation to me). Based on that I'm inclined to think that this would qualify.

[deleted]

Re: Common data model mistakes made by startups

#114
post #107

Earlier quoted context omitted.

Then it isn’t master data anymore; it’s just one field of a record of a commercial document. This is taking a long way around to the same point. (Presumably no-one is trying to reference-count GC their RDBMS. If so, I wish them all the luck in the world.)

I'm not quite sure what "master data" means in English (a non-native language to me) but Wikipedia tells me that it's "data about the business entities that provide context for business transactions" (and lists examples that sound relevant for this situation to me). Based on that I'm inclined to think that this would qualify.

[deleted]

Re: Common data model mistakes made by startups

#115
post #73

Earlier quoted context omitted.

I disagree. Any fairly competent DBA will know how to setup the constraints correctly. It's not rocket science. If you can think logically enough to program, you can think logically enough to set up constraints correctly.

Not everybody has a DBA :(

Well, my team has never had an official DBA either. But we do all the tasks of a DBA. I've installed/upgraded, configured, tested backups, and hardened Oracle and Postgres more times than I can remember. We do all our own DML and DDL work. It really isn't hard to do it right.

Re: Common data model mistakes made by startups

#116

I think the biggest mistake some startups make wrt their data model is not really thinking about it at all. The data model winds up being the byproduct of all the features they've implemented and the framework and the libraries they've used, rather than something that was deliberately designed.

This is a direct result of "move fast and break things", your database schema is the first thing broken and it never recovers.

It's baffling to me that for many companies I've worked for, their data model is basically 100% tech debt that can never be fixed because the cost is too high.

Re: Common data model mistakes made by startups

#117
post #98

Earlier quoted context omitted.

> - Not anticipating backfill needs. Bugs in logging and analytics stacks happen, so it's important to plan for backfills. Without a plan, backfills can be major fire drills or impossible. This matches my experience. Building tools that allow you to rebuild some or all of a dataset with minimal headache make any individual task much easier. Both in terms of safety, and in terms of things like branching/dev environmen…

what's the relation to bugs in logging and analytics? I'm not sure I see it also, is there a good resource on how to backfill?

> what's the relation to bugs in logging and analytics?

I'm not sure what you mean. Software has bugs, data has bugs, etc. To be able to fix a bug and rerun a solution is important in all areas of software, it has nothing to do with logs or analytics (but data and data model type questions usually are important to those domains).

> also, is there a good resource on how to backfill?

Not really, because "backfill" means something different to everyone that holds data. Starting with what questions to ask, I would ask "What do we do if a lot of our data shows up incorrect" and "What do we do if lots of our data goes missing", and solving problems in an individual data stack that arise from those questions.

As an example, at a previous job our ETL/ELT system was all started with a file showing up in an S3 bucket. The code that ingested the contents of those files occasionally had bugs that required reingesting of all data that was processed by that version of the code. Having tools to identify (at the data level) what data was affected by this bug, and then being able to delete that data from a datastore and reingest only those S3 files with a newer version of the ingestion code made these types of bugs much easier to manage over time.

Re: Common data model mistakes made by startups

#118
post #53

Some enterprise data model links here: https://dba.stackexchange.com/questions/12991/ready-to-use-d... Instead of soft deletes, move records to a history table I agree w session issue. Had to rebuild sessions before and is a pita compared to just recording them at source

Good list in there. Len silverstons data model resource books are amazing.. especially volume 3. Reading that book and getting to the point where I actually understood the most generalized patterns in it was a total game changer for me

Re: Common data model mistakes made by startups

#119
post #112
post #98

Earlier quoted context omitted.

what's the relation to bugs in logging and analytics? I'm not sure I see it also, is there a good resource on how to backfill?

For example, your app logs clicks on the "submit" button, but there's a bug in your UX and the button is clickable/tappable multiple times while the form is being processed, instead of being disabled while being processed. Some users are tap-happy and will tap many times thus counting for multiple submissions. If that's how you count actions in your dashboards it will overcount. In terms of resources, I'm not aware o…

Okay, I was confused as I thought you were referring to application logging and not logging that occurs in the data layer.

With a normalized and well defined schema, such inconsistent data is impossible. I guess your point is then to have a well defined process on how to go about resolving this when things go awry -- an important point that makes sense.

Re: Common data model mistakes made by startups

#120
post #98

Earlier quoted context omitted.

what's the relation to bugs in logging and analytics? I'm not sure I see it also, is there a good resource on how to backfill?

> what's the relation to bugs in logging and analytics? I'm not sure what you mean. Software has bugs, data has bugs, etc. To be able to fix a bug and rerun a solution is important in all areas of software, it has nothing to do with logs or analytics (but data and data model type questions usually are important to those domains). > also, is there a good resource on how to backfill? Not really, because "backfill" mean…

It was the logging part which puzzled me.

The reason I asked about resources is because I have data generated by a personal project. The initial data model was sloppy and so now I'm finding myself having to backfill to clean the data and it's rather painful. Though I haven't come across anything that deals with the subject so I'm just winging it on my own

Post reply on HN