Live data from Hacker News

Common data model mistakes made by startups

metabase.com

91–100 of 137 posts

Re: Common data model mistakes made by startups

#91

> Queries for business metrics are usually scattered, written by many people, and generally much less controlled. So do what you can to make it easy for your business to get the metrics it needs to make better decisions. A simple but useful thing is setting the database default time zone match the one where most of your team is (instead of UTC). This reduces the chance your metrics are wrong because you forgot to set…

I cannot overstate how bad this advice is. Everything should be UTC by default. You can explicitly use timestamp with timezones and frankly it's trivial to query something like midnight-to-midnight PST. Your team should learn this as early as possible. Build tooling around this, warn users, hell, educate them, but don't set up foot-guns like non-UTC. If I see a timestamp without a timezone, it must always be UTC. To…

This. I once joined a company with local timezone per deployment and it was a nightmare. Not only in terms of development and debugging, but even for all the support tools required and the numerous bugs we found.

I insisted that all the tools that were going to be installed under my watch would be UTC, and never experienced any time issue on them.

Re: Common data model mistakes made by startups

#92
I have seen so many people argue against soft deletes over the years. But I have also had so many instances where users 'accidentally' deleted a bunch of items and then call support to ask if there are any backups. And then I'll have to reconstruct the data from yesterday's backup plus today's changes. A soft delete will take care of this.

And no amount of "are you really really really sure you want to delete this?" confirmations are going to fix this. You could require the whole Spongebob Squarepants ravioli ravioli give me the formuoli song and dance and people will still delete hundreds or thousands of records by accident.

Re: Common data model mistakes made by startups

#93

Earlier quoted context omitted.

This is the rationalisation I get every time when I tell companies that their data model is a mess. Never mind that neither I nor the parent said anything about doing it up front. Of course they have to iterate, the problem is that there is no deliberate effort anywhere, it’s just piling more crap on top of old crap and deluding themselves that they are some kind of lean, agile visionaries because of it.

I think this is largely a consequence of microservices. What is the "data model" here? You're thinking database, to a microservice that's a repository implementation detail.

I don't think it's a consequence of microservices, I think the problem is as old as the programmable computer. I'm the old paper and even clay tablet processes had to be constantly refactored too.

And a common argument I hear from microservice fanboys is that their old monoliths were suffering from high coupling and low cohesion and that microservices help with that. But I don't see why they couldn't have just refactored their monolith to address the problems they were having.

Re: Common data model mistakes made by startups

#94

I have seen so many people argue against soft deletes over the years. But I have also had so many instances where users 'accidentally' deleted a bunch of items and then call support to ask if there are any backups. And then I'll have to reconstruct the data from yesterday's backup plus today's changes. A soft delete will take care of this. And no amount of "are you really really really sure you want to delete this?"…

On this case, one way is to make a past_ or deleted_$tablename where you insert the deleted row before deleting it from production table.

This way you can watch post mortem, restore etc...

AND it's not soft delete since the data is really gone from the production table, therefore no query tweaking

Only thing: you need to really delete when GDPR related deletion is requested.

Re: Common data model mistakes made by startups

#95
post #76

Earlier quoted context omitted.

One underlying reason for this is that DBMS systems have an unnecessary source of complexity: They have a separate Data Manipulation Language (DML) and a Data Description Language (DDL). They really ought to be unified, but few (any?) mainstream SQL databases are homoiconic in this way. E.g.: It should be possible to take a query definition, request its columns ("schema only" execution), and then insert or merge the…

it sounds like you're making an interesting point but I'm afraid I don't follow, could you elaborate?

The "bad" design that keeps cropping up over and over is the second system effect: Tables, relationships, and columns defined as data in a few simple tables, instead of being defined explicitly in the SQL schema as expected. This is less than optimal for lots of reasons: duplication of metadata, inefficient query plans, no foreign keys, inability to use most kinds of indexes effectively, etc...

However, the need is real: the ability to easily extend or generate table schemas without having to switch languages. You can argue that this is "not that hard", but you'd be absolutely wrong. It's obscenely difficult. I've tried, failed, and have given up. One use-case I had was automatically generating tables for importing data from PowerShell. My goal was to be able to write something like this:

    Get-Process | Export-Sql -ConnectionString '...' -TableName 'Processes'
And have the "Export-Sql" command automatically generate the table schema on the fly based on the input columns it sees. I even wanted to be able to represent object hierarchies as sets of related tables with parent-child foreign key relationships automatically put in. I got a proof-of-concept working, but there were just too many edge-cases. Things like maximum lengths for key or index columns, maximum row size, inability to switch column types on the fly, etc...

So what to do other people do? They also give up and resort to defining a single table that has the columns: "TableName, RowId, ColumnName, ColumnValue" and call it a day. I mean... what other options are there? Bang your head against the wall for months trying to deal with idiotic things like identifier length limits? Correctly escaping arbitrary input strings? Generating hundreds of distinct commands that cannot be parametrised and hope you don't have a SQL injection vulnerability lurking in there somewhere? It's nuts!

Have you seen just how much code it takes to take an arbitrary table, one with dozens of foreign key references, several filtered multi-column indexes, and views that depend on it, and then insert a column in a specific position? You have to drop everything, copy the table, rename, and then recreate everything. Doing this in code would be... I dunno... a few hundred thousand lines? That's absurd. You're not doing it. I'm not doing it. Microsoft did it once for SQL Server Management Studio, and I bet they're not rewriting that code in a hurry!

But go back to the "bad" schema example before: Is it really that bad? What if the database engine had the ability to store common prefixes just once, instead of repeating them for each row? What if the schema looked like this:

    Database, Owner, Table, Column, Value
Starting to look familiar? A bit like [server].[database].[dbo].[Table] perhaps, familiar to every user of Microsoft SQL Server?

There's a natural hierarchy for describing the data, that lends itself well to being represented as a B-Tree, along with the data itself! That's what the "bad" schema is doing: it's representing the data with the most natural representation! It's not wrong at all!

The problem is that database engines have all sorts of features and optimisations that we want that isn't directly compatible with the naive implementation of the bad schema. Constraints, foreign keys, efficient storage, indexing, etc...

However, none of these features are fundamentally incompatible with an API that merely "pretends" that the data is stored in a single flat list that can have its schema updated with a simple insert. The database engine could simply factor out the schema part and the data part into different physical storage layouts, with all the usual efficiencies such as not having to repeat column names for every row.

To summarise: we could have our cake and eat it too. Database engines could be developed that use ordinary select/insert/delete/update statements to modify the schema, and have it perform just as efficiently as a database that uses clumsy statements like "alter table add column". In this world, a database schema upgrade could be as simple as a single literal "MERGE" statement!

Re: Common data model mistakes made by startups

#96
post #60

Earlier quoted context omitted.

> It is a rookie blunder to link them relationally to master data for products and PII &c. Is it always? If that data is immutable, for example?

How are you going to satisfy data compliance, which may require the deletion of PII upon request or expiration, if your PII data is immutable?

The way to do it is to have foreign keys, but instead of hard delete you scrub data in the columns.

Re: Common data model mistakes made by startups

#97
post #76

Earlier quoted context omitted.

it sounds like you're making an interesting point but I'm afraid I don't follow, could you elaborate?

The "bad" design that keeps cropping up over and over is the second system effect: Tables, relationships, and columns defined as data in a few simple tables, instead of being defined explicitly in the SQL schema as expected. This is less than optimal for lots of reasons: duplication of metadata, inefficient query plans, no foreign keys, inability to use most kinds of indexes effectively, etc... However, the need is r…

While schema creation SQL can be a be a bit unwieldy, I'm not sure I appreciate which part is the problem or what you're trying to achieve.

Obviously you can SELECT * INTO FROM .. if you're just temporarily inserting data.

I'm not sure I see the value in automatically importing arbitrary data into a schemad database object. I think it's too complicated to be carried out by the database and should probably be done by some other piece of software and under human supervision/guidance.

Are you suggesting that INSERT/UPDATE statements also have the ability to modify objects? That sounds like it would add complexity without much gain as opposed to just running and alter table query.

I'm not saying that the process of writing schema modifying queries is painless but I'm not sure I'm convinced that we can have the cake and eat it

For your example I personally would've AWKED into a predefined schema or inserted the data as JSON

Re: Common data model mistakes made by startups

#98
post #8

I would personally add: - Having informal metrics and dimension definitions: you throw together something quick and dirty and then realize there's something semantically broken about your data definitions or unevenness. For example your Android app and iOS apps report "countries" differently, or they have meaningfully different notions of "active users" - Not anticipating backfill/restatement needs. Bugs in logging a…

> - Not anticipating backfill needs. Bugs in logging and analytics stacks happen, so it's important to plan for backfills. Without a plan, backfills can be major fire drills or impossible. This matches my experience. Building tools that allow you to rebuild some or all of a dataset with minimal headache make any individual task much easier. Both in terms of safety, and in terms of things like branching/dev environmen…

what's the relation to bugs in logging and analytics? I'm not sure I see it

also, is there a good resource on how to backfill?

Re: Common data model mistakes made by startups

#100
post #94

I have seen so many people argue against soft deletes over the years. But I have also had so many instances where users 'accidentally' deleted a bunch of items and then call support to ask if there are any backups. And then I'll have to reconstruct the data from yesterday's backup plus today's changes. A soft delete will take care of this. And no amount of "are you really really really sure you want to delete this?"…

On this case, one way is to make a past_ or deleted_$tablename where you insert the deleted row before deleting it from production table. This way you can watch post mortem, restore etc... AND it's not soft delete since the data is really gone from the production table, therefore no query tweaking Only thing: you need to really delete when GDPR related deletion is requested.

>On this case, one way is to make a past_ or deleted_$tablename where you insert the deleted row before deleting it from production table.

The problem with this that it gets really cumbersome if you have a complex system of tables that depend on the main table, you'll end up having to make deleted/archived versions of all those tables. In that case it's easier to have a deleted/archived flag in the main table.

Post reply on HN