Live data from Hacker News

Common data model mistakes made by startups

metabase.com

81–90 of 137 posts

Re: Common data model mistakes made by startups

#81
There are advantages for soft deletes for CRUD architecture, but are there any for CQRS/ES (Event Sourcing)?

I guess if your read model is based on RDBMS then it makes sense, otherwise it depends on the database system in question (i.e. some NoSQL databases like C*[1] and Riak[2] are implementing deletes by writing special tombstone values, which is kind of soft-delete but on the implementation level - but you can't easily restore the data like in case of RDBMS).

[1] https://thelastpickle.com/blog/2016/07/27/about-deletes-and-...

[2] https://docs.riak.com/riak/kv/latest/using/reference/object-...

Re: Common data model mistakes made by startups

#82

Earlier quoted context omitted.

INAL, but... you might want to revisit that code. article 17, right to erasure is about erasure of personal data, not about making non-indentifiable. of course they dont define erase or delete :-) (edit: typo)

well to me the transaction is the same as deleting a record and populating a NULL record. I don't see why the law should care in any way about a company populating NULL records.

> I don't see why the law should care in any way about a company populating NULL records.

It cares if the existence of this record still leaks private data. This is why talking about generic "records" here is pretty wrong - actual data is not interchangeable "records" where you can just slap on a generic cargo cult policy and think you're done.

Different use-cases require different data handling. Although, I do agree, for most CRUD cases it's enough to NULL out rows.

Re: Common data model mistakes made by startups

#83

Earlier quoted context omitted.

well to me the transaction is the same as deleting a record and populating a NULL record. I don't see why the law should care in any way about a company populating NULL records.

"NULL out all identifying information" is anonymization, not deleting the information.

I wrote zeroes to my hard drive. Would you consider the data on my hard drive merely anonymized, or is it deleted?

Re: Common data model mistakes made by startups

#84
post #7

> On the flip side, soft deletes require every single read query to exclude deleted records. You can use partial indexes to only index non-deleted rows. If you are worried about having to remember to exclude deleted rows from queries: Use a view to abstract away the implementation detail from your analytics queries.

You can also use the index to cluster database blocks of the table based on the index (postgres => cluster command). This means all active records will be written to the same database blocks, and all deleted records will be kept in separate database blocks. This can speed up queries that need to access a lot of active records.

This is a good alternative to moving deleted records from an active table to a deleted table.

Re: Common data model mistakes made by startups

#85
post #60

Earlier quoted context omitted.

> It is a rookie blunder to link them relationally to master data for products and PII &c. Is it always? If that data is immutable, for example?

How are you going to satisfy data compliance, which may require the deletion of PII upon request or expiration, if your PII data is immutable?

How are you going to respond to a warranty claim if you've wrongly deleted the order data when a subject requests that you delete all PII you have about them?

https://www.adyen.com/blog/gdpr-what-it-means-for-customer-p...

Re: Common data model mistakes made by startups

#86
post #3

> 5. The “right database for the job” syndrome I once saw something a little similar to this, except with one flavor of DB rather than several. A company you've likely heard of went hard for a certain Java graph database product, due to a combination of an internal advocate who seemed determined to be The GraphDB Guy and an engineering manager who was weirdly susceptible to marketing material. This because some of th…

Been through this as well. There was one database for relational data, one for logs, one for analytics, one for miscelaneous, one for binaries, one for time-series, one for key-values, one for caches and probably a lot more! Total nightmare.

Nobody fully knew how operations, schemas, indexing or queries in any of them worked. Usually someone had managed to hack something together in a week and then the rest of the team just did minor changes to existing queries. Joining between the databases was also a fun exercise.

I blame it all on docker. It's so easy to just docker-compose run grafana:latest, then dust off your hands and claim you have a database running. Articles from HN on how fancy setup Netflix have also contributes to this, you don't have the same ops capacity to replicate a FAANG stack.

In the end all of it got replaced with only mongodb and firefighting went down to 0. Everybody in the team knew how to do everything, from new queries to migrations and backup-recovery. It's probably worse in every aspect on each task the specialized databases were solving, but it works good enough and often bringing a really good swiss-army-knife is better than having a caravan of specialized machines which each require special expertise.

Re: Common data model mistakes made by startups

#87

Earlier quoted context omitted.

The chance that you don't have constraints set up correctly is indistinguishable from 100%.

I disagree. Any fairly competent DBA will know how to setup the constraints correctly. It's not rocket science. If you can think logically enough to program, you can think logically enough to set up constraints correctly.

I've yet to see anything I'd consider calling a startup having a DBA. I'm positively impressed if they even default to use foreign keys.

Re: Common data model mistakes made by startups

#88
post #10

If your company has a subscription business model, keep a history of user's subscriptions. They change over time and it is likely you will need to measure popularity and profitability of product offerings over time. Please don't force your analytics team to rely on event logs to reconstruct a subscription history.

I first learned what an "audit log" was because I had to use an audit log to figure out the states of record in the database at a time in the past, because some specific pieces of data were being lost in the "soft-update" database setup.

Re: Common data model mistakes made by startups

#89

Earlier quoted context omitted.

> if that table's primary key is a foreign key in another table - imagine deleting a user and then having no idea who made an order Assuming you have constraints set up correctly (on delete no action or on delete restrict) then how could this ever happen? If you don’t have constraints set up correctly…

The chance that you don't have constraints set up correctly is indistinguishable from 100%.

Foreign key constraints are a waste of effort for most at-scale web apps. They guard against a subset of problems that aren’t actually problems (orphaned rows) at a putative cost (for the db)

Re: Common data model mistakes made by startups

#90

> Typically semi-structured data have schemas that are only enforced by convention Technically, in Postgres you can (kind of) enforce arbitrary schemas for semi-structured data using CHECK constraints. Unfortunately this isn't well-documented and NoSQL DBs often don't support similar mechanisms.

I'd love a document database that supports JSON Schema validation on read and write.
Post reply on HN