Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Common data model mistakes made by startups

metabase.com

Post reply on HN
Print view View original

121–130 of 137 posts

Re: Common data model mistakes made by startups

#121

Post by void_mint »

cerved wrote: post #120
void_mint wrote: post #117

Earlier quoted context omitted.

> what's the relation to bugs in logging and analytics? I'm not sure what you mean. Software has bugs, data has bugs, etc. To be able to fix a bug and rerun a solution is important in all areas of software, it has nothing to do with logs or analytics (but data and data model type questions usually are important to those domains). > also, is there a good resource on how to backfill? Not really, because "backfill" mean…

It was the logging part which puzzled me. The reason I asked about resources is because I have data generated by a personal project. The initial data model was sloppy and so now I'm finding myself having to backfill to clean the data and it's rather painful. Though I haven't come across anything that deals with the subject so I'm just winging it on my own

> It was the logging part which puzzled me.

OP used "Log stacks", but "Log Stacks" are just a specific flavor of event-based timeseries/analytics stacks. If you were to build a log-ingest and log-aggregation system, you'd just be building an ETL but with a specific emphasis on logging.

> The reason I asked about resources is because I have data generated by a personal project. The initial data model was sloppy and so now I'm finding myself having to backfill to clean the data and it's rather painful. Though I haven't come across anything that deals with the subject so I'm just winging it on my own

Snowflake's stage system works similarly to what I'm describing. You can use S3 as a stage, and then load data from the stage into a table. If something bad happens to your data _in Snowflake_, you can just reload from the Stage (with an updated INSERT).

For more ad-hoc ELT/ETL systems (ie not just All-In on Snowflake), you'd have to just assess your own tooling and build it yourself. In general, when building ingest systems I try to document whatever I can per-record. Meaning, each record in a store includes what version of what software ingested it, and a reference to the raw-est form of that data possible (ie, a JSON blob of the original event or a S3 URL to that event's backing source). This lets you say "We identified a bug in the ingest layer at version 0.1.1, we need to reingest all that data with 0.2.0", and then easily identify and remove the exact data that encountered that bug (because you recorded 0.1.1 as a part of the record itself), and then build a list of exactly what S3 files need to be reingested by 0.2.0.

If you're comfortable expanding your dataset a bit to include that type of metadata you save yourself a lot of time when bad things happen (which they will). It's always a game of metadata/bloat/compute time vs. savings, though.

edit I'll add, none of this matters if your dataset is small enough to be imported from 0 in almost no time at all. If you could write a small script to just parse every file in S3 and insert it into a database, and the time it would take to finish doesn't upset you, you're totally fine just doing that. What I described above is for when your data becomes so large that reimporting from 0 is basically impossible.

Re: Common data model mistakes made by startups

#122

Post by ridaj »

cerved wrote: post #119
ridaj wrote: post #112

Earlier quoted context omitted.

For example, your app logs clicks on the "submit" button, but there's a bug in your UX and the button is clickable/tappable multiple times while the form is being processed, instead of being disabled while being processed. Some users are tap-happy and will tap many times thus counting for multiple submissions. If that's how you count actions in your dashboards it will overcount. In terms of resources, I'm not aware o…

Okay, I was confused as I thought you were referring to application logging and not logging that occurs in the data layer. With a normalized and well defined schema, such inconsistent data is impossible. I guess your point is then to have a well defined process on how to go about resolving this when things go awry -- an important point that makes sense.

Regardless of where the logging is, the can be bugs, and there will be bugs given sufficient amount of time and complexity. It's all about planning for recovery.

Re: Common data model mistakes made by startups

#123

Post by snowwrestler »

jokull wrote: post #96
snowwrestler wrote: post #67

Earlier quoted context omitted.

How are you going to satisfy data compliance, which may require the deletion of PII upon request or expiration, if your PII data is immutable?

The way to do it is to have foreign keys, but instead of hard delete you scrub data in the columns.

Seems like if you can scrub data, it is mutable? Maybe I misunderstand what “immutable” means.

Re: Common data model mistakes made by startups

#124

Post by snowwrestler »

mvc wrote: post #85
snowwrestler wrote: post #67

Earlier quoted context omitted.

How are you going to satisfy data compliance, which may require the deletion of PII upon request or expiration, if your PII data is immutable?

How are you going to respond to a warranty claim if you've wrongly deleted the order data when a subject requests that you delete all PII you have about them? https://www.adyen.com/blog/gdpr-what-it-means-for-customer-p...

You store the sales contract separately, which is what this whole subthread is about.

https://news.ycombinator.com/item?id=27251168

Re: Common data model mistakes made by startups

#125

Post by ranguna »

atatatat wrote: post #74
snowwrestler wrote: post #67

Earlier quoted context omitted.

How are you going to satisfy data compliance, which may require the deletion of PII upon request or expiration, if your PII data is immutable?

Was just considering this when I came across your comment. I'm hoping someone here can suggest a one-way audit-log audit-trail sort of solution, because I need this for the medical industry.

I would say the structure of the records themselves can stay, but not the data itself.

If you have a user table, maybe you can just randomly hash the sensitive user data but keep the record.

Not 100% sure about this though, since you could probably derive the user with statistics like: if it's known that 1 person gets a specific disease every 10 years and you have an obfuscated record of a person connected with that disease, it's fairly straightforward to derive who that person is just through that connection.

Re: Common data model mistakes made by startups

#126

Post by ranguna »

duncans wrote: post #83
FigmentEngine wrote: post #37

Earlier quoted context omitted.

"NULL out all identifying information" is anonymization, not deleting the information.

I wrote zeroes to my hard drive. Would you consider the data on my hard drive merely anonymized, or is it deleted?

It's deleted because all connections (and meta data) has been erased.

In a database, if you null all fields but keep the entries and their inter table relationship intact, you still have identifiable data up to a certain point.

Imagine you have data of specific people and you only have one person per country, having a user and country relationship, if you null a single data of a specific user, you'll still be able to find the user just by analysing to which country that user was connected + some external information.

This is the classical problem of deriving personal data from statistical reports and why data anonymisatiom is so complex.

Re: Common data model mistakes made by startups

#127

Post by Tabular-Iceberg »

tomnipotent wrote: post #104
Tabular-Iceberg wrote: post #61

Earlier quoted context omitted.

This is the rationalisation I get every time when I tell companies that their data model is a mess. Never mind that neither I nor the parent said anything about doing it up front. Of course they have to iterate, the problem is that there is no deliberate effort anywhere, it’s just piling more crap on top of old crap and deluding themselves that they are some kind of lean, agile visionaries because of it.

No one gets it right, and it's just grandstanding to pretend that "deliberate effort" is the distinguishing difference between good or bad data models. Unless you're dealing with highly specialized technical scenarios, most software is written to solve ambiguous and nebulous business problems that even the business doesn't necessarily understand.

> most software is written to solve ambiguous and nebulous business problems that even the business doesn't necessarily understand

Obviously if you're not deliberate about what problem you're trying to solve in the first place, no amount of deliberate effort will produce a good data model. Designing business processes must be done with the same deliberate effort, and also need to be constantly refactored.

Re: Common data model mistakes made by startups

#128

Post by Tabular-Iceberg »

laichzeit0 wrote: post #110
Ericson2314 wrote: post #71

Earlier quoted context omitted.

I hope this is sarcasm poking fun at why microservices are terrible and small companies should stop LARPing FANG and avoid them?

I’m not saying micro services are terrible at all. But I think the low coupling leads to a lot of “problems” when you try and re-aggregate data for analytics purposes when fundamentally micro services are meant to enable teams to work on them without worrying about how some other micro service is storing its data. It’s a trade off you make.

> fundamentally micro services are meant to enable teams to work on them without worrying about how some other micro service is storing its data

What exactly is it that microservices bring to the table to achieve that, that doesn't already exist out of the box in every single language in common business use?

Re: Common data model mistakes made by startups

#129

Post by laichzeit0 »

Tabular-Iceberg wrote: post #128
laichzeit0 wrote: post #110

Earlier quoted context omitted.

I’m not saying micro services are terrible at all. But I think the low coupling leads to a lot of “problems” when you try and re-aggregate data for analytics purposes when fundamentally micro services are meant to enable teams to work on them without worrying about how some other micro service is storing its data. It’s a trade off you make.

> fundamentally micro services are meant to enable teams to work on them without worrying about how some other micro service is storing its data What exactly is it that microservices bring to the table to achieve that, that doesn't already exist out of the box in every single language in common business use?

It lowers coupling between teams. If my team is responsible for microservice A then I have no dependency on the team working on microservice B. Microservice A writes to its own database. There are no migration dependencies, etc. It's a distributed system. You have to accept that one microservice could be using mongodb, another postgresql, another mssql. You're typically communicating through APIs, one microservice is not reading directly from the database of another microservice. Materializing all this crap when you want to do analytics is where the problem comes in as now you are forced to construct some unified data model.

Re: Common data model mistakes made by startups

#130

Post by tomnipotent »

Tabular-Iceberg wrote: post #127
tomnipotent wrote: post #104

Earlier quoted context omitted.

No one gets it right, and it's just grandstanding to pretend that "deliberate effort" is the distinguishing difference between good or bad data models. Unless you're dealing with highly specialized technical scenarios, most software is written to solve ambiguous and nebulous business problems that even the business doesn't necessarily understand.

> most software is written to solve ambiguous and nebulous business problems that even the business doesn't necessarily understand Obviously if you're not deliberate about what problem you're trying to solve in the first place, no amount of deliberate effort will produce a good data model. Designing business processes must be done with the same deliberate effort, and also need to be constantly refactored.

Have you worked at a start-up? Or in a field new to you, or new altogether? No amount of well-intended deliberate design will make up for domain expertise. Ever. The best you can do is mitigate as much future refactoring as possible, but eve than that's not always the best use of time.

> Designing business processes must be done with the same deliberate effort

That's not always realistic. Businesses processes solve existing problems, sometimes tackle new problems, but just like data models hindsight is 20/20. And just like data models, business processes are constantly evolving and never done.

Post reply on HN