Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody ever got fired for using a struct

feldera.com

Post reply on HN
Print view View original

51–60 of 140 posts

Re: Nobody ever got fired for using a struct

#51

Post by jeswin »

reverius42 wrote: post #48
jeswin wrote: post #40

Earlier quoted context omitted.

Where did I say any of that? I'm saying that if you care about performance, data structures should be designed with approach specific tradeoffs in mind. And like I've said above, in typical business apps, it's ok to start with data structures because (a) performance is usually not a problem, (b) staying close to the domain is cleaner.

You said: "You can't design a data structure without knowing how you will use it." But the whole discussion involves knowing how you will use it; the advocacy is for careful consideration of data structures (based on how you will use them) resulting in less pain when designing/choosing algorithms.

My point is that one doesn't follow the other. To design good data structures, you need to know how it'll get used (the algorithm).

> If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident.

This is what I was responding to.

Re: Nobody ever got fired for using a struct

#52

Post by gz09 »

fiddlerwoaroof wrote: post #49
sublinear wrote: post #32

Earlier quoted context omitted.

Can you clarify which knowledge domains those enterprises fall under with examples of what problems they were trying to solve? If it's not obvious, I agree with the hard disagree. Every time I see a table with that many columns, I have a hard time believing there isn't some normalization possible. Schemas that stubbornly stick to high-level concepts and refuse to dig into the subfeatures of the data are often seen fr…

Normalization is possible but not practical in a lot of cases: nearly every “legacy” database I’ve seen has at least one table that just accumulates columns because that was the quickest way to ship something. Also, normalization solves a problem that’s present in OLTP applications: OLAP/Big Data applications generally have problems that are solved by denormalization.

Yep, this comment sums it up well.

We have many large enterprises from wildly different domains use feldera and from what I can tell there is no correlation between the domain and the amount of columns. As fiddlerwoaroof says, it seems to be more a function of how mature/big the company is and how much time it had to 'accumulate things' in their data model. And there might be very good reasons to design things the way they did, it's very hard to question it without being a domain expert in their field, I wouldn't dare :).

Re: Nobody ever got fired for using a struct

#53

Post by rpcope1 »

sublinear wrote: post #32
gz09 wrote: post #4

Earlier quoted context omitted.

Hi, I'm the author of the article. As to your hard disagree, I guess it depends... While this particular user is on the higher end (in terms of columns), it's not our only user where column counts are huge. We see tables with 100+ columns on a fairly regular basis especially when dealing with larger enterprises.

Can you clarify which knowledge domains those enterprises fall under with examples of what problems they were trying to solve? If it's not obvious, I agree with the hard disagree. Every time I see a table with that many columns, I have a hard time believing there isn't some normalization possible. Schemas that stubbornly stick to high-level concepts and refuse to dig into the subfeatures of the data are often seen fr…

I think you believe the average developer, especially on enterprise software where you see this sort of shit, is far more competent or ambitious than they actually are. Many would be horrified to see the number of monkeys banging out nasty DDL in Hibernate or whatever C# uses that have no idea what "normal forms" or "relational algebra" are and are actively resistant to even attempting to learn.

Re: Nobody ever got fired for using a struct

#54

Post by shakna »

SoftTalker wrote: post #2

> But SQL schemas often look like this. Columns are nullable by default, and wide tables are common. Hard disagree. That database table was a waving red flag. I don't know enough/any rust so don't really understand the rest of the article but I have never in my life worked with a database table that had 700 columns. Or even 100.

Salesforce by default comes with some where your tables have 50 columns before you start tweaking anything.

100s is not unusual. Thousands happens before you realise.

Re: Nobody ever got fired for using a struct

#55

Post by shakna »

bananamogul wrote: post #16
SoftTalker wrote: post #2

> But SQL schemas often look like this. Columns are nullable by default, and wide tables are common. Hard disagree. That database table was a waving red flag. I don't know enough/any rust so don't really understand the rest of the article but I have never in my life worked with a database table that had 700 columns. Or even 100.

That statement jumped out at me as well. I've worked as a DBA on tons of databases backing a wide variety of ERPs, web apps, analytics, data warehouses...700 columns?!? No.

You've never seen an SAP database where the business object had a couple hundred fields? Its pretty much required if you're touching international data.

Re: Nobody ever got fired for using a struct

#56

Post by neilyio »

amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

Delightful metaphor, I'll be looking everywhere for a chance to use that now!

Re: Nobody ever got fired for using a struct

#58

Post by vlovich123 »

amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

Protobufs definitely doesn’t solve the problems described. Capnproto may solve it but I’m not 100% sure. JSON/XML/ASN.1 definitely don’t.

It’s like you listed a bunch of serialization technologies without grokking the problem outlined in the post doesn’t have much to do with rkyv itself.

Re: Nobody ever got fired for using a struct

#60

Post by ssivark »

jeswin wrote: post #34
duc_minh wrote: post #28

> Sometimes the best optimization is not a clever algorithm. Sometimes it is just changing the shape of the data. This is basically Rob Pike's Rule 5: If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident.( https://users.ece.utexas.edu/~adnan/pike.html )

I wouldn't give too much credit to rules like this. Data structures are often created with an approach in mind. You can't design a data structure without knowing how you will use it. If anything it's the other way round, if you're not talking about business domain modeling (where data structures first is a valid approach).

To elaborate on @jeswin's point above (IDK why it got downvoted)... a data structure is basically like a cache for the processing algorithm. The business logic and algorithm needs will dictate what details can be computed on-the-fly -vs- pre-generated and stored (be it RAM or disk). Eg: if you're going to be searching a lot then it makes sense to augment the database with some kind of "index" for fast lookup. Or if you are repeatedly going to be pllotting some derived quantity then maybe it makes sense to derive that once and store with the struct.

It's not enough for a data structure to represent the "fundamental" degrees of freedom needed to model the situation; the algorithmic needs (vis-a-vis the available resources) most definitely matter a lot.

Post reply on HN