Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody ever got fired for using a struct

feldera.com

Post reply on HN
Print view View original

81–90 of 140 posts

Re: Nobody ever got fired for using a struct

#81

Post by bob1029 »

SoftTalker wrote: post #2

> But SQL schemas often look like this. Columns are nullable by default, and wide tables are common. Hard disagree. That database table was a waving red flag. I don't know enough/any rust so don't really understand the rest of the article but I have never in my life worked with a database table that had 700 columns. Or even 100.

Some businesses are genuinely this complicated. Splitting those facts into additional tables isn't going to help very much unless it actually mirrors the shape of the business. If it doesn't align, you are forcing a lot of downstream joins for no good reason.

Re: Nobody ever got fired for using a struct

#82

Post by xarope »

SoftTalker wrote: post #2

> But SQL schemas often look like this. Columns are nullable by default, and wide tables are common. Hard disagree. That database table was a waving red flag. I don't know enough/any rust so don't really understand the rest of the article but I have never in my life worked with a database table that had 700 columns. Or even 100.

OLTP tables typically are normalized.

But OLAP tables (data lake/warehouse stuff), for speed purposes, are intentionally denormalized and yes, you can have 100+ columns of nullable stuff.

Re: Nobody ever got fired for using a struct

#83

Post by bob1029 »

everyone wrote: post #39

Just cus structs and classes work differently, and classes are much more common. I tend to make everything a class, unless there is a really good reason to make it a struct.

Classes are a safe default even if you expect things to go very, very fast.

The overhead of screwing up NUMA concerns vastly outstrips any kind of class vs struct differences. It's really one of the very last things you should be worrying about.

Allocating an array of a class vs an array of struct might seem like you're getting a wildly different memory arrangement, but from the perspective of space & time this distinction is mostly pointless. Where the information resides at any given moment is the most important thing (L1/L2/L3/DRAM/SSD/GPU/AWS). Its shape is largely irrelevant.

Re: Nobody ever got fired for using a struct

#84

Post by hdjrudni »

amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

Don't forget PHP's serialize/unserialize, it's also sketchy. Looks like they at least put up a big warning in their docs: https://www.php.net/manual/en/function.unserialize.php

Not hating on PHP, to be clear. It has its warts, but it has served me well.

Re: Nobody ever got fired for using a struct

#85

Post by gz09 »

grey-area wrote: post #77
locknitpicker wrote: post #62

Earlier quoted context omitted.

> Hard disagree. That database table was a waving red flag. Exactly this. This article is not about structs or Rust. This article is about poor design of the whole persistence layer. I mean, hundreds of columns? Almost all of them optional? This is the kind of design that gets candidates to junior engineer positions kicked off a hiring round. Nobody gets fired for using a struct? If it's an organization that tolerate…

The database table is someone else’s data. That’s why this company exists and is explained in the article. They don’t have the option to clean up the data.

Yep pretty much, feldera is the engine we don't control what SQL people throw at us.

Re: Nobody ever got fired for using a struct

#87

Post by kleiba »

> This struct we saw earlier had 700+ of optional fields. In Rust you would never design a struct like this. You would pick a different layout long before reaching 700 Options. But SQL schemas often look like this.

Really? I've never had to do any serious db work in my career, but this is a surprise to me.

Re: Nobody ever got fired for using a struct

#88

Post by magicalhippo »

SoftTalker wrote: post #2

> But SQL schemas often look like this. Columns are nullable by default, and wide tables are common. Hard disagree. That database table was a waving red flag. I don't know enough/any rust so don't really understand the rest of the article but I have never in my life worked with a database table that had 700 columns. Or even 100.

> I have never in my life worked with a database table that had 700 columns

Main table at work is about 600, though I suspect only 300-400 are actively used these days. A lot come from name and address fields, we have about 10 sets of those in the main table, and around 14 fields per.

Back when this was created some 20+ years ago it was faster and easier to have it all in one row rather than to do 20+ joins.

We probably would segment it a bit more if we did it from scratch, but only some.

Re: Nobody ever got fired for using a struct

#89

Post by tialaramex »

porise wrote: post #63

Why is rust allowed to reorder fields? If I know that fields are going to be generally accessed together, this prevents me from ordering them so they fit in cache lines.

You can choose in Rust to explain the representation you want for your data type. Unlike C or C++ that's not a non-portable vendor extension it's just part of the language, look at the repr documentation: https://doc.rust-lang.org/nomicon/other-reprs.html

So if you want "what C does" you can just repr(C) and that's what you get. For most people that's not a good trade unless they're doing FFI with a language that shares this representational choice.

Re: Nobody ever got fired for using a struct

#90

Post by scott_w »

jeswin wrote: post #34
duc_minh wrote: post #28

> Sometimes the best optimization is not a clever algorithm. Sometimes it is just changing the shape of the data. This is basically Rob Pike's Rule 5: If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident.( https://users.ece.utexas.edu/~adnan/pike.html )

I wouldn't give too much credit to rules like this. Data structures are often created with an approach in mind. You can't design a data structure without knowing how you will use it. If anything it's the other way round, if you're not talking about business domain modeling (where data structures first is a valid approach).

> If anything it's the other way round, if you're not talking about business domain modeling (where data structures first is a valid approach).

And even there, the data models usually come about to make specific business processes easier (or even possible). An Order Summary is structured a specific way to allow both the Fulfilment and Invoicing processes possible, which feed down into Payment and Collections processes (and related artefacts).

Post reply on HN