Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody ever got fired for using a struct

feldera.com

Post reply on HN
Print view View original

31–40 of 140 posts

Re: Nobody ever got fired for using a struct

#31

Post by gz09 »

woah wrote: post #13
SoftTalker wrote: post #2

> But SQL schemas often look like this. Columns are nullable by default, and wide tables are common. Hard disagree. That database table was a waving red flag. I don't know enough/any rust so don't really understand the rest of the article but I have never in my life worked with a database table that had 700 columns. Or even 100.

No idea what these guys do exactly but their tagline says "Feldera's award-winning incremental compute engine runs SQL pipelines of any complexity" So it sounds like helping customers with databases full of red flags is their bread and butter

> it sounds like helping customers with databases full of red flags is their bread and butter

Yes that captures it well. Feldera is an incremental query engine. Loosely speaking: it computes answers to any of your SQL queries by doing work proportional to the incoming changes for your data (rather than the entire state of your database tables).

If you have queries that take hours to compute in a traditional database like Spark/PostgreSQL/Snowflake (because of their complexity, or data size) and you want to always have the most up-to-date answer for your queries, feldera will give you that answer 'instantly' whenever your data changes (after you've back-filled your existing dataset into it).

There is some more information about how it works under the hood here: https://docs.feldera.com/literature/papers

Re: Nobody ever got fired for using a struct

#32

Post by sublinear »

gz09 wrote: post #4
SoftTalker wrote: post #2

> But SQL schemas often look like this. Columns are nullable by default, and wide tables are common. Hard disagree. That database table was a waving red flag. I don't know enough/any rust so don't really understand the rest of the article but I have never in my life worked with a database table that had 700 columns. Or even 100.

Hi, I'm the author of the article. As to your hard disagree, I guess it depends... While this particular user is on the higher end (in terms of columns), it's not our only user where column counts are huge. We see tables with 100+ columns on a fairly regular basis especially when dealing with larger enterprises.

Can you clarify which knowledge domains those enterprises fall under with examples of what problems they were trying to solve?

If it's not obvious, I agree with the hard disagree. Every time I see a table with that many columns, I have a hard time believing there isn't some normalization possible.

Schemas that stubbornly stick to high-level concepts and refuse to dig into the subfeatures of the data are often seen from inexperienced devs or dysfunctional/disorganized places too inflexible to care much. This isn't really negotiable. There will be issues with such a schema if it's meant to scale up or be migrated or maintained long term.

Re: Nobody ever got fired for using a struct

#34

Post by jeswin »

duc_minh wrote: post #28

> Sometimes the best optimization is not a clever algorithm. Sometimes it is just changing the shape of the data. This is basically Rob Pike's Rule 5: If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident.( https://users.ece.utexas.edu/~adnan/pike.html )

I wouldn't give too much credit to rules like this. Data structures are often created with an approach in mind. You can't design a data structure without knowing how you will use it.

If anything it's the other way round, if you're not talking about business domain modeling (where data structures first is a valid approach).

Re: Nobody ever got fired for using a struct

#35

Post by tdeck »

roblh wrote: post #15
holden_nelson wrote: post #8

Earlier quoted context omitted.

https://jimmyhmiller.com/ugliest-beautiful-codebase

I kinda love this. That sounds like an incredibly entertaining place to work for between 1 and 2 years in your late 20s and not a second longer.

If you enjoyed this, you'd probably enjoy thedailywtf.com, which is full of stories like that.

Re: Nobody ever got fired for using a struct

#36

Post by amluto »

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one.

I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industrial strength on-disk format, start with a tool for defining on-disk formats, and map back to your language. This gives you far better safety, portability across languages, and often performance as well.

Depending on your needs, the right tool might be Parquet or Arrow or protobuf or Cap’n Proto or even JSON or XML or ASN.1. Note that there are zero programming languages in that list. The right choice is probably not C structs or pickles or some other language’s idea of pickles or even a really cool library that makes Rust do this.

(OMG I just discovered rkyv_dyn. boggle. Did someone really attempt to reproduce the security catastrophe that is Java deserialization in Rust? Hint: Java is also memory-safe, and that has not saved users of Java deserialization from all the extremely high severity security holes that have shown up over the years. You can shoot yourself in the foot just fine when you point a cannon at your foot, even if the cannon has no undefined behavior.)

Re: Nobody ever got fired for using a struct

#37

Post by sublinear »

jeswin wrote: post #34
duc_minh wrote: post #28

> Sometimes the best optimization is not a clever algorithm. Sometimes it is just changing the shape of the data. This is basically Rob Pike's Rule 5: If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident.( https://users.ece.utexas.edu/~adnan/pike.html )

I wouldn't give too much credit to rules like this. Data structures are often created with an approach in mind. You can't design a data structure without knowing how you will use it. If anything it's the other way round, if you're not talking about business domain modeling (where data structures first is a valid approach).

If you don't know enough to design a data structure, requirements are missing and someone talking to the client is dropping the ball big time.

Re: Nobody ever got fired for using a struct

#40

Post by jeswin »

sublinear wrote: post #37
jeswin wrote: post #34

Earlier quoted context omitted.

I wouldn't give too much credit to rules like this. Data structures are often created with an approach in mind. You can't design a data structure without knowing how you will use it. If anything it's the other way round, if you're not talking about business domain modeling (where data structures first is a valid approach).

If you don't know enough to design a data structure, requirements are missing and someone talking to the client is dropping the ball big time.

Where did I say any of that?

I'm saying that if you care about performance, data structures should be designed with approach specific tradeoffs in mind. And like I've said above, in typical business apps, it's ok to start with data structures because (a) performance is usually not a problem, (b) staying close to the domain is cleaner.

Post reply on HN