Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Applying “make invalid states unrepresentable”

kevinmahoney.co.uk

Post reply on HN
Print view View original

171–180 of 193 posts

Re: Applying “make invalid states unrepresentable”

#171

Post by thewebcount »

goto11 wrote: post #142
cle wrote: post #133

Earlier quoted context omitted.

I've seen this too. And I've also seen the converse--complexity that was added because engineers refused anything but the most myopic designs, using thought-terminating cliches like "YAGNI" or "that's hypothetical". "Never think ahead" is obviously not good advice. There's no silver bullet here--we have to think about how likely future scenarios are, and plan for them based on the business context and needs. Many of…

Under what circumstances did following YAGNI lead to added complexity?

A real world case I recently ran into where thankfully we realized we would need it and added it is versioning. A struct you pass to or receive from an API that has a version in it means the difference between being able to make changes to the internals without changing the externals and not being able to. We didn't need the version field until the second version was released. Had we just said, "Oh, we aren't going to need it," on the first version we would have been boned.

Re: Applying “make invalid states unrepresentable”

#172

Post by trevor-e »

srtjstjsj wrote: post #145
trevor-e wrote: post #117

Earlier quoted context omitted.

Can you elaborate more on the "required" fields point? We've been using a similar feature for several years now in APIs at my work and haven't run into any issues, though we do only use it very sparingly for fields that logically can never be missing. At some point a client has to make the call for what they consider essential, so pushing it in the schema makes this less ambiguous from what I've seen. Maybe it's fine…

https://capnproto.org/faq.html#how-do-i-make-a-field-require... Required now means requires forever because people can't migrate safely. But technically you can change a protocol descriptor from required to optional, which is invalid (usually, in a distributed non-transactional system (the common kin) but nothing stops you from doing it. So why not make required forever? Well, do you really want to commit to anything…

After reading that article my take-away is not that "required" is bad and should never be used ever, but rather it was bad with how Google wanted to use it. And since this is Google's project, it makes sense for them to remove the feature if it's causing data center outages, it's not worth the risk at that point.

For example, in the case of the message bus they say "And even though the message bus doesn’t care about message content", and later on "The right answer is for applications to do validation as-needed in application-level code." Strict schema and validation is most helpful for application developers, not some middleware routing code. Was it not possible for them to write a parser that doesn't fully validate the message for use-cases like this?

Re: Applying “make invalid states unrepresentable”

#173

Post by grogers »

joshuamorton wrote: post #167
yongjik wrote: post #159

Earlier quoted context omitted.

I may be in the minority, but after happily using protobuf for years, I believe that there's nothing inherently wrong with required fields - instead, what's "wrong" is the protocol buffer API. Namely, when constructing a protobuf, theoretically, there might be two different ways: (A) first gather all the fields, and then construct the protobuf from these fields; (B) first construct an empty protobuf, and fill in the…

This isn't actually the issue with required fields (some languages, like java and (usually) python, use a construct-once style). Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database). You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it)…

If you have old clients that expect that field you are removing to be there in a meaningful way, you still have to update all clients before you can stop setting it. Having the protobuf schema itself use optional or required doesn't change that, it just makes the dependency explicit there, instead of only in the code at the endpoints.

Changing required to optional isn't a magic fix for protocol compatibility. If it were (for your limited use case) you can just make that change to the protobuf client side as it doesn't affect the wire representation/interpretation.

Re: Applying “make invalid states unrepresentable”

#174

Post by JoeAltmaier »

Surely the data model is helpful, not only in keeping data integrity intact by disallowing invalid state. But also to help you think about your data, and discover simplifications and subtle rules to improve your model.

I attacked the old 8-queens problem years ago as part of a contest in Byte Magazine. All the solutions published modelled the board as an 8X8 array with a 1 or zero to indicate the presence of a queen. They all ran slow and suffered from invalid game states confusing the algorithms.

My solution was to observe that only 1 queen could be in each column (all solutions require queens to not be able to capture one another, and they can capture vertically). So I represented the board by an array of 8 values, the height of the queen in the column represented by indexing the array.

Further, since only one queen can be in each row, the values were the numbers 1-8.

My solution then, was to seed the array with the value 1, 2, 3, 4, 5, 6, 7, 8. Then test if array[i]-array[j] == (i-j) or (j-i), which would mean a diagonal capture.

Simply permuting the values, searched a subset of board states that had to contain all possible solutions. And the permutation tree could be truncated as soon as for any (i,j) the test failed.

Anyway, the program was tiny and finished in negligible time. A pity I didn't enter the contest!

Re: Applying “make invalid states unrepresentable”

#175

Post by aunty_helen »

choeger wrote: post #9
etripe wrote: post #3

This is a good introduction on a conceptual level. I think a large contributor to the problem is story-oriented development, where all that matters in the sprint is "getting it done" and not looking at the broader context. To make unrepresentable states practical, Scott Wlaschin has an excellent write-up here (0). His book (plugged in that article) is also excellent. [0] https://fsharpforfunandprofit.com/posts/design…

> I think a large contributor to the problem is story-oriented development, where all that matters in the sprint is "getting it done" and not looking at the broader context. I think you have a point here. This design offers much better safety, comparable to "parsing instead of validating". But it requires up-front design. And that is indeed "verboten" in modern software development management style. Why is it "verbot…

I like how you've said it here, but one thing that scrum doesn't have in it is relief from your professional duty as an engineer.

If a system needs to be designed in a particular way, do so. That's how long it takes and that's why in planning you discuss how it will be designed.

The design of how you're going to build the system is taken care of before the task is split into easily digestible bits that meet a definition of ready.

If you need to build it before you know how to build it, scrum allows for research spikes into this. A focused, timeboxed interval that so you have the ability to estimate the actual difficulty of work to be completed.

Re: Applying “make invalid states unrepresentable”

#177

Post by joshuamorton »

grogers wrote: post #173
joshuamorton wrote: post #167

Earlier quoted context omitted.

This isn't actually the issue with required fields (some languages, like java and (usually) python, use a construct-once style). Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database). You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it)…

If you have old clients that expect that field you are removing to be there in a meaningful way, you still have to update all clients before you can stop setting it. Having the protobuf schema itself use optional or required doesn't change that, it just makes the dependency explicit there, instead of only in the code at the endpoints. Changing required to optional isn't a magic fix for protocol compatibility. If it w…

> If you have old clients that expect that field you are removing to be there in a meaningful way

Right, there's the rub. `required` means that anyone who deserializes your proto falls into this category. That's a much larger group than "anyone who reads a specific field". So the list of clients now is forced to include any and all middleware that may read your proto (imagine a routing layer or some kind of analytics system or whatnot).

(Note also that there's lots of ways to make reading a field that is empty fallback to doing some reasonable non-catastrophic behavior, required doesn't let you do those things).

Re: Applying “make invalid states unrepresentable”

#178

Post by dllthomas »

nemetroid wrote: post #78
the-smug-one wrote: post #45

Earlier quoted context omitted.

You could always split your types into frontend and backend types where the backend ones are more open and the frontend ones are more restricted. I don't necessarily mean FE/BE as on the web. A lot of code is only interested in shuffling around data anyway, the shape is fairly uninteresting.

I would put it the other way around: make your basic representation restricted, but present a more permissive API. This way, the data model helps enforce your constraints, but you don't need to redesign the API when the requirements (inevitably) change.

If you make your front-end interface more complex than is needed to represent current data, you may save clients a migration in the future. But you force your clients to deal with the complexity right now, and you could wind up needing to change things in a different way in the future. Neither of these concerns clearly dominates all the time - it's a question of just how much effort you're saving them, just how likely various changes are, etc.

On the other hand, if you make your internal representation support more complexity than what you hand out to your users, there arises the question of how you simplify (probably lossily!) or break when you wind up with data that can't be expressed with the simpler interface. But you might well be saving yourself a costly data migration down the line. Also a decision with trade-offs that could potentially go either way.

Re: Applying “make invalid states unrepresentable”

#179

Post by dllthomas »

rocqua wrote: post #92
teddyh wrote: post #68

This is very much like database normalization, in that it has the benefit of making invalid data impossible, but the drawback of often making queries into the data much more cumbersome and usually also inefficient. As with database normalization, it is a good idea to first do it as much as possible, and then denormalize again until it is fast enough.

I have always has the idea of a database that does the denormalizations you want automatically for you. Essentially, you keep the DB in a normalized state. You define views of the DB that you want. Then the DB keeps those views as tables for you, and the DB does all of the hard work of keeping those view tables consistent with the denormalized data. Essentially the DB does the atomicity, cache-invalidation, and cache…

https://en.wikipedia.org/wiki/Materialized_view

Re: Applying “make invalid states unrepresentable”

#180

Post by marshray »

Customer signed a document? Great, job one is to get all the data from those filled-in blanks into the database accurately.

Whatever the customer signed is the authoritative contract, not some imaginary entry in a schema intentionally constructed to make it impossible to represent scenarios not envisioned by the developer.

Post reply on HN