Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Applying “make invalid states unrepresentable”

kevinmahoney.co.uk

Post reply on HN
Print view View original

181–190 of 193 posts

Re: Applying “make invalid states unrepresentable”

#181

Post by lalos »

joshuamorton wrote: post #167
yongjik wrote: post #159

Earlier quoted context omitted.

I may be in the minority, but after happily using protobuf for years, I believe that there's nothing inherently wrong with required fields - instead, what's "wrong" is the protocol buffer API. Namely, when constructing a protobuf, theoretically, there might be two different ways: (A) first gather all the fields, and then construct the protobuf from these fields; (B) first construct an empty protobuf, and fill in the…

This isn't actually the issue with required fields (some languages, like java and (usually) python, use a construct-once style). Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database). You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it)…

What about changing it to optional on the server side, and clients that don't upgrade will always send that required payload and you will know to parse it but then ignore it. Updated clients know its optional and also ignore it and don't set it. Once you have low frequency of messages containing that required field you can do the final cleanup.

Re: Applying “make invalid states unrepresentable”

#182

Post by dodobirdlord »

joshuamorton wrote: post #168
dodobirdlord wrote: post #157

Earlier quoted context omitted.

Protocol buffers already require you to commit to some things forever, like the type of a field, or whether two fields belong in a oneof together. I’m not saying that “required” was a great feature, but it’s not exactly unique.

No they don't. An optional field can be deprecated and replaced with a different field. This can be done to change the type (also some types can be changed, although you probably shouldn't). Required usually cannot be deprecated.

You can deprecate an optional field and reserve the field number, but if you reintroduce use of that field number with a field of a different type that is not backward compatible. Types can be changed if they are binary compatible, but in that case they haven’t actually changed, because the binary format is the canonical format.

Re: Applying “make invalid states unrepresentable”

#183

Post by joshuamorton »

dodobirdlord wrote: post #182
joshuamorton wrote: post #168

Earlier quoted context omitted.

No they don't. An optional field can be deprecated and replaced with a different field. This can be done to change the type (also some types can be changed, although you probably shouldn't). Required usually cannot be deprecated.

You can deprecate an optional field and reserve the field number, but if you reintroduce use of that field number with a field of a different type that is not backward compatible. Types can be changed if they are binary compatible, but in that case they haven’t actually changed, because the binary format is the canonical format.

Right, I'm not sure what your point is. You can always add more fields, so being unable to change the type of a field isn't a problem, since you can introduce a new field and start using it. You cannot however, stop using a required field. The best you can do is set it to a nonsense value and leave a comment saying "well we need to set this to something, but we don't actually use it anywhere.

Because to be sure of that you can remove it, you need to be sure that every storage system and every piece of middleware and every since thing that links your proto anywhere in the world that you might care about is upgraded, otherwise if they encounter a new message they'll crash.

If you only have a single client and server, and you control both, this is doable. If you don't have that though, you cannot.

Re: Applying “make invalid states unrepresentable”

#184

Post by andrewflnr »

lalos wrote: post #181
joshuamorton wrote: post #167

Earlier quoted context omitted.

This isn't actually the issue with required fields (some languages, like java and (usually) python, use a construct-once style). Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database). You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it)…

What about changing it to optional on the server side, and clients that don't upgrade will always send that required payload and you will know to parse it but then ignore it. Updated clients know its optional and also ignore it and don't set it. Once you have low frequency of messages containing that required field you can do the final cleanup.

I think the problem is when outdated clients try to parse responses with a missing, used-to-be-required field.

Re: Applying “make invalid states unrepresentable”

#185

Post by andrewflnr »

aunty_helen wrote: post #175
choeger wrote: post #9

Earlier quoted context omitted.

> I think a large contributor to the problem is story-oriented development, where all that matters in the sprint is "getting it done" and not looking at the broader context. I think you have a point here. This design offers much better safety, comparable to "parsing instead of validating". But it requires up-front design. And that is indeed "verboten" in modern software development management style. Why is it "verbot…

I like how you've said it here, but one thing that scrum doesn't have in it is relief from your professional duty as an engineer. If a system needs to be designed in a particular way, do so. That's how long it takes and that's why in planning you discuss how it will be designed. The design of how you're going to build the system is taken care of before the task is split into easily digestible bits that meet a definit…

I get the impression that anything management touches gets turned (one might say corrupted) into a tool for detaching from engineering details. That's not what sole was meant to be about, and it's not what waterfall was about, but it happens anyway.

Details will always matter, but management's paycheck depends on not understanding that.

Re: Applying “make invalid states unrepresentable”

#186

Post by acidbaseextract »

maest wrote: post #86
etripe wrote: post #3

This is a good introduction on a conceptual level. I think a large contributor to the problem is story-oriented development, where all that matters in the sprint is "getting it done" and not looking at the broader context. To make unrepresentable states practical, Scott Wlaschin has an excellent write-up here (0). His book (plugged in that article) is also excellent. [0] https://fsharpforfunandprofit.com/posts/design…

Sum types are one of the main things I miss when working in Python. Is anyone aware of any good ways of adding sum types to Python?

There's a library: https://sumtypes.readthedocs.io/en/latest/

It's serviceable, and even does static checking of total coverage of cases. I will say that it makes linters freak out.

Re: Applying “make invalid states unrepresentable”

#187

Post by goto11 »

jax_the_dog wrote: post #147
goto11 wrote: post #142

Earlier quoted context omitted.

Under what circumstances did following YAGNI lead to added complexity?

Not OP, but it would add complexity because that method you "weren't going to need" turns out to actually be needed. Now you have to work around your simplified design because you decided that you didn't need anything more.

That is not adding complexity, since the end result (the added method) is the same. You are just postponing some work until it is needed, which is prudent anyway due to opportunity cost.

Re: Applying “make invalid states unrepresentable”

#188

Post by goto11 »

tsimionescu wrote: post #154
goto11 wrote: post #142

Earlier quoted context omitted.

Under what circumstances did following YAGNI lead to added complexity?

A common pattern is writing code as a series of isolated cases, when taking some time to design the general case would greatly reduce the amount of code. You add a bool parameter to a function to modify one small bit of what it does, then another one, and you add some new return value, and before long, you've got a class with several getters and instance variables represented as code in a single function, with parame…

What you describe a development style where the first iteration is well-designed but subsequent modifications are applied as a series of hacks and kludges. This way any code will turn into a big ball of mud over time. This is a problem.

But trying to anticipate everything in the first iteration is not the solution. The solution is to write maintainable code and apply each change with the same discipline and thought which was used when in the initial iteration. Follow the boy scout rule: After any change, the code should be in a better state than before.

Re: Applying “make invalid states unrepresentable”

#189

Post by jbergens »

steve_g wrote: post #46
judofyr wrote: post #26

In general I agree that it's nice to make invalid states unrepresentable, but I'm not sure if I agree that this counts as a fundamental "invalid state". There is nothing about contracts which require that you can only have one active at the same time, or that that current one must be open ended. From a practical point of view it might be advantageous if you maintain only a single contract with a customer at all times…

There are trade-offs, of course, but I'm generally not a fan of using implicit defaults for business applications (i.e., the application infers the default when there's no data). If things go well, business data outlives business applications. After years or decades, it can be a major pain to figure out all the "secret" values that aren't actually in the data.

I think this depends on if you are talking about figuring out what contract a company had some time ago (say 18 months), or if you talk about the current default value. The last one should be very easy to find in the code and if you are migrating to a new system the information should be transferred in some way. For historical things you obviously has to store the data somewhere and then just figure out a nice way to do that. In this case it could just be a table of default contracts with some fields and a startdate. Then you can query for any point in the past and get the default contract that was active at that point.

Re: Applying “make invalid states unrepresentable”

#190

Post by mr_gibbins »

The OP's conclusions are askew. If you remove the default contracts from the contracts table you are left with fixed term contracts. This means customers without fixed term contracts are assumed to have default contracts. Agree so far. But removing the default contracts removes the contract start date. Meaning the default start date HAS to be the customer start date and not a separate date. This changes the functionality of the system, not the state representation.
Post reply on HN