Defunctionalization: Everybody does it, nobody talks about it (2019) • Hacker News Board

Re: Defunctionalization: Everybody does it, nobody talks about it (2019)

#21

Post by BobbyJo » Sat, Aug 07, 2021, 2:15 AM UTC

cratermoon wrote: post #12

choeger wrote: post #5

The thing is, and the article correctly mentions it, that defunctionalization sacrifices composability. This is not just a technical matter, mind you. If you design your distributed system around defunctionalized higher-order functions, you have to ensure that client and server have the same understanding of valid parameters. That's a non-trivial problem. So while this might be a cool technique for whole-program comp…

So you don't think that a website would not benefit this? Imagine a design where the user selects options in the browser front end for how to filter product search results, and the system has contract whereby the front end can call the search API with a set of valid filters and get the results back. Yes, I know that typically the front end would ask the back end, for example, all things in a category, then do the fil…

> Yes, I know that typically the front end would ask the back end, for example, all things in a category, then do the filtering within the search results at the browser

Please no.

Re: Defunctionalization: Everybody does it, nobody talks about it (2019)

#22

Post by galaxyLogic » Sat, Aug 07, 2021, 4:36 AM UTC

Isn't de-functionalization the only way systems can communicate over the network? You have to serialize your data say by turning it into JSON which you can not transport functions between the systems - which would also be a security risk.

Re: Defunctionalization: Everybody does it, nobody talks about it (2019)

#23

Post by remexre » Sat, Aug 07, 2021, 6:56 AM UTC

codeismath wrote: post #18

It's been on my list of things to explore for a while... the relationship between defunctionalization, continuations, and "Algebraic Effects". The reason... I like seeing how different languages handle asynchronous code vs synchronous code. F# has their way, C# has theirs, JavaScript async/await, colored functions, Project Loom, co-routines, React Fibers, etc. I'm intrigued that a language that has built-in algebraic…

I don't think the article's SIGIO example made much sense -- if you have delimited continuations, you _don't_ need defunctionalize them if they're staying in-process.

Re: Defunctionalization: Everybody does it, nobody talks about it (2019)

#24

Post by choeger » Sat, Aug 07, 2021, 12:38 PM UTC

cratermoon wrote: post #12

choeger wrote: post #5

The thing is, and the article correctly mentions it, that defunctionalization sacrifices composability. This is not just a technical matter, mind you. If you design your distributed system around defunctionalized higher-order functions, you have to ensure that client and server have the same understanding of valid parameters. That's a non-trivial problem. So while this might be a cool technique for whole-program comp…

So you don't think that a website would not benefit this? Imagine a design where the user selects options in the browser front end for how to filter product search results, and the system has contract whereby the front end can call the search API with a set of valid filters and get the results back. Yes, I know that typically the front end would ask the back end, for example, all things in a category, then do the fil…

You would use higher-order functions particularly when you cannot determine the scope of your input ahead of time. For instance when you provide something that searches an unstructured database of documents (e.g., mongo).

In such a scenario, you could apply defunctionalization. But you'd need to re-apply this technique whenever a new filter comes up or one becomes obsolete. Then you would have to make sure that both your client and server have the same version of the API.

If you design a functional system instead, say by allowing filters in a domain-specific language, ala jq, you write your filter function once and never need to touch the API, because it is "complete".

Which alternative do you prefer?

Re: Defunctionalization: Everybody does it, nobody talks about it (2019)

#25

Post by cratermoon » Sat, Aug 07, 2021, 3:23 PM UTC

BobbyJo wrote: post #21

cratermoon wrote: post #12

Earlier quoted context omitted.

So you don't think that a website would not benefit this? Imagine a design where the user selects options in the browser front end for how to filter product search results, and the system has contract whereby the front end can call the search API with a set of valid filters and get the results back. Yes, I know that typically the front end would ask the back end, for example, all things in a category, then do the fil…

> Yes, I know that typically the front end would ask the back end, for example, all things in a category, then do the filtering within the search results at the browser Please no.

This is fairly common when the data involved are relatively small. It's much faster and a much better user experience to query the server once for a couple of hundred items and allow the user to view it with different criteria than it is to do a round trip to the server every time the user wants to change or refine the selected data.

What's your objection to it?

Re: Defunctionalization: Everybody does it, nobody talks about it (2019)

#26

Post by cratermoon » Sat, Aug 07, 2021, 3:31 PM UTC

choeger wrote: post #24

cratermoon wrote: post #12

Earlier quoted context omitted.

So you don't think that a website would not benefit this? Imagine a design where the user selects options in the browser front end for how to filter product search results, and the system has contract whereby the front end can call the search API with a set of valid filters and get the results back. Yes, I know that typically the front end would ask the back end, for example, all things in a category, then do the fil…

You would use higher-order functions particularly when you cannot determine the scope of your input ahead of time. For instance when you provide something that searches an unstructured database of documents (e.g., mongo). In such a scenario, you could apply defunctionalization. But you'd need to re-apply this technique whenever a new filter comes up or one becomes obsolete. Then you would have to make sure that both…

> Which alternative do you prefer?

It depends. How hard it is to write the generalized functional system and how many filters do I need to support initially? How much do filters really change? The "but what if you need to change something" objection comes up quite a lot. In practice it is often better to apply the knowledge and requirements I have and evolve the system as my understanding changes than to try to anticipate all possible scenarios up front. In the latter case, if I put a lot of effort into something and I make the wrong call I either don't have as many changes as I thought, so it's wasted effort, or changes happen in ways I didn't anticipate, so my solution activity works against me.

Re: Defunctionalization: Everybody does it, nobody talks about it (2019)

#27

Post by sullyj3 » Sun, Aug 08, 2021, 12:13 AM UTC

This seems like exactly the sort of issue the Unison language is trying to tackle. They want straightforward execution of distributed computations using the same language, with no friction across machine boundaries, no sacrificing composibility.

The article mentions:

> remote procedure calls with higher-order functional arguments would require serializing those functions, which is not easy to do safely or efficiently.

This is exactly what Unison is designed to do. Code is not stored as text, but is stored in a serialized AST form and identified by hash. Therefore, transmission between nodes is trivial. The language is still a work in progress, but it's making rapid strides.

Here's an explanation: https://www.youtube.com/watch?v=gCWtkvDQ2ZI

The part relevant to distributed computation is at 30:08, but the earlier parts go through the basics of how code is stored, and why this design choice was made, which might help with understanding what's going on.

Re: Defunctionalization: Everybody does it, nobody talks about it (2019)

#28

Post by BobbyJo » Sun, Aug 08, 2021, 1:30 PM UTC

cratermoon wrote: post #25

BobbyJo wrote: post #21

Earlier quoted context omitted.

> Yes, I know that typically the front end would ask the back end, for example, all things in a category, then do the filtering within the search results at the browser Please no.

This is fairly common when the data involved are relatively small. It's much faster and a much better user experience to query the server once for a couple of hundred items and allow the user to view it with different criteria than it is to do a round trip to the server every time the user wants to change or refine the selected data. What's your objection to it?

I've just never seen a scenario where the amount of data wasn't too large for the browser to be responsive while doing things this way.