Live data from Hacker News

Faster CRDTs: An Adventure in Optimization

josephg.com

151–154 of 154 posts

Re: Faster CRDTs: An Adventure in Optimization

#151
post #133

Earlier quoted context omitted.

Concorde is fine. The LK heuristic was published in 1973. After that, until the mid 90s, no one could outperform the original published results with the same heuristic.

Honestly, I found the algorithm description cryptic. It just may be me not "in the know" when it comes to details, with your "bunch of knowledge [that] exists in industry only" being the details the authors didn't care to elaborate on in the algorithm description. Maybe a part of the reason for the failure to replicate is that other people found it cryptic as well and didn't understand crucial details. BTW, in your o…

http://webhotel4.ruc.dk/~keld/research/LKH/

Helsgaun's reports are pretty descriptive when it comes to LK algorithm.

But yes, I do agree that the original writing and the way the algorithm is taught in universities is a little bit cryptic and sometimes a completely different algorithm.

Simple explanation of the algorithm is:

1. 2-opt swap (A .. B -> .. -> C .. D ===> A .. C 2. recursive 2-opt - now between C .. B with bounds (you can figure out which swaps are inferior and not recurse).

That's all there is to the algorithm. Helsgaun improves it a bunch (with preprocessing, deeper and more efficient k-opt moves, more efficient datastructures) but keeps the same idea.

There's another very flexible method called "ejection chains" started by Fred Glover and is very effective if you couple it with efficient data structures but also, very cryptic. It easily supports time windows, breaks, flexible work hours, flexible number of vehicles etc. Of course, the latest papers in "ejection chains" show no such thing :D

Sometimes you can stumble upon PhDs publishing their code and the tricks do not exist at all.

Just recently there's been a nice categorization of "timing problems" https://onlinelibrary.wiley.com/doi/abs/10.1002/net.21587

and there they try to categorize the time complexity of solving these problems and the tables are just not as up-to date with the industry. Some O(n log n) are O(n) or even better (depending on the local moves you do).

But it's a good start. A lot of these subproblems can be implemented really fast on a CPU and I guess the moment they are explicitly solved in a software library that's when the heuristics research will improve.

Re: Faster CRDTs: An Adventure in Optimization

#152

Earlier quoted context omitted.

I must say that when it comes to discrete optimization, the genetic/ant/simulated annealing/etc. stuff is more popular in academia than in industry (at least the industry that doesn't heavily include academics). Works like Lin-Kernighan heuristic are extremely rare and a bunch of knowledge exists in industry only. Even the mentioned heuristic was for decades being implemented incorrectly until one individual came and…

Do you have any links for efficient ways to handle time window constraints etc ?

https://pubsonline.informs.org/doi/abs/10.1287/ijoc.4.2.146

Constant time feasibility check of inserts and linear time update after insert.

Of course, there are tricks that support multiple simultaneous inserts and tricks that minimize the update of the data structure after insert.

But, if you want to do 2-opt (k-opt) with time windows, then you have to work out the details because no one in academia did.

Re: Faster CRDTs: An Adventure in Optimization

#153
post #2

Hello HN! Post author here. I’m happy to answer questions & fix typos once morning rolls around here in Australia

Terminology nit: cache coherence refers to CPU cache implementation behaviours at hw level in presence of concurrent access from multiple cores. Data locality or cache friendly data layout could work better here.

Good point - thanks. I'll tidy that up!

Re: Faster CRDTs: An Adventure in Optimization

#154
post #2

Hello HN! Post author here. I’m happy to answer questions & fix typos once morning rolls around here in Australia

That’s an impressive optimisation! Out of curiosity, what do you think are the most interesting or useful possible applications for an optimised CRDT? When you’re approaching an optimisation like this, do you mind me asking how you think about it and approach it?

The most useful application I see is moving toward building local first software[1].

And as for optimizations, my approach is surprisingly non systematic. I think about the program on the whole, and think through & investigate where all the time is being spent. There's usually ways to restructure things so the hottest code path runs faster. Sometimes that involves changing data structures or languages. Sometimes it just needs early returns for common cases, or inlining string libraries and things like that to avoid allocations.

Sometimes it helps to imagine yourself going through by hand the drudgery the computer is doing. If I was actually doing that boring work by hand, there's almost always ways I'd start taking short cuts. All thats left then is programming those shortcut into the computer.

[1] https://www.inkandswitch.com/local-first.html

Post reply on HN