Deep learning to translate between programming languages

As far as I can tell, this article gives no definition of what they mean by "successfully translate" in their testing. I would think it should be equivalent to the typical use of porting, which usually means you move the codebase from one language to another, reproduce all existing functionality, introduce no new bugs, and maintain relative performance. I am not seeing anything in this article other than their translator translates functions to functions. It also doesn't mention what failure looks like. Does it mean bugs are introduced? Does it mean the translator doesn't return? How are the percentages arrived at?

Also, I don't fully understand the why, which is also not really addressed. What problem is automatically translating a mess of a code base (I am assuming it's a mess, otherwise, why would you want to port it) to another programming language, leaving the mess in place, solving? Isn't the idea of porting to another language usually to help improve the codebase and/or to help integrate it into some other system? Is the idea of automatic translators that it does the "heavy" lifting that is then gone in and cleaned up by experts of the new language? How do you manage the failures, since that would seem you'd still need an expert in the original language?

Sounds pretty risky to me...

> To better measure the performance of TransCoder and other code translation techniques, we’ve created a new metric called computational accuracy, which evaluates whether the hypothesis function generates the same outputs as the reference when given the same inputs. We are also releasing our test set and the scripts and unit tests we used to compute this metric.

It reads as if they used a test suite to confirm wether the translation represents the same function.

Wasn't this on HN a few weeks ago? Or was that a similar project?

It's a fascinating idea. But now you need some way to check the result. Last time around, someone noted that some of the translations were plausible but wrong.

There's clearly some guessing. The translation of one function from Python to C++ resulted in untyped bounds becoming ints, which is OK. But it also resulted in a data array becoming ints, although the Python operations were generic over any type with arithmetic operators. The Python code might have been used for floats. It's not doing type inference by looking at the callers; it's just guessing.

Still, it's promising, even if it doesn't do the whole job. There's a C++ to Rust translator, but it turns array indexing into pointer arithmetic on special C++ compatible types. It's compiling into lower level operations. Deep learning has the potential to recognize and use idioms of the target language. Maybe.

But this needs a checker. Perhaps something that runs both language versions in lockstep on test data and checks for disagreement at key points.

Cool idea...I would be more interested in auto-documentation tool based on this.