There's been talk in the past about github adding integrations with common build tools (automatically?). What if you could compile every llvm-compiled project on github and run a diffusion model over the intermediate representations?
Diffusion on syntax trees for program synthesis
61–70 of 95 posts
Re: Diffusion on syntax trees for program synthesis
#62Earlier quoted context omitted.
It is a computationally clever application of the chain rule to minimize the amount of computation needed to compute gradients for all parameters in the network.
> to minimize the amount of computation IMO backprop is the most trivial implementation of differentiation in neural networks. Do you know an easier way to compute gradients with larger overhead? If so, please share it.
My first forays into making neural networks used replacement rules to modify an expression tree until all the “D” operators went away, but that takes exponential complexity in network depth if you aren’t careful. Finite differences is linear in number of parameters, as is differentiation by Dual Numbers
Re: Diffusion on syntax trees for program synthesis
#63Re: Diffusion on syntax trees for program synthesis
#64It's funny, this kind of subtree mutation was looked at pretty deeply by Koza and Adamı in the 90s under the rubric of Genetic Algorithms, but with a slightly different optimization function One ref in the paper to 2000 for GAs for fast generation of program trees, but that's missing the main show Hope they're reading this and dig into those guys work
Wow, what a flash from the past! Was playing around a LOT with GP around that time, and the name Koza is certainly familiar. I even think I did some semi-similar things, instead of my normal approach which was simplistic but inefficient in that lots of invalid code was generated.
Re: Diffusion on syntax trees for program synthesis
#65Earlier quoted context omitted.
What's with the last author/first author thing in science papers? I've read several times that the author listed last is usually the most significant contributor, and the first author the least significant, due to some kind of tradition around modesty plus favourably introducing new names. (Which then of course doesn't work, if everyone knows it's happening...) Here, you've interpreted it as the reverse, and by that…
Conventions vary by field, but within a specific field they're usually pretty consistent. In natural sciences (except large physics papers) the convention is that the first author is the one doing most of the practical work. The last author is the PI (principal investigator) of the group who had a hand in designing the experiments and oversaw the research. Now, the latter can mean anything from barely doing any work…
Thank you, that makes my apparently half-formed prior understanding make a lot more sense. Seems the path forward is researching the greater context of the last authors work, and of course the common convention for each field.
Eg, as a sibling commented this convention may not be common in ML research, I wonder then if it may be something less common in emergent fields where researchers are generally be younger and less likely to follow older traditions.
Re: Diffusion on syntax trees for program synthesis
#66There's been talk in the past about github adding integrations with common build tools (automatically?). What if you could compile every llvm-compiled project on github and run a diffusion model over the intermediate representations?
what's the output?
Re: Diffusion on syntax trees for program synthesis
#67Earlier quoted context omitted.
What's with the last author/first author thing in science papers? I've read several times that the author listed last is usually the most significant contributor, and the first author the least significant, due to some kind of tradition around modesty plus favourably introducing new names. (Which then of course doesn't work, if everyone knows it's happening...) Here, you've interpreted it as the reverse, and by that…
>the author listed last is usually the most significant contributor Where did you read that? That's definitely not the case in machine learning.
I've read it in several places over the years, and just had a search now to find a reference to cite. Here's a PubMed paper on the subject:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3010799/
(And note how points 1 through 4 all quite conflict with each other!)
Re: Diffusion on syntax trees for program synthesis
#68There's been talk in the past about github adding integrations with common build tools (automatically?). What if you could compile every llvm-compiled project on github and run a diffusion model over the intermediate representations?
what's the output?
I guess the output would be an llvm intermediate representation that you can compile down and run, right? I'm stretching pretty far past my knowledge here.
Re: Diffusion on syntax trees for program synthesis
#69Earlier quoted context omitted.
>the author listed last is usually the most significant contributor Where did you read that? That's definitely not the case in machine learning.
I've read it in several places over the years, and just had a search now to find a reference to cite. Here's a PubMed paper on the subject: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3010799/ (And note how points 1 through 4 all quite conflict with each other!)
If it helps think of the first author as the lead engineer or the CEO, and the last author as the board or the VC. In some areas of science (or some teams) the last author is closer to a CEO, in others closer to a VC (they almost always have the powers of the board). This picture does not contradict the guideline in the reference you shared. Typically, most of the work and writing of the paper is done by the first author, though sometimes, for example when a student gives up, only most of the writing of the paper. The main ideas may be from the first author or from the last author, and rarely in between, but the sorting typically goes by amount of labor/contributions on the task. In some narrow subfields, including most math, sorting is alphabetical or random.
Re: Diffusion on syntax trees for program synthesis
#70Earlier quoted context omitted.
what's the output?
I guess the output would be an llvm intermediate representation that you can compile down and run, right? I'm stretching pretty far past my knowledge here.
I think the question - at least, for me - is "what do you expect to do with this system?" What will you output with your diffusion model?