Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Autoresearch@home

ensue-network.ai

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Show HN: Autoresearch@home

#11

Post by itissid »

Lerc wrote: post #10

When training lots of models with subtly different parameters like this, Is there anything to be learned from the differences in logprobs between them for the same input. Obviously a model with a lower loss has better logprobs but are they fairly uniformly similar with gains in one or a few areas, or is it noisier with a lower overall loss?

> are they fairly uniformly similar with gains in one or a few areas, or is it noisier with a lower overall loss?

It seems like you want to know what median, 5-95 or 1-99 differences might be? I also wonder how the "residual" plot looks like... If there are too many residual data points for a scatter plot then a histogram might be useful to visualize the modes. I suspect that as loss decreases multiple modes should condense or altogether collapse into one.

Re: Show HN: Autoresearch@home

#14

Post by austinbaggio »

Bossie wrote: post #13

What is being researched? Any objective?

The objective is to train a small GPT language model to the lowest possible validation bits-per-byte (val_bpb) in 5-minute runs, using AI agents to autonomously iterate on the code. This builds on Karpathy's autoresearch: https://x.com/AustinBaggio/status/2031888719943192938?s=20

Re: Show HN: Autoresearch@home

#15

Post by HanClinto »

Trigger warning: very stupid question to follow.

To my smooth-brain naiveté, this feels like the sort of thing that we could reward with some sort of cryptocurrency in a blockchain? It's difficult to achieve gains, but it's relatively quick to independently verify (5 minute training runs).

I know "blockchain all the things" is sooooo 8 years ago, but I'm looking at the descending graph of progress here, and wondering if being able to claim improvement tokens (even for no reason other than NFT-esque bragging rights) wouldn't be a cool thing here?

I'm asking this as someone who knows next to nothing about crypto or blockchain or any of those things, but mainly thinking of trying to gamify assigning GPU rigs to "mining" these measurable improvements.

Re: Show HN: Autoresearch@home

#16

Post by gavinray »

HanClinto wrote: post #15

Trigger warning: very stupid question to follow. To my smooth-brain naiveté, this feels like the sort of thing that we could reward with some sort of cryptocurrency in a blockchain? It's difficult to achieve gains, but it's relatively quick to independently verify (5 minute training runs). I know "blockchain all the things" is sooooo 8 years ago, but I'm looking at the descending graph of progress here, and wondering…

I had this idea in 2019 -- "p2p machine learning" with some sort of incentive.

It makes a lot of sense to incentivize contributing. My idea was that you could earn "credits" for participating, which you could exchange for compute resources from the "swarm" later, on any given problem (training, local inference, etc).

Re: Show HN: Autoresearch@home

#18

Post by gabia »

austinbaggio wrote: post #6
gabia wrote: post #5

Cool! However when I click the commit_url links I get a 404 page at github.

We thought about storing all of the commits on Ensue too, but we wanted to match the spirit of Andrej's original design, which leans heavily on github. Curious what you were looking for when trying to inspect the code?

I was hoping to see the code change the agent made! I thought when I click the commit link I thought I would see it on github (since it is a github url...), but the links don't seem to work, they take me to github 404. e.g. https://github.com/mutable-state-inc/autoresearch-at-home/co... I'm not sure what that has to do with Ensue so I've probably misunderstood how this works.
Post reply on HN