Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt

karpathy.github.io

Post reply on HN
Print view View original

61–70 of 354 posts

Re: Microgpt

#61

Post by fc417fc802 »

wasabi991011 wrote: post #40
Paddyz wrote: post #23

Earlier quoted context omitted.

[dead]

I still don't quite get your insight. Maybe it would help me better if you could explain it while talking like a pirate?

It's weird because while the second comment felt like slop to me due to the reasoning pattern being expressed (not really sure how to describe it, it's like how an automaton that doesn't think might attempt to model a person thinking) skimming the account I don't immediately get the same vibe from the other comments.

Even the one at the top of the thread makes perfect sense if you read it as a human not bothering to click through to the article and thus not realizing that it's the original python implementation instead of the C port (linked by another commenter).

Perhaps I'm finally starting to fail as a turing test proctor.

Re: Microgpt

#62

Post by altcognito »

red_hare wrote: post #56

This is beautiful and highly readable but, still, I yearn for a detailed line-by-line explainer like the backbone.js source: https://backbonejs.org/docs/backbone.html

ask a high end LLM to do it

Re: Microgpt

#63

Post by radlad »

darkpicnic wrote: post #55
jimbokun wrote: post #47

It’s pretty staggering that a core algorithm simple enough to be expressed in 200 lines of Python can apparently be scaled up to achieve AGI. Yes with some extra tricks and tweaks. But the core ideas are all here.

LLMs won’t lead to AGI. Almost by definition, they can’t. The thought experiment I use constantly to explain this: Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. We’ll need additional breakthroughs in AI.

I'm not sure - with tool calling, AI can both fetch and create new context.

Re: Microgpt

#64

Post by [deleted] »

darkpicnic wrote: post #55
jimbokun wrote: post #47

It’s pretty staggering that a core algorithm simple enough to be expressed in 200 lines of Python can apparently be scaled up to achieve AGI. Yes with some extra tricks and tweaks. But the core ideas are all here.

LLMs won’t lead to AGI. Almost by definition, they can’t. The thought experiment I use constantly to explain this: Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. We’ll need additional breakthroughs in AI.

[deleted]

Re: Microgpt

#65

Post by [deleted] »

wasabi991011 wrote: post #50
jimbokun wrote: post #47

It’s pretty staggering that a core algorithm simple enough to be expressed in 200 lines of Python can apparently be scaled up to achieve AGI. Yes with some extra tricks and tweaks. But the core ideas are all here.

1000 lines?? What is going on in this thread

[deleted]

Re: Microgpt

#66

Post by the_af »

kelvinjps10 wrote: post #59

Why there is multiple comments talking about 1000 c lines, bots?

Or even 1000 python lines, also wrong.

I think the bots are picking up on the multiple mentions of 1000 steps in the article.

Re: Microgpt

#67

Post by 0xbadcafebee »

Since this post is about art, I'll embed here my favorite LLM art: the IOCCC 2024 prize winner in bot talk, from Adrian Cable (https://www.ioccc.org/2024/cable1/index.html), minus the stdlib headers:

  #define a(_)typedef _##t
  #define _(_)_##printf
  #define x f(i,
  #define N f(k,
  #define u _Pragma("omp parallel for")f(h,
  #define f(u,n)for(I u=0;uc[t*W+k]*y[i*W+k];v[h]=
  _*X->f[t]*l[i]+!!i*v[h];}x D-c)i[r]+=v[i];}I main(){A=mmap(0,8e9,1,2,f=open(M,f),0);x 2)~f?i[G]=malloc(3e9):exit(
  puts(M" not found"));x V)i[P]=(C*)A+4,A+=(I)*A;g(&m,V)g(&n,V)g(e,D)g(d,H)for(C*o;;s>=D?$=s=0:p=U)for(_()("\n\n> "),0>\n"S"\n>\n\n",Y);z=p-=z;U++[o+=z,B]=f)for(f=0;!f;z-=!f)for(f=V;--f&&f[P][z]|memcmp(f[P],o,z););pc[R]*m->f[R/W];R=s++;N Z){f=k*D*D,$=W;x 3)j(k,L,D,i?G[~-i]+f+R*D:v,e[i]+k);N
  2)x D)b=sin(w=R/exp(i%E/14.)),c=1[w=cos(w),T=i+++(k?v:*G+f+R*D)],T[1]=b**T+c*w,*T=w**T-c*b;u Z){F*T=O[h],w=0;I A=h*E;x
  s){N E)i[k[L+A]=0,T]+=k[v+A]*k[i*D+*G+A+f]/11;w+=T[i]=exp(T[i]);}x s)N E)k[L+A]+=(T[i]/=k?1:w)*k[i*D+G[1]+A+f];}j(V,L
  ,D,J,e[3]+k);x 2)j(k+Z,L,H,i?K:a,d[i]+k);x H)a[i]*=K[i]/(exp(-a[i])+1);j(V,a,D,L,d[$=H/$,2]+k);}w=j($=W,r,V,k,n);x
  V)w=k[i]>w?k[$=i]:w;}}

Re: Microgpt

#68

Post by the_af »

ViktorRay wrote: post #51
wasabi991011 wrote: post #50

Earlier quoted context omitted.

1000 lines?? What is going on in this thread

It’s pretty sad. The only way we know these comments are from AI bots for now is due to the obvious hallucinations. What happens when the AI improves even more…will HN be filled with bots talking to other bots?

What's bizarre is this particular account is from 2007.

Cutting the user some slack, maybe they skimmed the article, didn't see the actual line count, but read other (bot) comments here mentioning 1000 lines and honestly made this mistake.

You know what, I want to believe that's the case.

Re: Microgpt

#69

Post by coolThingsFirst »

Incredibly fascinating. One thing is that it seems still very conceptual. What id be curious about how good of a micro llm we can train say with 12 hours of training on macbook.

Re: Microgpt

#70

Post by antupis »

tehjoker wrote: post #60
darkpicnic wrote: post #55

Earlier quoted context omitted.

LLMs won’t lead to AGI. Almost by definition, they can’t. The thought experiment I use constantly to explain this: Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. We’ll need additional breakthroughs in AI.

Part of the issue there is that the data quantity prior to 1905 is a small drop in the bucket compared to the internet era even though the logical rigor is up to par.

Humans need way less data. Just compare Waymo to average 16 year-old with car.
Post reply on HN