Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autoregressive Language Model on the 6502 Processor

mattbeton.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Autoregressive Language Model on the 6502 Processor

#11

Post by striking »

derefr wrote: post #7

> The model weights and inference code need to be contained within 25KB of user-space memory Wouldn’t it be era-appropriate to allow relying on banked memory? You’d still need to hold the inference code, but you could effectively stream(ing page) the weights as you compute on them.

From a look at https://en.wikipedia.org/wiki/BBC_Micro#Specifications I think the 6502 versions of the beeb didn't have banked RAM so to keep it loadable from tape the limits might be as stated.

But with substantial additional effort, maybe some banked ROMs could be added..?

Re: Autoregressive Language Model on the 6502 Processor

#13

Post by froh »

striking wrote: post #11
derefr wrote: post #7

> The model weights and inference code need to be contained within 25KB of user-space memory Wouldn’t it be era-appropriate to allow relying on banked memory? You’d still need to hold the inference code, but you could effectively stream(ing page) the weights as you compute on them.

From a look at https://en.wikipedia.org/wiki/BBC_Micro#Specifications I think the 6502 versions of the beeb didn't have banked RAM so to keep it loadable from tape the limits might be as stated. But with substantial additional effort, maybe some banked ROMs could be added..?

the banking was done via an external ram controller whose bank control register was mapped to some (unbanked) controller address.

Re: Autoregressive Language Model on the 6502 Processor

#14

Post by vintermann »

From my experience, modern machine learning models don't scale well down at all, and it's almost certainly better to just use a simple Markov chain variant of some sort - like Niall on the Amiga, or whatever Terry Pratchett used to come up with Foul Ol' Ron's catchphrase, "Millennium hand and shrimp".

Re: Autoregressive Language Model on the 6502 Processor

#16

Post by forinti »

striking wrote: post #11
derefr wrote: post #7

> The model weights and inference code need to be contained within 25KB of user-space memory Wouldn’t it be era-appropriate to allow relying on banked memory? You’d still need to hold the inference code, but you could effectively stream(ing page) the weights as you compute on them.

From a look at https://en.wikipedia.org/wiki/BBC_Micro#Specifications I think the 6502 versions of the beeb didn't have banked RAM so to keep it loadable from tape the limits might be as stated. But with substantial additional effort, maybe some banked ROMs could be added..?

You could buy a board called Integra-B to expand a model B.

https://www.youtube.com/watch?v=lT07uPRRPfU

A few games required some extra RAM.

Update: Actually, there were other boards. I remember one particular advert for a board that could handle 256KB of RAM/ROM. I'm now actually curious if adding too many ROMs would make the * commands noticeably slower (the OS would look for the commands in the ROMs, one by one, until it found the code to handle them).

Re: Autoregressive Language Model on the 6502 Processor

#17

Post by numpad0 »

striking wrote: post #11
derefr wrote: post #7

> The model weights and inference code need to be contained within 25KB of user-space memory Wouldn’t it be era-appropriate to allow relying on banked memory? You’d still need to hold the inference code, but you could effectively stream(ing page) the weights as you compute on them.

From a look at https://en.wikipedia.org/wiki/BBC_Micro#Specifications I think the 6502 versions of the beeb didn't have banked RAM so to keep it loadable from tape the limits might be as stated. But with substantial additional effort, maybe some banked ROMs could be added..?

Banking RAM is most simply done by physically wiring RAMs in parallel and routing Chip Select/Chip Enable pins individually from GPIO. People were doing it in PDAs until late 00s. Also, I/O ports on these old machines and CPUs up until switch to PCI Express were philosophically just DRAM slots in alternate shapes, so RAMs could be more or less simply installed and RAMs on adapter cards just accessed. This is also technically applicable to PCIe from various perspectives other than performance.
Post reply on HN