Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A simulation of me: fine-tuning an LLM on 240k text messages

edwarddonner.com

Post reply on HN
Print view View original

11–20 of 145 posts

Re: A simulation of me: fine-tuning an LLM on 240k text messages

#11

Post by Sohcahtoa82 »

I tried this myself using GPT-2J and nearly 15 years of IRC logs trimmed down to only contexts that include me. Each training item would be my message and the preceding 20 messages. Still ended up being an 80 MB log file of over 1M lines.

The results were...not good at all. I ran an IRC bot that would watch the channel, and any time someone sent a message, it would send the last 20 messages to the trained model to see what it would predict the next message to be. It generated mostly garbage. Generated chat messages were on-topic, but non-sensical. It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related.

At some point, I'll probably try again with a far better model. Likely once I do my next system overhaul so I can do it locally. The only thing I'm not sure of is how to properly train and generate. GPT3 and its predecessors are very much simple text-completion AIs, whereas it seems all the models today are designed for a conversation back-and-forth between the AI and a user.

Re: A simulation of me: fine-tuning an LLM on 240k text messages

#12

Post by jokethrowaway »

Sohcahtoa82 wrote: post #11

I tried this myself using GPT-2J and nearly 15 years of IRC logs trimmed down to only contexts that include me. Each training item would be my message and the preceding 20 messages. Still ended up being an 80 MB log file of over 1M lines. The results were...not good at all. I ran an IRC bot that would watch the channel, and any time someone sent a message, it would send the last 20 messages to the trained model to se…

> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related.

An accurate depiction of what's going on!

I had terrible results with GPT2 but the newer models are much better. I recommend fine tuning mistral (I had better results than any llama based model)

Re: A simulation of me: fine-tuning an LLM on 240k text messages

#13

Post by alchemist1e9 »

mewpmewp2 wrote: post #8
wyldfire wrote: post #2

> I immediately recognized this would be in extremely poor taste and removed the data. Enough on that topic! Poor taste for a blog post. But I wonder if LLMs could advance to a point where it might be therapeutic to have a conversation with a deceased loved one.

> it might be therapeutic to have a conversation with a deceased loved one. Would it? I'm not a specialist, but it would seem a bit off to me to approach it like that. I'd think best is to accept that your deceased one has gone, and it seems like having a fake one might just prolong the grievance process and possibly also develop into an obsession of trying to replace your deceased one with the simulation, upgrading…

Have you ever watched the original Superman movies and Clark talks with the AI hologram of his biological father from his home planet?

I believe in the future our descendants will likely find it normal to talk with AI versions of their ancestors, perhaps not us parents right now, but I suspect my children will willingly train and generate AI versions of themselves for future generations to interact with.

Re: A simulation of me: fine-tuning an LLM on 240k text messages

#14

Post by [deleted] »

AriedK wrote: post #10
wyldfire wrote: post #2

> I immediately recognized this would be in extremely poor taste and removed the data. Enough on that topic! Poor taste for a blog post. But I wonder if LLMs could advance to a point where it might be therapeutic to have a conversation with a deceased loved one.

It has been tried. Joshua Barneau tried it with GPT-3, training it on his deceased fiancees text- and facebook messages. It’s a moving read: https://www.sfchronicle.com/projects/2021/jessica-simulation... It ‘worked’ for him for a while.

[deleted]

Re: A simulation of me: fine-tuning an LLM on 240k text messages

#15

Post by a_e_k »

wyldfire wrote: post #2

> I immediately recognized this would be in extremely poor taste and removed the data. Enough on that topic! Poor taste for a blog post. But I wonder if LLMs could advance to a point where it might be therapeutic to have a conversation with a deceased loved one.

Something like that is the core idea behind Black Mirror S2E1, "Be Right Back" (2013).

https://en.wikipedia.org/wiki/Be_Right_Back

Re: A simulation of me: fine-tuning an LLM on 240k text messages

#16

Post by Sohcahtoa82 »

jokethrowaway wrote: post #12
Sohcahtoa82 wrote: post #11

I tried this myself using GPT-2J and nearly 15 years of IRC logs trimmed down to only contexts that include me. Each training item would be my message and the preceding 20 messages. Still ended up being an 80 MB log file of over 1M lines. The results were...not good at all. I ran an IRC bot that would watch the channel, and any time someone sent a message, it would send the last 20 messages to the trained model to se…

> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. An accurate depiction of what's going on! I had terrible results with GPT2 but the newer models are much better. I recommend fine tuning mistral (I had better results than any llama based model)

> I recommend fine tuning mistral

What kind of hardware does that take?

I've got an RTX 3080, i9-9900K, and 32 GB of RAM right now. I eventually plan on upgrading to an RTX 5090 when those come out, as well as i9-15900K (Or whatever they're gonna call it) and 128 GB of RAM once the next Intel CPU comes out.

My understanding is that even with an RTX 5090 (Assuming 32 GB of VRAM, nothing has been announced yet), if I use fp32, I still won't have enough GPU memory to do inference on a 7B model, let alone training. Though if I quantize down to fp16, I could do inference, but still not training.

Re: A simulation of me: fine-tuning an LLM on 240k text messages

#17

Post by zoklet-enjoyer »

I remember having a similar idea near the end of high school. So 2006ish. I spent a lot of time online, on forums and instant messaging friends I knew in real life and just online. I saved all of my conversations because I thought it would be useful later on in life if I wanted to write a biography, read conversations like a diary, or analyze it with a computer program and create a chatbot. I saved text messages and emails because of this idea too. I don't have any of that data anymore. Lost most of it between 2010 and 2013.

Re: A simulation of me: fine-tuning an LLM on 240k text messages

#18

Post by eightysixfour »

Sohcahtoa82 wrote: post #16
jokethrowaway wrote: post #12

Earlier quoted context omitted.

> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. An accurate depiction of what's going on! I had terrible results with GPT2 but the newer models are much better. I recommend fine tuning mistral (I had better results than any llama based model)

> I recommend fine tuning mistral What kind of hardware does that take? I've got an RTX 3080, i9-9900K, and 32 GB of RAM right now. I eventually plan on upgrading to an RTX 5090 when those come out, as well as i9-15900K (Or whatever they're gonna call it) and 128 GB of RAM once the next Intel CPU comes out. My understanding is that even with an RTX 5090 (Assuming 32 GB of VRAM, nothing has been announced yet), if I u…

Pretty sure an fp32 7b model uses just shy of 27gb of vram.

The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you try and run it fp32 or fp16?

Post reply on HN