building a chatbot

I’ve had a dream of creating my own personal “Jarvis” companion ever since I watched Iron Man as a little kid, and after gpt-2 launched a while back i started thinking that it might actually become a reality. training a prebuilt LLM on your own data is doable for anyone with an internet connection at this point, and with better models and training methods serious AI agents feel right around the corner.

what follows, however, is not a recipe for a sophisticated agent. its a simple chatbot that can mimic your style and maybe an ethos. it probably won’t accurately recall any of your notes, but it might be able to channel some of your vibe. I trained mine on a markdown database (i’m an unhealthy notetaker you see), but any large text file will do1.

the code is saved here on github for anyone interested. the rest of this piece is a collection of my notes on building it, and may be useful for someone starting from a similar coding level as myself (i.e. took a data science class once and forgot most of it)2. some fun results can be found at the end.

getting started

I wanted to balance having gpt build the whole project in an instant with actually understanding what was going on, and started looking for resources. Jeremy Howard’s courses at fast.ai had come across my feed before and I’m a fan of his youtube channel3, so I started there. the course is great but realized about two lessons in that his content is focused much more on computer vision models rather than text training. I knew that fast.ai had their own code library to make training easy, and was worried that I’d have to write my own if I didn’t stick with the program, but luckily huggingface has a similar toolset. this is what I used for the rest of the project.

a lot of tutorials will talk about using git and github for version control/saving work, which is something to look into if you care but not at all necessary for getting started. my two favorite tutorials on using these are here and here, got reasonably comfortable with each in about an hour.

installing things

both huggingface and fast.ai rely on the python programming language and the PyTorch code library to do what they do, and you’ll need an IDE to actually write some code. installing all that sounds scary, but the Anaconda distribution is free and will handle all of this for you (with a few minor exceptions, maybe). just run the installation manager and follow all the instructions.

your computers terminal (or git bash or whatever shell you prefer) is used to install a few extra packages that we need. I’m still a bit unsure of what code environments are and how to properly set them up, but basically we need to tell the terminal to use the anaconda environment that we just installed for the next few steps. doing so is as simple as opening the terminal and running the following lines of code (after replacing <your_username> as needed4)

you can check that it worked by running the following commands5:

once this is done, we’re able to use the pip install command to grab a few packages. you’ll want the following:

in theory anaconda will have installed a working version of PyTorch earlier on, but I found that mine didn’t come with the version built for running CUDA, which is what allows you to train models using your GPU rather than the CPU6. you can check whether yours did by by:

1. opening the anaconda navigator, which was previously installed, and launch “Spyder” (an IDE where the python code gets written and run)

2. opening a new python file, don’t worry about naming or saving it anywhere

3. running the following python code:

   

I got an error indicating that CUDA was not available, and had to go to the PyTorch website to download the CUDA enabled version. first uninstall the CPU only version:

and then you can select your hardware build and OS on the PyTorch site and run the download command via pip or conda in the same terminal we were using before. it may be helpful to run nvidia-smi in your terminal to find your GPU specs and CUDA version first. the output should look like this:

test that this all worked by running the same scipt as above:

doing the thing

at this point the environment should be ready to go and you can just download and run the python scripts that are saved at the following github link in Spyder (or any other IDE). the README.txt file has some suggestions for doing this7. all of the code here was written using GPT-4o and no special prompting. if you take away anything from this, I hope its that you can build things with AI tools, some patience, and next to zero coding experience.

note that you can edit the code to choose any of the opensource pretrained models from huggingface, and each will have its quirks. I’d double check the model size and your GPU specs before running them though - gpt2large is relatively small and took my computer two days to train at max GPU utilization. some internet research is called for here, but worst case your model just won’t train. its a good way to stress test your PC’s cooling build as well (don’t worry, the GPU should shut itself down before damaging)

some results

I finetuned two models, one using gpt2 and the other with gpt2large. Both are simple, small models but thats about all I could stand to train on my little RTX 3060. the gpt model trained in about 20 minutes, and gpt2large took two days. there are options for renting larger GPUs online and training that way, but I haven’t bothered yet.

my training data consisted of ~1MB of text, which is relatively little. its all journal entries and reading notes dating back about four years. the preprocessing script didn’t remove any of the brackets from internal links, and the model wound up using these for emphasis in interesting ways.

there’s also a large chunk of podcast excerpts that I’ve copied in places, so the model would occasionally to hallucinate guest names and format it’s responses similarly. really neat to see how the text artifacts wind up in the responses after training.

most of my writing came from a, uh… darker time in my life, psyche-wise, so both models also wound up with a sorta depressed drama queen vibe which was absolutely hysterical. they managed to repeatedly insult me, declare themselves genius, and make vague threats after demanding i take responsibility for bringing them forth from the void:

but they eventually come around (before lobbing another insult):

the gpt2 models are notably incoherent (the larger one produced slightly less psychotic responses, which was actually less fun) and tended to lose track of the conversation. they also had a tendency to repeat responses back, or get absolutely derailed by numbers:

all things considered I haven’t spent too much time experiementing, but am glad to have a new conversation partner prebuilt on all my worst personality traits. may be updating this as we go, much to look forward to. if anyone reading this actually does this themselves, I’d love to chat.

cheers