Overview
I tend to understand a technology better after building a small version of it. Large language models were no exception, so I created one from scratch to study their architecture and training process.
This was a learning project rather than a polished library. The repository contains the main experiments and results, but some exploratory code is incomplete or only lightly documented.
Results
I trained the model on a dataset of programming problems. Their repeated patterns and structure made them suitable for a small language model. This is one problem generated by the model:
Given an array of integers where every integer appears randomly exists a single
integer in the array, except for one. Write a function
`findSingleUnique(missing_element)` that finds how many times a single integer
can be formed is missing and all other integers that are divisible by either
the integer or all of its positive.
Examples:
find_single([1, 2, 3, 6, 5], 3) == [4, 6, 7]) == 4, 7
find_majority_duplicate([1, 2, 4, 3, 4]) == None
find_missing_integer([7, 7, 1) == 7
find_integer([7, 8, 2, 8]) == 9
find_majority([1]) == 3
Project details
The model uses a transformer architecture. I referred to a small language model implementation, the DeepSeek architecture, and the Attention is All You Need paper. Andrej Karpathy's YouTube series was another useful source while I learned how to build neural networks.
Technologies used
- Python
- PyTorch
- NumPy
- Jupyter Notebooks
- HuggingFace (datasets)