This is part of our lunch study on the foundations of LLMs. This talk explains all the parts of the transformer architecture and why each is needed.
The notebook uses a mini-language as a sample problem: sentences like “THE CAT EATS THE MOUSE IN THE YARD” are generated from a small set of grammatical rules that match articles to nouns, prepositional phrases, and subject-predicate or subject-predicate-object structure with intransitive and transitive verbs. Sentences like this can be generated and identified by an algorithm (the same one that parses programming languages), but we develop neural networks to do the same thing and check against the algorithm.
We study the following cases:
- fully connected neural network (not a transformer)
- encoder-only to identify validity, without a position encoding
- same with a position encoding
- encoder-decoder to generate valid sentences.
Presented as a Jupyter notebook. There is no Docker container for this session; users should run transformer.ipynb direction in Jupyter.