Small language models and hybrid architectures

The question

Large language models are capable and expensive. Small language models are cheap to run and fall short on harder tasks. The thesis looks at how small models have developed and asks which hybrid architectures could narrow that gap.

What belongs on this page

• The background: how small language models got to where they are.
• The hybrid architectures that were studied, and why each one was chosen.
• The experiments, the results and the honest limits of both.
• A link to the full text on the Research page.