https://davidan.dev/tags/fundamentals/ Recent content in fundamentals on David An Hugo -- gohugo.io en-us Wed, 01 Oct 2025 00:00:00 +0000 https://davidan.dev/posts/tokenization/ Wed, 01 Oct 2025 00:00:00 +0000 https://davidan.dev/posts/tokenization/ Lately, all we have heard about is tokenization and embeddings and the role they play in the greater LLM and AI ecosystem. These two concepts are one of the most fundamental concepts in language modeling and remain the foundation of the technology we interact with on a daily basis. In this article, we will cover some of the basics around tokenizing and embedding sequences of texts and the nuances of them.