Lexical Analysis: Tokens, Lexemes, and Patterns

Lexical Analysis: Tokens, Lexemes, and Patterns

Understanding the First Phase of Compiler Design

---

Photo by Nicolas Arnold on Unsplash

• Role of the Lexical Analyzer
• Understanding Lexemes and Patterns
• What are Tokens?
• Lexical Analysis Process Example
• Summary of Compilation phases

---

Photo by Markus Spiske on Unsplash

1

The Role of Lexical Analysis

Bridging source code and machine-executable structure

---

Photo by Sasun Bughdaryan on Unsplash

• The Lexical Analyzer (Scanner) is the first phase of a compiler.
• Its primary task is to read the source code character by character.
• It groups characters into meaningful sequences called lexemes.
• It identifies the category of each lexeme and generates a token.
• It handles language-specific requirements like skipping whitespace and comments.
• It simplifies the task of the subsequent phase: The Parser.

What is a Lexeme? A lexeme is the actual sequence of characters in the source program that matches a pattern for a token. It is the raw 'word' extracted from the code. Example: 'int', 'x', '10'.

What is a Token? A token is an abstract category (symbolic name) assigned to a lexeme by the scanner. It is a pair consisting of a token name and an optional attribute value. Example: 'KEYWORD', 'IDENTIFIER', 'NUMBER'.

• The lexer processes the stream 'x = 10'.
• 'x' is identified as an IDENTIFIER.
• '=' is identified as an ASSIGNMENT operator.
• '10' is identified as a NUMBER constant.
• The output is a stream of tokens passed to the parser.

Lexical analysis transforms raw source code into structured data, enabling efficient parsing.

Final Thoughts

---

Photo by Sergei Nikulin on Unsplash