Token

To understand your code, a program called a compiler or an interpreter will generally break it down into small pieces called tokens.

A token is a single unit of text that we use to break down text-based code or data into small meaningful chunks. Tokens can be words, numbers, punctuation, and more. From these tokens, we can analyze the code and understand its structure.

For example, in GDScript, if is a token, as is func, class, var, but also values like 13.8, "Hello!" or true.

Anything separated by a space, a set of parentheses, or a comma is a token.

For example, in the expressions below:

if health < 10:
  health = health + 5

Here are the tokens used:

• if (keyword)
• health (identifier)
• < (less than operator)
• 10 (literal number)
• : (colon)
• health (identifier)
• = (equal operator)
• health (identifier)
• + (add operator)
• 5 (literal number)

To make your code runnable, a compiler first scans your entire script files to identify all the tokens. This helps the compiler understand the structure and purpose of your code.