Question

A lexical analyzer uses the following token definitions
 𝑙𝑒𝑡𝑡𝑒𝑟→[𝐴−𝑍𝑎−𝑧]
 𝑑𝑖𝑔𝑖𝑡→[0 −9]
 𝑖𝑑→𝑙𝑒𝑡𝑡𝑒𝑟 (𝑙𝑒𝑡𝑡𝑒𝑟 | 𝑑𝑖𝑔𝑖𝑡)*
 𝑛𝑢𝑚𝑏𝑒𝑟→𝑑𝑖𝑔𝑖𝑡+
 𝑤𝑠→(𝑏𝑙𝑎𝑛𝑘 | 𝑡𝑎𝑏 | 𝑛𝑒𝑤𝑙𝑖𝑛𝑒)+
For the string given below,
𝑥1 23𝑚𝑚 78 𝑦 7𝑧 𝑧𝑧5 14𝐴 8𝐻 𝐴𝑎𝑌𝑐𝐷
the number of tokens (excluding 𝑤𝑠) that will be produced by the lexical analyzer
is __________. (answer in integer)

Hint:

An id token can absorb trailing digits after its first letter, but a number token made of leading digits cannot absorb a following letter, so a new token starts wherever digits are followed by a letter.

Answer 1

Correct Answer: 13

Since \(id\) must begin with a letter and \(number\) must consist purely of digits, a chunk of non-whitespace characters splits into a new token every time the character class changes from digit to letter, but not the other way around (because once an \(id\) starts with a letter, it can absorb trailing digits too). Applying this rule chunk by chunk: \(x1\) begins with a letter so digits after it are absorbed into the same \(id\), giving 1 token. \(23mm\) begins with digits, which form their own \(number\) token, and the letters that follow cannot be absorbed backward, so they form a second, separate \(id\) token, giving 2 tokens. \(78\) is purely numeric, 1 token. \(y\) is a lone letter, 1 token. \(7z\) again splits into a leading \(number\) and a trailing \(id\), 2 tokens. \(zz5\) begins with letters, so the trailing digit gets absorbed into the same \(id\), giving just 1 token. \(14A\) and \(8H\) each split into a leading \(number\) and trailing \(id\), contributing 2 tokens apiece. Finally \(AaYcD\) is entirely letters, forming 1 token. Summing across all nine whitespace-separated chunks: \(1+2+1+1+2+1+2+2+1 = 13\) tokens are produced in total, excluding whitespace.