Here, I found several months ago, I have write this suggestion Re: New code completion remarks/issues (http://forums.next.codeblocks.org/index.php/topic,11187.msg79616.html#msg79616)
Today, I found more evidence, see here:
http://www.macs.hw.ac.uk/~alison/alg/lectures/l7.pdf
Here is the test in this pdf:
QuoteThe tokeniser should extract both the text
(lexeme) and the class of the item. So, a token
for "35" should contain the text (35) and the
type NUM.
also
QuoteSo.. a suitable datatype for a token will be a
struct or class such as the following:
struct token {
char* text;
tokentype type;
};
where tokentype is an enumerated type
specifying possible types of token.
So, a token type is necessary. :D
By the way, this site:
The Mini C++ Interpreter (http://www.excode.ru/art5843.html) give a simple example of a C++ parser. There is a chapter describing this mini C++ interpreter in the book "The art of C++".