6.4 Lexical elements

token:
                keyword
                identifier
                constant
                string-literal
                punctuator

preprocessing-token:
                header-name
                identifier
                pp-number
                character-constant
                string-literal
                punctuator
                each non-white-space character that cannot be one of the above

767 Each preprocessing token that is converted to a token shall have the lexical form of a keyword, an identifier, a constant, a string literal, or a punctuator.

768 A token is the minimal lexical element of the language in translation phases 7 and 8.

769 The categories of tokens are: keywords, identifiers, constants, string literals, and punctuators.

770 A preprocessing token is the minimal lexical element of the language in translation phases 3 through 6.

771 The categories of preprocessing tokens are: header names, identifiers, preprocessing numbers, character constants, string literals, punctuators, and single non-white-space characters that do not lexically match the other preprocessing token categories.⁵⁸⁾

772 If a ' or a " character matches the last category, the behavior is undefined.

773 Preprocessing tokens can be separated by white space;

774 this consists of comments (described later), or white-space characters (space, horizontal tab, new-line, vertical tab, and form-feed), or both.

775 As described in 6.10, in certain circumstances during translation phase 4, white space (or the absence thereof) serves as more than preprocessing token separation.

776 White space may appear within a preprocessing token only as part of a header name or between the quotation characters in a character constant or string literal.

777 58) An additional category, placemarkers, is used internally in translation phase 4 (see 6.10.3.3); it cannot occur in source files.

778 If the input stream has been parsed into preprocessing tokens up to a given character, the next preprocessing token is the longest sequence of characters that could constitute a preprocessing token.

779 There is one exception to this rule: a header name preprocessing token is only recognized within a #include preprocessing directive, and within such a directive, a sequence of characters that could be either a header name or a string literal is recognized as the former.

780 EXAMPLE 1 The program fragment 1Ex is parsed as a preprocessing number token (one that is not a valid floating or integer constant token), even though a parse as the pair of preprocessing tokens 1 and Ex might produce a valid expression (for example, if Ex were a macro defined as +1). Similarly, the program fragment 1E1 is parsed as a preprocessing number (one that is a valid floating constant token), whether or not E is a macro name.

781 EXAMPLE 2

The program fragment x+++++y is parsed as x ++ ++ + y, which violates a constraint on increment operators, even though the parse x ++ + ++ y might yield a correct expression.

782 Forward references: character constants (6.4.4.4), comments (6.4.9), expressions (6.5), floating constants (6.4.4.2), header names (6.4.7), macro replacement (6.10.3), postfix increment and decrement operators (6.5.2.4), prefix increment and decrement operators (6.5.3.1), preprocessing directives (6.10), preprocessing numbers (6.4.8), string literals (6.4.5).