jfun.parsec.tokens
public class Tokenizers extends Object
Since: version 0.6
Method Summary | |
---|---|
static Tokenizer | forChar()
Creates a tokenizer that's gonna tokenize a single quoted character literal possibly with escape character '\' |
static Tokenizer | forDecimal()
Get the Tokenizer object that creates a decimal number literal token. |
static Tokenizer | forDecLong()
Get the Tokenizer object that interprets the recognized character range
as a decimal integer and translate it to a long value. |
static Tokenizer | forHexLong()
Get the Tokenizer object that interprets the recognized character range
as a hex integer and translate it to a long value. |
static Tokenizer | forInteger()
Get the Tokenizer object that creates an integer literal token. |
static Tokenizer | forOctLong()
Get the Tokenizer object that interprets the recognized character range
as a oct integer and translate it to a long value. |
static Tokenizer | forQuotedString(char open, char close)
Get the Tokenizer object that converts a string literal quoted by a pair of
opening and closing characters.
|
static Tokenizer | forQuotedString(String open, String close)
Get the Tokenizer object that converts a string literal quoted by a pair of
opening and closing strings.
|
static Tokenizer | forReservedWord()
Get the Tokenizer object that creates a reserved word token. |
static Tokenizer | forSimpleStringLiteral()
Get the Tokenizer object that converts a string literal quoted by '"'
to a string object.
back-slash character is escaped. |
static Tokenizer | forSqlStringLiteral()
Get the Tokenizer object that converts a sql string literal quoted by single quote
to a string object.
double single quote is interpreted as one single quote. |
static Tokenizer | forString()
Get the Tokenizer object that creates a string object. |
static <T> Tokenizer | forTypedToken(T type)
Create a tokenizer that tokenizes the recognized character range
to a TypedToken object. |
static Tokenizer | forWord()
Get the Tokenizer object that creates a word token. |
Returns: the tokenizer instance.
Parameters: type the token type.
Returns: the Tokenizer object.