jfun.parsec.tokens

Class Tokenizers

public class Tokenizers extends Object

The facade class for all pre-built Tokenizer implementations.

Since: version 0.6

Author: Ben Yu Apr 27, 2006 7:40:16 PM

Method Summary
static TokenizerforChar()
Creates a tokenizer that's gonna tokenize a single quoted character literal possibly with escape character '\'
static TokenizerforDecimal()
Get the Tokenizer object that creates a decimal number literal token.
static TokenizerforDecLong()
Get the Tokenizer object that interprets the recognized character range as a decimal integer and translate it to a long value.
static TokenizerforHexLong()
Get the Tokenizer object that interprets the recognized character range as a hex integer and translate it to a long value.
static TokenizerforInteger()
Get the Tokenizer object that creates an integer literal token.
static TokenizerforOctLong()
Get the Tokenizer object that interprets the recognized character range as a oct integer and translate it to a long value.
static TokenizerforQuotedString(char open, char close)
Get the Tokenizer object that converts a string literal quoted by a pair of opening and closing characters.
static TokenizerforQuotedString(String open, String close)
Get the Tokenizer object that converts a string literal quoted by a pair of opening and closing strings.
static TokenizerforReservedWord()
Get the Tokenizer object that creates a reserved word token.
static TokenizerforSimpleStringLiteral()
Get the Tokenizer object that converts a string literal quoted by '"' to a string object. back-slash character is escaped.
static TokenizerforSqlStringLiteral()
Get the Tokenizer object that converts a sql string literal quoted by single quote to a string object. double single quote is interpreted as one single quote.
static TokenizerforString()
Get the Tokenizer object that creates a string object.
static <T> TokenizerforTypedToken(T type)
Create a tokenizer that tokenizes the recognized character range to a TypedToken object.
static TokenizerforWord()
Get the Tokenizer object that creates a word token.

Method Detail

forChar

public static Tokenizer forChar()
Creates a tokenizer that's gonna tokenize a single quoted character literal possibly with escape character '\'

Returns: the tokenizer instance.

forDecimal

public static Tokenizer forDecimal()
Get the Tokenizer object that creates a decimal number literal token.

forDecLong

public static Tokenizer forDecLong()
Get the Tokenizer object that interprets the recognized character range as a decimal integer and translate it to a long value.

forHexLong

public static Tokenizer forHexLong()
Get the Tokenizer object that interprets the recognized character range as a hex integer and translate it to a long value.

forInteger

public static Tokenizer forInteger()
Get the Tokenizer object that creates an integer literal token.

forOctLong

public static Tokenizer forOctLong()
Get the Tokenizer object that interprets the recognized character range as a oct integer and translate it to a long value.

forQuotedString

public static Tokenizer forQuotedString(char open, char close)
Get the Tokenizer object that converts a string literal quoted by a pair of opening and closing characters. The Tokenizer result is a TokenQuoted object.

forQuotedString

public static Tokenizer forQuotedString(String open, String close)
Get the Tokenizer object that converts a string literal quoted by a pair of opening and closing strings. The Tokenizer result is a TokenQuoted object.

forReservedWord

public static Tokenizer forReservedWord()
Get the Tokenizer object that creates a reserved word token.

forSimpleStringLiteral

public static Tokenizer forSimpleStringLiteral()
Get the Tokenizer object that converts a string literal quoted by '"' to a string object. back-slash character is escaped.

forSqlStringLiteral

public static Tokenizer forSqlStringLiteral()
Get the Tokenizer object that converts a sql string literal quoted by single quote to a string object. double single quote is interpreted as one single quote.

forString

public static Tokenizer forString()
Get the Tokenizer object that creates a string object.

forTypedToken

public static <T> Tokenizer forTypedToken(T type)
Create a tokenizer that tokenizes the recognized character range to a TypedToken object.

Parameters: type the token type.

Returns: the Tokenizer object.

forWord

public static Tokenizer forWord()
Get the Tokenizer object that creates a word token.