SYNOPSIS
# isa Plucene::Analysis::Tokenizer
my $next = $chartokenizer->next;
DESCRIPTION
This is an abstract base class for simple, character-oriented tokenizers.METHODS
token_re
This should be defined in subclasses.normalize
This will normalise the character before it is added to the token.next
my $next = $chartokenizer->next;
This will return the next token in the string, or undef at the end of the string.