hfst-tokenize(1) =perform matching/lookup on text streams


hfst-tokenize [,--segment | --xerox | --cg/] [,OPTIONS/...] ,RULESET/


perform matching/lookup on text streams

Common options:

-h, --help
Print help message
-V, --version
Print version info
-v, --verbose
Print verbosely while processing
-q, --quiet
Only print fatal erros and requested output
-s, --silent
Alias of --quiet
-n --newline
Newline as input separator (default is blank line)
-a --print-all
Print nonmatching text
-w --print-weight
Print weights
Tokenize multicharacter symbols (by default only one utf-8 character is tokenized at a time regardless of what is present in the alphabet)
-t, --time-cutoff=,S/
Limit search after having used S seconds per input
Segmenting / tokenization mode (default)
Xerox output
cg output
FinnPos output

Use standard streams for input and output (for now).


Report bugs to <[email protected]> or directly to our bug tracker at: <https://sourceforge.net/tracker/?atid=1061990&group_id=224521&func=browse>

hfst-tokenize home page: <https://kitwiki.csc.fi/twiki/bin/view/KitWiki//HfstTokenize>
General help using HFST software: <https://kitwiki.csc.fi/twiki/bin/view/KitWiki//HfstHome>


Copyright © 2010 University of Helsinki, License GPLv3: GNU GPL version 3 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.