KinoSearch::Analysis::TokenizerKinoSearch::Analysis::Tokenizer is a customizable tokenizing. | |
Download |
KinoSearch::Analysis::Tokenizer Ranking & Summary
Advertisement
- License:
- Perl Artistic License
- Price:
- FREE
- Publisher Name:
- Marvin Humphrey
- Publisher web site:
- http://search.cpan.org/~creamyg/KinoSearch-0.162/lib/KinoSearch/Search/SearchServer.pm
KinoSearch::Analysis::Tokenizer Tags
KinoSearch::Analysis::Tokenizer Description
KinoSearch::Analysis::Tokenizer is a customizable tokenizing. KinoSearch::Analysis::Tokenizer is a customizable tokenizing.SYNOPSIS my $whitespace_tokenizer = KinoSearch::Analysis::Tokenizer->new( token_re => qr/S+/, ); # or... my $word_char_tokenizer = KinoSearch::Analysis::Tokenizer->new( token_re => qr/w+/, ); # or... my $apostrophising_tokenizer = KinoSearch::Analysis::Tokenizer->new; # then... once you have a tokenizer, put it into a PolyAnalyzer my $polyanalyzer = KinoSearch::Analysis::PolyAnalyzer->new( analyzers => , );Generically, "tokenizing" is a process of breaking up a string into an array of "tokens". # before: my $string = "three blind mice"; # after: @tokens = qw( three blind mice );KinoSearch::Analysis::Tokenizer decides where it should break up the text based on the value of token_re. # before: my $string = "Eats, Shoots and Leaves."; # tokenized by $whitespace_tokenizer @tokens = qw( Eats, Shoots and Leaves. ); # tokenized by $word_char_tokenizer @tokens = qw( Eats Shoots and Leaves );Requirements:· Perl Requirements: · Perl
KinoSearch::Analysis::Tokenizer Related Software