Class | Ferret::Analysis::TokenStream
In: | ext/r_analysis.c
Parent: | Object

## Summary

A TokenStream enumerates the sequence of tokens, either from fields of a document or from query text.

This is an abstract class. Concrete subclasses are:

Tokenizer: | a TokenStream whose input is a string
TokenFilter: | a TokenStream whose input is another TokenStream

### Methods

next text text=

### Public Instance methods

token_stream.next → token

Return the next token from the TokenStream or nil if there are no more tokens.

token_stream.text = text → text

Return the text that the TokenStream is tokenizing

token_stream.text = text → text

Set the text attribute of the TokenStream to the text you wish to be tokenized. For example, you may do this;

        token_stream.text = File.read(file_name)

[Validate]