Class | Ferret::Index::IndexWriter
In: | ext/r_index.c
Parent: | Object

## Summary

The IndexWriter is the class used to add documents to an index. You can also delete documents from the index using this class. The indexing process is highly customizable and the IndexWriter has the following parameters;

dir: | This is an Ferret::Store::Directory object. You should either pass a +:dir+ or a +:path+ when creating an index.
path: | A string representing the path to the index directory. If you are creating the index for the first time the directory will be created if it‘s missing. You should not choose a directory which contains other files as they could be over-written. To protect against this set +:create_if_missing+ to false.
create_if_missing: | Default: true. Create the index if no index is found in the specified directory. Otherwise, use the existing index.
create: | Default: false. Creates the index, even if one already exists. That means any existing index will be deleted. It is probably better to use the create_if_missing option so that the index is only created the first time when it doesn‘t exist.
field_infos: | Default FieldInfos.new. The FieldInfos object to use when creating a new index if +:create_if_missing+ or +:create+ is set to true. If an existing index is opened then this parameter is ignored.
analyzer: | Default: Ferret::Analysis::StandardAnalyzer. Sets the default analyzer for the index. This is used by both the IndexWriter and the QueryParser to tokenize the input. The default is the StandardAnalyzer.
chunk_size: | Default: 0x100000 or 1Mb. Memory performance tuning parameter. Sets the default size of chunks of memory malloced for use during indexing. You can usually leave this parameter as is.
max_buffer_memory: | Default: 0x1000000 or 16Mb. Memory performance tuning parameter. Sets the amount of memory to be used by the indexing process. Set to a larger value to increase indexing speed. Note that this only includes memory used by the indexing process, not the rest of your ruby application.
term_index_interval: | Default: 128. The skip interval between terms in the term dictionary. A smaller value will possibly increase search performance while also increasing memory usage and impacting negatively impacting indexing performance.
doc_skip_interval: | Default: 16. The skip interval for document numbers in the index. As with +:term_index_interval+ you have a trade-off. A smaller number may increase search performance while also increasing memory usage and impacting negatively impacting indexing performance.
merge_factor: | Default: 10. This must never be less than 2. Specifies the number of segments of a certain size that must exist before they are merged. A larger value will improve indexing performance while slowing search performance.

max_buffered_docs:

Default: 10000. The maximum number of documents that may be stored in memory before being written to the index. If you have a lot of memory and are indexing a large number of small documents (like products in a product database for example) you may want to set this to a much higher number (like Ferret::FIX_INT_MAX). If you are worried about your application crashing during the middle of index you might set this to a smaller number so that the index is committed more often. This is like having an auto-save in a word processor application.

max_merge_docs: | Set this value to limit the number of documents that go into a single segment. Use this to avoid extremely long merge times during indexing which can make your application seem unresponsive. This is only necessary for very large indexes (millions of documents).

max_field_length:

Default: 10000. The maximum number of terms added to a single field. This can be useful to protect the indexer when indexing documents from the web for example. Usually the most important terms will occur early on in a document so you can often safely ignore the terms in a field after a certain number of them. If you wanted to speed up indexing and same space in your index you may only want to index the first 1000 terms in a field. On the other hand, if you want to be more thorough and you are indexing documents from your file-system you may set this parameter to Ferret::FIX_INT_MAX.

use_compound_file: | Default: true. Uses a compound file to store the index. This prevents an error being raised for having too many files open at the same time. The default is true but performance is better if this is set to false.

### Deleting Documents

Both IndexReader and IndexWriter allow you to delete documents. You should use the IndexReader to delete documents by document id and IndexWriter to delete documents by term which we‘ll explain now. It is preferrable to delete documents from an index using IndexWriter for performance reasons. To delete documents using the IndexWriter you should give each document in the index a unique ID. If you are indexing documents from the file-system this unique ID will be the full file path. If indexing documents from the database you should use the primary key as the ID field. You can then use the delete method to delete a file referenced by the ID. For example;

      index_writer.delete(:id, "/path/to/indexed/file")

### Methods

<< add_document add_readers analyzer analyzer= chunk_size chunk_size= close commit delete doc_count doc_skip_interval doc_skip_interval= field_infos max_buffer_memory max_buffer_memory= max_buffered_docs max_buffered_docs= max_field_length max_field_length= max_merge_docs max_merge_docs= merge_factor merge_factor= new optimize term_index_interval term_index_interval= use_compound_file use_compound_file= version

### Constants

WRITE_LOCK_TIMEOUT | = | INT2FIX(1)
COMMIT_LOCK_TIMEOUT | = | INT2FIX(10)
WRITE_LOCK_NAME | = | rb_str_new2(WRITE_LOCK_NAME)
COMMIT_LOCK_NAME | = | rb_str_new2(COMMIT_LOCK_NAME)
DEFAULT_CHUNK_SIZE | = | INT2FIX(default_config.chunk_size)
DEFAULT_MAX_BUFFER_MEMORY | = | INT2FIX(default_config.max_buffer_memory)
DEFAULT_TERM_INDEX_INTERVAL | = | INT2FIX(default_config.index_interval)
DEFAULT_DOC_SKIP_INTERVAL | = | INT2FIX(default_config.skip_interval)
DEFAULT_MERGE_FACTOR | = | INT2FIX(default_config.merge_factor)
DEFAULT_MAX_BUFFERED_DOCS | = | INT2FIX(default_config.max_buffered_docs)
DEFAULT_MAX_MERGE_DOCS | = | INT2FIX(default_config.max_merge_docs)
DEFAULT_MAX_FIELD_LENGTH | = | INT2FIX(default_config.max_field_length)
DEFAULT_USE_COMPOUND_FILE | = | default_config.use_compound_file ? Qtrue : Qfalse

### Public Class methods

IndexWriter.new(options = {}) → index_writer

     Create a new IndexWriter. You should either pass a path or a directory to
     this constructor. For example, here are three ways you can create an
     IndexWriter;

       dir = RAMDirectory.new()
       iw = IndexWriter.new(:dir => dir)

       dir = FSDirectory.new("/path/to/index")
       iw = IndexWriter.new(:dir => dir)

       iw = IndexWriter.new(:path => "/path/to/index")

See IndexWriter for more options.

### Public Instance methods

iw << document → iw
iw.add_document(document) → iw

Add a document to the index. See Document. A document can also be a simple hash object.

iw << document → iw
iw.add_document(document) → iw

Add a document to the index. See Document. A document can also be a simple hash object.

iw.add_readers(reader_array) → iw

Use this method to merge other indexes into the one being written by IndexWriter. This is useful for parallel indexing. You can have several indexing processes running in parallel, possibly even on different machines. Then you can finish by merging all of the indexes into a single index.

index_writer.analyzer → Analyzer

Get the Analyzer for this IndexWriter. This is useful if you need to use the same analyzer in a QueryParser.

index_writer.analyzer → Analyzer

Set the Analyzer for this IndexWriter. This is useful if you need to change the analyzer for a special document. It is risky though as the same analyzer will be used for all documents during search.

iw.chunk_size → number

Return the current value of chunk_size

iw.chunk_size = chunk_size → chunk_size

Set the chunk_size parameter

index_writer.close → nil

Close the IndexWriter. This will close and free all resources used exclusively by the index writer. The garbage collector will do this automatically if not called explicitly.

iw.commit → iw

Explicitly commit any changes to the index that may be hanging around in memory. You should call this method if you want to read the latest index with an IndexWriter.

iw.delete(field, term) → iw
iw.delete(field, terms) → iw

Delete all documents in the index with the given `term` or `terms` in the field `field`. You should usually have a unique document id which you use with this method, rather then deleting all documents with the word "the" in them. There are of course exceptions to this rule. For example, you may want to delete all documents with the term "viagra" when deleting spam.

iw.doc_count → number

Returns the number of documents in the Index. Note that deletions won‘t be taken into account until the IndexWriter has been committed.

iw.doc_skip_interval → number

Return the current value of doc_skip_interval

iw.doc_skip_interval = doc_skip_interval → doc_skip_interval

Set the doc_skip_interval parameter

index_writer.field_infos → FieldInfos

Get the FieldInfos object for this IndexWriter. This is useful if you need to dynamically add new fields to the index with specific properties.

iw.max_buffer_memory → number

Return the current value of max_buffer_memory

iw.max_buffer_memory = max_buffer_memory → max_buffer_memory

Set the max_buffer_memory parameter

iw.max_buffered_docs → number

Return the current value of max_buffered_docs

iw.max_buffered_docs = max_buffered_docs → max_buffered_docs

Set the max_buffered_docs parameter

iw.max_field_length → number

Return the current value of max_field_length

iw.max_field_length = max_field_length → max_field_length

Set the max_field_length parameter

iw.max_merge_docs → number

Return the current value of max_merge_docs

iw.max_merge_docs = max_merge_docs → max_merge_docs

Set the max_merge_docs parameter

iw.merge_factor → number

Return the current value of merge_factor

iw.merge_factor = merge_factor → merge_factor

Set the merge_factor parameter

iw.optimize → iw

Optimize the index for searching. This commits any unwritten data to the index and optimizes the index into a single segment to improve search performance. This is an expensive operation and should not be called too often. The best time to call this is at the end of a long batch indexing process. Note that calling the optimize method do not in any way effect indexing speed (except for the time taken to complete the optimization process).

iw.term_index_interval → number

Return the current value of term_index_interval

iw.term_index_interval = term_index_interval → term_index_interval

Set the term_index_interval parameter

iw.use_compound_file → number

Return the current value of use_compound_file

iw.use_compound_file = use_compound_file → use_compound_file

Set the use_compound_file parameter

index_writer.version → int

Returns the current version of the index writer.

[Validate]