# TODO

Path: | TODO
Last Update: | Sun Apr 29 17:47:50 +0000 2012

TODO

####

- C
  - IMPORTANT: + FIX file descriptor overflow. See Tickets 341 and 343
  - add .. operator to query parser. For example, [100 200] could be written as 100..200 or 100...201 like in Ruby Ranges
  - remove exception handling from C code. All errors to be handled by return values.
  - Move to sqlite‘s locking model. Ferret should work fine in a multi-process environment.
  - Add optional logging. To be enabled at compilation time, perhaps?
  - Add support for changing zlib and bzlib compression parameters
  - Improve unit test coverage to 100%
  - Add benchmark suite
  - Add Rakefile for development purposes + task to publish gcov and benchmark results to ferret wiki
  - Index rebuilding of old versioned indexes.
  - Add a globally accessable, threadsafe symbol table. This will be very useful for storing field names so that no objects need to strdup the field-names but can just store the symbol representative instead. + this has been done but it can be improved using actual Symbol structs

          instead of plain char*

  - Make threading optional at compile time
  - to_json should limit output to prevent memory overflow on large indexes. Perhaps we could use some type of buffered read for this.
  - Make BitVector run as fast as bitset from C++ STL. See;

          c/benchmark/bm_bitvector.c

  - Add a symbol table for field names. This will mean that we won‘t need to worry about mallocing and freeing field names which happens all over the place.
  - Divide the headers into public and private (the private headers to be stored in the src directory).
  - Group-by search. ie you should be able to pass a field to group search results by
  - Auto-loading of documents during search. ie actual documents get returned instead of document numbers.
- Ruby bindings
  - argument checking for every method. We need a new api for argument checking so that the arguments get checked at the start of each method that could cause a segfault.
  - improve memory management. It was way to complex at the moment. I also need to document how it works so that other developers understand what is going on.
  - Replace Data_Wrap_Struct with ferret alternative which handles rewrapping of structs automatically and also knows when to release a struct by using refcounting.
- Ruby
  - integrate rcov
  - improve unit test coverage to 100%
- Documentation.
  - generate Ruby binding documentation with custom build template similar jaxdoc rubyforge.org/projects/jaxdoc
  - all documentation should meet DOCUMENTATION_STANDARDS
  - documentation in C code to be generated by doxygen

Someday Maybe

###### =======

- apply for Google Summer of Code 2009
- optimize read and write vint
  - test the following outside of ferret before implementing
  - perform a binary scan using bit-wise or to find out how many bytes need to be written
  - if the write/read will overflow the buffer, split it into two, refreshing the buffer in between
  - use Duff‘s device to write bytes now that we know how many we need
- add a super fast language based dictionary compression
- add portable stacktrace function. Perhaps implement as an external library.
  - See www.nongnu.org/libunwind/
  - See www.tlug.org.za/wiki/index.php/Obtaining_a_stack_trace_in_C_upon_SIGSEGV
- investigate unscored searching
- user defined sorting
- Fix highlighting to work for external fields
- investigate faster string hashing method

Done

####

- add rake install task
- FIX :create parameter so that it only deletes the files owned by Ferret.
- fix compression. Currently nothing is happening if you set a field to :compress. I guess we‘ll just assume zlib is installed, as I think it has to be for Ruby to be installed.
- add bzlib support
- integrate gcov
- add a field cache to IndexReader
- setup email alerts for svn commits
- Ranged, unordered searching. Ie search through the index until you have the required number of documents and then break. This will require the ability to start searches from a particular doc-num. + See searcher_search_unordered in the C code and Searcher#scan in Ruby
- improve unit test code. I‘d like to implement some way to print out a stack trace when a test fails so that it is easy to find the source of the error.
- catch segfaults and print stack trace so users can post helpful bug tickets. again, see the same links for adding stacktrace to unit tests.
- Add string Sort descripter
- fix memory bug
- add MultiReader interface
- add lexicographical sort (byte sort)
- Add highlighting
- add field compression
- Fix highlighting to work for compressed fields
- Add Ferret::Index::Index
- Fix: + Working Query: field1:value1 AND NOT field2:value2 + Failing Query: field1:value1 AND ( NOT field2:value2 )
- update benchmark suite to use getrusage

[Validate]