index optimization and add fsdirectory

index optimization

Remember that index optimization consumes substantial CPU and input/output (I/O) resources, so use it judiciously. It is a trade-off of a large onetime cost for faster search- ing. If you update your index only rarely, and do lots of searching between updates, this trade-off is worthwhile. If a single computer is doing both indexing and searching, consider scheduling optimize after hours or over the weekend so that it doesn’t inter- fere with ongoing searching.

Another important cost to be aware of is that optimizing requires substantial tem- porary disk space. Because Lucene must merge segments together, while the merge is running temporary disk space is used to hold the files for the new segment. But the old segments can’t be removed until the merge is complete and the changes are com- mitted, either by calling IndexWriter.commit or by closing the IndexWriter. This means you should expect the size of your index to roughly triple (temporarily) during optimization. Once optimization completes, and once you call commit(), disk usage will fall back to a lower level than the starting size. Any open readers on the index will also potentially impact the transient disk usage. Section 11.3.1 describes overall disk usage of Lucene in more detail.

FSDirectory implementation

SimpleFSDirectory

A simplistic Directory that stores files in the file system, using java.io.* APIs. It doesn’t scale well with many threads.

NIOFSDirectory

A Directory that stores files in the file system, using java.nio.* APIs. This does scale well with threads on all platforms except Microsoft Windows, due to a longstanding issue with Sun’s Java Runtime Environ- ment (JRE).

MMapDirectory

A Directory that uses memory-mapped I/O to access files. This is a good choice on 64-bit JREs, or on 32-bit JREs where the size of the index is relatively small.

RAMDirectory

A Directory that stores all files in RAM.

FileSwitchDirectory

A Directory that takes two directories in, and switches between these

directories based on file extension.

from. lucene in action