Index Merging

MERGEPOLICY

IndexWriter relies on a subclass of the abstract MergePolicy base class to decide
when a merge should be done. Whenever new segments are flushed, or a previously
selected merge has completed, the MergePolicy is consulted to determine if a merge
is now necessary, and if so, precisely which segments will be merged. Besides picking
“normal” segment merges to do, the MergePolicy also selects merges necessary to
optimize the index and to run expungeDeletes.
Lucene provides two core merge policies, both subclassing from LogMergePolicy.
The first, which is the default used by IndexWriter, is LogByteSizeMergePolicy. This
policy measures the size of a segment as the total size in bytes of all files for that segment.
The second one, LogDocMergePolicy, makes the same merging decisions
except it measures size of a segment by the document count of the segment. Note that
neither merge policy takes deletions into account. If you have mixed document sizes,
it’s best to use LogByteSizeMergePolicy because it’s a more accurate measure of segment
size.

from lucene in action