VMware, Inc.
Memory use in a distributed index and query system

Abstract:

In a method of memory use in a distributed index and query system, a processing thread serializes a list of documents into a first memory object exclusively borrowed from a memory object pool of a memory to achieve a first segment comprising a serialized data array of the documents, an offset index, and a document count. The serialized data array is compressed by the processing thread into a second memory object exclusively borrowed from the memory object pool to achieve a second segment comprising a compressed serialized data array, the offset index, and the document count. Subsequent to the compression, the first memory object is release back to the memory object pool. The second segment is written to a data storage device, and subsequent to the writing, the second memory object is released back to the memory object pool.