Package org.apache.lucene.index
Class DocumentsWriter
- java.lang.Object
-
- org.apache.lucene.index.DocumentsWriter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,Accountable
final class DocumentsWriter extends java.lang.Object implements java.io.Closeable, Accountable
This class accepts multiple added documents and directly writes segment files. Each added document is passed to the indexing chain, which in turn processes the document into the different codec formats. Some formats write bytes to files immediately, e.g. stored fields and term vectors, while others are buffered by the indexing chain and written only on flush. Once we have used our allowed RAM buffer, or the number of added docs is large enough (in the case we are flushing by doc count instead of RAM usage), we create a real segment and flush it to the Directory. Threads: Multiple threads are allowed into addDocument at once. There is an initial synchronized call toDocumentsWriterFlushControl.obtainAndLock()
which allocates a DWPT for this indexing thread. The same thread will not necessarily get the same DWPT over time. Then updateDocuments is called on that DWPT without synchronization (most of the "heavy lifting" is in this call). Once a DWPT fills up enough RAM or hold enough documents in memory the DWPT is checked out for flush and all changes are written to the directory. Each DWPT corresponds to one segment being written. When flush is called by IndexWriter we check out all DWPTs that are associated with the currentDocumentsWriterDeleteQueue
out of theDocumentsWriterPerThreadPool
and write them to disk. The flush process can piggy-back on incoming indexing threads or even block them from adding documents if flushing can't keep up with new documents being added. Unless the stall control kicks in to block indexing threads flushes are happening concurrently to actual index requests. Exceptions: Because this class directly updates in-memory posting lists, and flushes stored fields and term vectors directly to files in the directory, there are certain limited times when an exception can corrupt this state. For example, a disk full while flushing stored fields leaves this file in a corrupt state. Or, an OOM exception while appending to the in-memory posting lists can corrupt that posting list. We call such exceptions "aborting exceptions". In these cases we must call abort() to discard all docs added since the last flush. All other exceptions ("non-aborting exceptions") can still partially update the index structures. These updates are consistent, but, they represent only a part of the document seen up until the exception was hit. When this happens, we immediately mark the document as deleted so that the document is always atomically ("all or none") added to the index.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description (package private) static interface
DocumentsWriter.FlushNotifications
-
Field Summary
Fields Modifier and Type Field Description private boolean
closed
private LiveIndexWriterConfig
config
private DocumentsWriterDeleteQueue
currentFullFlushDelQueue
(package private) DocumentsWriterDeleteQueue
deleteQueue
(package private) DocumentsWriterFlushControl
flushControl
private DocumentsWriter.FlushNotifications
flushNotifications
private InfoStream
infoStream
private java.util.concurrent.atomic.AtomicInteger
numDocsInRAM
private boolean
pendingChangesInCurrentFullFlush
private java.util.concurrent.atomic.AtomicLong
pendingNumDocs
(package private) DocumentsWriterPerThreadPool
perThreadPool
private DocumentsWriterFlushQueue
ticketQueue
-
Constructor Summary
Constructors Constructor Description DocumentsWriter(DocumentsWriter.FlushNotifications flushNotifications, int indexCreatedVersionMajor, java.util.concurrent.atomic.AtomicLong pendingNumDocs, boolean enableTestPoints, java.util.function.Supplier<java.lang.String> segmentNameSupplier, LiveIndexWriterConfig config, Directory directoryOrig, Directory directory, FieldInfos.FieldNumbers globalFieldNumberMap)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description (package private) void
abort()
Called if we hit an exception at a bad time (when updating the index files) and must discard all currently buffered docs.private void
abortDocumentsWriterPerThread(DocumentsWriterPerThread perThread)
Returns how many documents were aborted.(package private) boolean
anyChanges()
(package private) boolean
anyDeletions()
private boolean
applyAllDeletes()
If buffered deletes are using too much heap, resolve them and write disk and return true.private long
applyDeleteOrUpdate(java.util.function.ToLongFunction<DocumentsWriterDeleteQueue> function)
private boolean
assertTicketQueueModification(DocumentsWriterDeleteQueue deleteQueue)
void
close()
(package private) long
deleteQueries(Query... queries)
(package private) long
deleteTerms(Term... terms)
private boolean
doFlush(DocumentsWriterPerThread flushingDWPT)
private void
ensureOpen()
(package private) void
finishFullFlush(boolean success)
(package private) long
flushAllThreads()
(package private) boolean
flushOneDWPT()
(package private) int
getBufferedDeleteTermsSize()
(package private) long
getFlushingBytes()
Returns the number of bytes currently being flushed This is a subset of the value returned byramBytesUsed()
(package private) long
getMaxCompletedSequenceNumber()
returns the maximum sequence number for all previously completed operations(package private) long
getNextSequenceNumber()
(package private) int
getNumBufferedDeleteTerms()
(package private) int
getNumDocs()
Returns how many docs are currently buffered in RAM.(package private) java.io.Closeable
lockAndAbortAll()
Locks all currently active DWPT and aborts them.private boolean
postUpdate(DocumentsWriterPerThread flushingDWPT, boolean hasEvents)
private boolean
preUpdate()
(package private) void
purgeFlushTickets(boolean forced, IOUtils.IOConsumer<DocumentsWriterFlushQueue.FlushTicket> consumer)
long
ramBytesUsed()
Return the memory usage of this object in bytes.(package private) void
resetDeleteQueue(DocumentsWriterDeleteQueue newQueue)
private boolean
setFlushingDeleteQueue(DocumentsWriterDeleteQueue session)
(package private) void
subtractFlushedNumDocs(int numFlushed)
(package private) long
updateDocuments(java.lang.Iterable<? extends java.lang.Iterable<? extends IndexableField>> docs, DocumentsWriterDeleteQueue.Node<?> delNode)
(package private) long
updateDocValues(DocValuesUpdate... updates)
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
-
-
-
Field Detail
-
pendingNumDocs
private final java.util.concurrent.atomic.AtomicLong pendingNumDocs
-
flushNotifications
private final DocumentsWriter.FlushNotifications flushNotifications
-
closed
private volatile boolean closed
-
infoStream
private final InfoStream infoStream
-
config
private final LiveIndexWriterConfig config
-
numDocsInRAM
private final java.util.concurrent.atomic.AtomicInteger numDocsInRAM
-
deleteQueue
volatile DocumentsWriterDeleteQueue deleteQueue
-
ticketQueue
private final DocumentsWriterFlushQueue ticketQueue
-
pendingChangesInCurrentFullFlush
private volatile boolean pendingChangesInCurrentFullFlush
-
perThreadPool
final DocumentsWriterPerThreadPool perThreadPool
-
flushControl
final DocumentsWriterFlushControl flushControl
-
currentFullFlushDelQueue
private volatile DocumentsWriterDeleteQueue currentFullFlushDelQueue
-
-
Constructor Detail
-
DocumentsWriter
DocumentsWriter(DocumentsWriter.FlushNotifications flushNotifications, int indexCreatedVersionMajor, java.util.concurrent.atomic.AtomicLong pendingNumDocs, boolean enableTestPoints, java.util.function.Supplier<java.lang.String> segmentNameSupplier, LiveIndexWriterConfig config, Directory directoryOrig, Directory directory, FieldInfos.FieldNumbers globalFieldNumberMap)
-
-
Method Detail
-
deleteQueries
long deleteQueries(Query... queries) throws java.io.IOException
- Throws:
java.io.IOException
-
deleteTerms
long deleteTerms(Term... terms) throws java.io.IOException
- Throws:
java.io.IOException
-
updateDocValues
long updateDocValues(DocValuesUpdate... updates) throws java.io.IOException
- Throws:
java.io.IOException
-
applyDeleteOrUpdate
private long applyDeleteOrUpdate(java.util.function.ToLongFunction<DocumentsWriterDeleteQueue> function) throws java.io.IOException
- Throws:
java.io.IOException
-
applyAllDeletes
private boolean applyAllDeletes() throws java.io.IOException
If buffered deletes are using too much heap, resolve them and write disk and return true.- Throws:
java.io.IOException
-
purgeFlushTickets
void purgeFlushTickets(boolean forced, IOUtils.IOConsumer<DocumentsWriterFlushQueue.FlushTicket> consumer) throws java.io.IOException
- Throws:
java.io.IOException
-
getNumDocs
int getNumDocs()
Returns how many docs are currently buffered in RAM.
-
ensureOpen
private void ensureOpen() throws AlreadyClosedException
- Throws:
AlreadyClosedException
-
abort
void abort() throws java.io.IOException
Called if we hit an exception at a bad time (when updating the index files) and must discard all currently buffered docs. This resets our state, discarding any docs added since last flush.- Throws:
java.io.IOException
-
flushOneDWPT
final boolean flushOneDWPT() throws java.io.IOException
- Throws:
java.io.IOException
-
lockAndAbortAll
java.io.Closeable lockAndAbortAll() throws java.io.IOException
Locks all currently active DWPT and aborts them. The returned Closeable should be closed once the locks for the aborted DWPTs can be released.- Throws:
java.io.IOException
-
abortDocumentsWriterPerThread
private void abortDocumentsWriterPerThread(DocumentsWriterPerThread perThread) throws java.io.IOException
Returns how many documents were aborted.- Throws:
java.io.IOException
-
getMaxCompletedSequenceNumber
long getMaxCompletedSequenceNumber()
returns the maximum sequence number for all previously completed operations
-
anyChanges
boolean anyChanges()
-
getBufferedDeleteTermsSize
int getBufferedDeleteTermsSize()
-
getNumBufferedDeleteTerms
int getNumBufferedDeleteTerms()
-
anyDeletions
boolean anyDeletions()
-
close
public void close() throws java.io.IOException
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Throws:
java.io.IOException
-
preUpdate
private boolean preUpdate() throws java.io.IOException
- Throws:
java.io.IOException
-
postUpdate
private boolean postUpdate(DocumentsWriterPerThread flushingDWPT, boolean hasEvents) throws java.io.IOException
- Throws:
java.io.IOException
-
updateDocuments
long updateDocuments(java.lang.Iterable<? extends java.lang.Iterable<? extends IndexableField>> docs, DocumentsWriterDeleteQueue.Node<?> delNode) throws java.io.IOException
- Throws:
java.io.IOException
-
doFlush
private boolean doFlush(DocumentsWriterPerThread flushingDWPT) throws java.io.IOException
- Throws:
java.io.IOException
-
getNextSequenceNumber
long getNextSequenceNumber()
-
resetDeleteQueue
void resetDeleteQueue(DocumentsWriterDeleteQueue newQueue)
-
subtractFlushedNumDocs
void subtractFlushedNumDocs(int numFlushed)
-
setFlushingDeleteQueue
private boolean setFlushingDeleteQueue(DocumentsWriterDeleteQueue session)
-
assertTicketQueueModification
private boolean assertTicketQueueModification(DocumentsWriterDeleteQueue deleteQueue)
-
flushAllThreads
long flushAllThreads() throws java.io.IOException
- Throws:
java.io.IOException
-
finishFullFlush
void finishFullFlush(boolean success) throws java.io.IOException
- Throws:
java.io.IOException
-
ramBytesUsed
public long ramBytesUsed()
Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsed
in interfaceAccountable
-
getFlushingBytes
long getFlushingBytes()
Returns the number of bytes currently being flushed This is a subset of the value returned byramBytesUsed()
-
-