final class SuffixingNGramTokenFilter extends TokenFilter
AttributeSource.State
Modifier and Type | Field and Description |
---|---|
private java.lang.String |
anyToken |
private int |
curCodePointCount |
private int |
curGramSize |
private int |
curPos |
private int |
curPosInc |
private int |
curPosLen |
private char[] |
curTermBuffer |
private int |
curTermLength |
private KeywordAttribute |
keywordAtt |
private int |
maxTokenLength |
private OffsetAttribute |
offsetAtt |
private PositionIncrementAttribute |
posIncAtt |
private PositionLengthAttribute |
posLenAtt |
private CharArraySet |
seenInfixes |
private CharArraySet |
seenSuffixes |
private java.lang.String |
suffix |
private CharTermAttribute |
termAtt |
private int |
tokEnd |
private int |
tokStart |
input
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
SuffixingNGramTokenFilter(TokenStream input,
java.lang.String suffix,
java.lang.String wildcardToken,
int maxTokenLength)
Creates SuffixingNGramTokenFilter.
|
Modifier and Type | Method and Description |
---|---|
boolean |
incrementToken()
Returns the next token in the stream, or null at EOS.
|
void |
reset()
This method is called by a consumer before it begins consumption using
TokenStream.incrementToken() . |
close, end
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
private final java.lang.String suffix
private final int maxTokenLength
private final java.lang.String anyToken
private char[] curTermBuffer
private int curTermLength
private int curCodePointCount
private int curGramSize
private int curPos
private int curPosInc
private int curPosLen
private int tokStart
private int tokEnd
private final CharTermAttribute termAtt
private final PositionIncrementAttribute posIncAtt
private final PositionLengthAttribute posLenAtt
private final OffsetAttribute offsetAtt
private final KeywordAttribute keywordAtt
private final CharArraySet seenSuffixes
private final CharArraySet seenInfixes
public SuffixingNGramTokenFilter(TokenStream input, java.lang.String suffix, java.lang.String wildcardToken, int maxTokenLength)
input
- TokenStream
holding the input to be tokenizedsuffix
- a string to suffix to all ngramswildcardToken
- a token to emit if the input token is longer than maxTokenLengthmaxTokenLength
- tokens longer than this will not be ngrammedpublic final boolean incrementToken() throws java.io.IOException
incrementToken
in class TokenStream
java.io.IOException
public void reset() throws java.io.IOException
TokenFilter
TokenStream.incrementToken()
.
Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call super.reset()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on further usage).
NOTE:
The default implementation chains the call to the input TokenStream, so
be sure to call super.reset()
when overriding this method.
reset
in class TokenFilter
java.io.IOException