Package org.apache.lucene.analysis.cn.smart

Class SmartChineseAnalyzer

java.lang.Object
org.apache.lucene.analysis.Analyzer
org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer
All Implemented Interfaces:
Closeable, AutoCloseable
public final class SmartChineseAnalyzer extends org.apache.lucene.analysis.Analyzer
SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.

This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.

The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!

Since:
3.1
WARNING: This API is experimental and might change in incompatible ways in the next release.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer

    org.apache.lucene.analysis.Analyzer.ReuseStrategy, org.apache.lucene.analysis.Analyzer.TokenStreamComponents

  • Field Summary

    Fields inherited from class org.apache.lucene.analysis.Analyzer

    GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY

  • Constructor Summary

    Constructors
    Constructor
    Description
    SmartChineseAnalyzer()
    Create a new SmartChineseAnalyzer, using the default stopword list.
    SmartChineseAnalyzer(boolean useDefaultStopWords)
    Create a new SmartChineseAnalyzer, optionally using the default stopword list.
    SmartChineseAnalyzer(org.apache.lucene.analysis.CharArraySet stopWords)
    Create a new SmartChineseAnalyzer, using the provided Set of stopwords.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.lucene.analysis.Analyzer.TokenStreamComponents
    createComponents(String fieldName)
     
    static org.apache.lucene.analysis.CharArraySet
    getDefaultStopSet()
    Returns an unmodifiable instance of the default stop-words set.
    protected org.apache.lucene.analysis.TokenStream
    normalize(String fieldName, org.apache.lucene.analysis.TokenStream in)
     

    Methods inherited from class org.apache.lucene.analysis.Analyzer

    attributeFactory, close, getOffsetGap, getPositionIncrementGap, getReuseStrategy, initReader, initReaderForNormalization, normalize, tokenStream, tokenStream

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • SmartChineseAnalyzer

      public SmartChineseAnalyzer()
      Create a new SmartChineseAnalyzer, using the default stopword list.

    • SmartChineseAnalyzer

      public SmartChineseAnalyzer(boolean useDefaultStopWords)
      Create a new SmartChineseAnalyzer, optionally using the default stopword list.

      The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!

      Parameters:
      useDefaultStopWords - true to use the default stopword list.

    • SmartChineseAnalyzer

      public SmartChineseAnalyzer(org.apache.lucene.analysis.CharArraySet stopWords)
      Create a new SmartChineseAnalyzer, using the provided Set of stopwords.

      Note: the set should include punctuation, unless you want to index punctuation!

      Parameters:
      stopWords - Set of stopwords to use.

  • Method Details

    • getDefaultStopSet

      public static org.apache.lucene.analysis.CharArraySet getDefaultStopSet()
      Returns an unmodifiable instance of the default stop-words set.
      Returns:
      an unmodifiable instance of the default stop-words set.

    • createComponents

      public org.apache.lucene.analysis.Analyzer.TokenStreamComponents createComponents(String fieldName)
      Specified by:
      createComponents in class org.apache.lucene.analysis.Analyzer

    • normalize

      protected org.apache.lucene.analysis.TokenStream normalize(String fieldName, org.apache.lucene.analysis.TokenStream in)
      Overrides:
      normalize in class org.apache.lucene.analysis.Analyzer