文本分类TF-IDF算法的改进研究

中国互联网环境的发展，让大量蕴含丰富信息的新词得以普及。而传统的特征词权重TF-IDF（Term Frequency and Inverted Document Frequency）算法主要考虑TF和IDF两个方面的因素，未考虑到新词这一新兴词类的优势。针对特征项中的新词对分类结果的影响，提出基于网络新词改进文本分类TF-IDF算法。在文本预处理中识别新词，并在向量空间模型表示中改变特征权重计算公式。实验结果表明把新词发现加入文本预处理，可以达到特征降维的目的，并且改进后的特征权重算法能优化文本分类的结果。

Improved Approach to TF-IDF Algorithm in Text Classification

With the development of Internet environment in China, a lot of new words with rich information have been popularized. The traditional term weight algorithm named TF-IDF（Term Frequency and Inverted Document Frequency） mainly considers two factors named TF and IDF without the advantage of new words. In view of the influence of new words in feature items on classification results, an improved TF-IDF algorithm based on new words of network is proposed in text classification. Research recognizes new words in the text preprocessing, and improves the weight calculation formula of them in the vector space model representation. Experimental results show that adding new word discovery process to text preprocessing can reduce feature dimension, meanwhile, the improved TF-IDF algorithm can optimize the result of text classification.