基于机器学习的日志异常检测综述

日志异常检测是当前数据中心智能运维管理的典型核心应用场景.随着机器学习技术的快速发展和逐步成熟,将机器学习技术应用于日志异常检测任务已经形成热点.首先,文章介绍了日志异常检测任务的一般流程,并指出了相关过程中的技术分类和典型方法.其次,论述了日志分析任务中机器学习技术应用的分类及特点,并从日志不稳定性、噪声干扰、计算存储要求、算法可移植性等方面分析了日志分析任务的技术难点.再次,对领域内相关研究成果进行了梳理总结和技术特点的比较分析.最后,文章从日志语义表征、模型在线更新、算法并行度和通用性3个方面讨论了日志异常检测今后的研究重点及思考.

Survey on Log Anomaly Detection Based on Machine Learning

Log anomaly detection is a typical core application scenario of artificial intelligence for IT operations (AIOPS) in the current data center. With the rapid development and gradual maturity of machine learning technology, the application of machine learning to log anomaly detection has become a hot spot. Firstly, this study introduces the general procedure of log anomaly detection and points out the technical classifications and typical methods in the related process. Secondly, the classifications and characteristics of the application of machine learning technology in log analysis tasks are discussed, and we probe into the technical difficulties of log analysis tasks in terms of log instability, noise interference, computation & storage requirements, and algorithm portability. Thirdly, the related research productions in the field are summarized and their technical characteristics are compared and analyzed. Finally, the study discusses the future research focus and thinking of log anomaly detection from three aspects: log semantic representation, online model update, algorithm parallelism and versatility.