基于Spark Streaming的实时交通数据处理平台

交通大数据是解决城市交通问题的最基本条件，是制定宏观城市交通发展战略规划和进行微观道路交通管理与控制的重要保障.针对于智能交通系统中数据产生快、实时性强、数据量大的特点，本文基于Spark Streaming和Apache Kafka的组合构建了一个实时交通数据处理平台，用于处理通过双基基站采集的数据，采用时间窗口机制从持续的Kafka分布式消息队列中获取数据，并按照规则将数据分类处理后保存到数据库.本文对平台的系统架构和内部结构进行了详细的介绍，并通过实验验证了系统的实时处理能力，完全可以在大规模高并发的数据流下进行应用.

Real-Time Traffic Data Processing System Based on Spark Streaming

Traffic big data is the most basic condition for solving urban traffic problems. It is an important guarantee for formulating macro-city traffic development strategy and construction planning. And it is also an important guarantee for carrying out micro-road traffic management and control. In view of the characteristics of intelligent transportation system such as fast data generation, high real-time performance and large amount of data, this paper constructs a real-time traffic data processing platform based on the combination of Spark Streaming and Apache Kafka to process the data collected by dual base stations. Using time window mechanism to get data from Kafka, and process the data to the database according to the rules. In this study, the system architecture and internal structure of the platform are introduced in detail, and the real-time processing capability of the system is verified through experiments, which can be applied under large-scale and high-concurrency data flow.