Workflow Transformation for Real-Time Big Data Processing

Abstract—Abstract With the explosion of big data, processing and analyzing large numbers of continuous data streams in real-time,such as social media stream, sensor data streams, log streams,stock exchanges streams, etc., has become a crucial requirement for many scientific and industrial applications in recent years.Increased volume of streaming data as well as the demand for more complex real-time analytics require for execution of processing pipelines among heterogeneous event processing engines as a workflow. In this paper, we propose a workflow transformation for cost minimization in real-time big data processing on the heterogeneous systems. We first give the definition of stream-based workflow; and then we define eight different patterns as rules for workflow transformation; next, we give our workflow transformation algorithm based on our designed rules. Finally, our experiment shows that our proposed workflow transformation method can reduce the communication and computation cost effectively.