Title: IWFMS: An Internal Workflow Management System/Optimizer for Hadoop

Year of Publication: Dec - 2014
Page Numbers: 59-68
Authors: Lian Liu, Yao Shen
Conference Name: The International Conference on Computer Science, Computer Engineering, and Social Media (CSCESM2014)
- Greece

Abstract:


The scale of jobs running on parallel computation platform such as Hadoop is increasing quickly, thus workflow engines that manage data processing jobs have become increasingly important. However, traditional workflow management systems are mostly outsiders to Hadoop and cannot fulfill many important requirements, such as user constraints and scheduling optimizations. In this paper, we present IWFMS, an internal workflow scheduling system/optimizer that 1) Manage the resource allocation and execution of jobs in workflows to achieve higher efficiency, 2) Schedule workflows to meet a much richer set of user constraints such as deadlines, priorities and workflow trigger events. We discuss the architecture of its key components and evaluate its features and performance.