SDN helps Big Data to optimize access to data
This chapter introduces the state of the art in the emerging area of combining high performance computing (HPC) with Big Data Analysis. To understand the new area, the chapter first surveys the existing approaches to integrating HPC with Big Data. Next, the chapter introduces several optimization solutions that focus on how to minimize the data transfer time from computation-intensive applications to analysis intensive applications as well as minimizing the end-to-end time-to-solution. The solutions utilize Software Defined Network (SDN) to adaptively use both high speed interconnect network and high performance parallel file systems to optimize the application performance. A computational framework called DataBroker is designed and developed to enable a tight integration of HPC with data analysis. Multiple types of experiments have been conducted to show different performance issues in both message passing and parallel file systems and to verify the effectiveness of the proposed research approaches.
SDN helps Big Data to optimize access to data, Page 1 of 2
< Previous page Next page > /docserver/preview/fulltext/books/pc/pbpc015e/PBPC015E_ch14-1.gif /docserver/preview/fulltext/books/pc/pbpc015e/PBPC015E_ch14-2.gif