SCS Lab has transitioned into Gnosis Research Center (GRC). This website is archived. Please visit the new website at https://grc.iit.edu.
free web creation software

IRIS:

I/O Redirection Via Integrated Storage
(CCF-1744317,
CNS-1526887,
CNS-0751200)

Background

Parallel File Systems Charasteristics:

Advantages

1) Client can access files concurrently
2) scalability, and capability
3) Distribute large files across multiple nodes
4) Hierarchical global name space
5) High bandwidth via parallel data transfer.


Disadvantages

1) Bad performance in small accesses, unaligned requests, and heavy metadata operations.
2) Performance depends on files, directories, and tree structures.
3) Maintaining data consistency generate overheads, and can create issues such as fragmentation, journaling, and simultaneous operations on the same file system structures.
4) Storage subsystem may pose limitations because of RAID, disk sizes, and other limiting factors by either hardware or software.

Object Store Charasteristics:

Advantages

1) Encapsulate data, metadata, a globally unique identifier, and data attributes into a single immutable entity termed object.
2) Scalability, flexibility, rapid data retrieval, and distributed access.
3) Easily expandable and well suited for applications requesting non contiguous data accesses and/or heavy metadata operations.
4) Offer consistent data access throughput and extensible metadata.
5) NoSQL schemes demonstrates a huge variety of different implementations each with its own strengths and weaknesses.

Disadvantages


1) They are ill suited for access patterns with frequently changing data.
2) Slow in access involving complex operations since each update operation leads to the creation of a new object and the destruction of the previous one followed by an update to the metadata.
3) Object Stores are not POSIX-compliant. this blocks their wide adoption by the HPC community. 

IRIS Project Synopsis

Mobirise

There is an ocean of available storage solutions in modern high-performance and distributed systems. These solutions consist of Parallel File Systems (PFS) for the more traditional high-performance computing (HPC) systems and of Object Stores for emerging cloud environments. More often than not, these storage solutions are tied to specific APIs and data models and thus, bind developers, applications, and entire computing facilities to using certain interfaces. Each storage system is designed and optimized for certain applications but does not perform well for others. Furthermore, modern applications have become more and more complex consisting of a collection of phases with different computation and I/O requirements. In this paper, we propose a unified storage access system, called IRIS (i.e., I/O Redirection via Integrated Storage). IRIS enables unified data access and seamlessly bridges the semantic gap between file systems and object stores. With IRIS, emerging High-Performance Data Analytics software has capable and diverse I/O support. IRIS can bring us closer to the convergence of HPC and Cloud environments by combining the best storage subsystems from both worlds. Experimental results show that IRIS can grant more than 7x improvement in performance than existing solutions.

IRIS Design & Architecture

IRIS Objectives

01.

Enable MPI-based applications to access and store data in an Object Store
 

02.

Enable HPDA-based applications to access and store data in a PFS

03.

Enable a hybrid storage access layer agnostic to files or objects.

IRIS Features

01.

Middleware library

02.

Seamless integration to applications (link to IRIS)

03.

Non-blocking I/O

04.

Tunable data consistency

Mobirise

05.

Relaxed metadata ops

06.

Caching within IRIS

07.

Prefetching for faster read

08.

Currently supports:
POSIX and MPI-IO ,
HDF5 and pNetCDF ,
S3 and Openstack Swift ,
MongoDB and Hyperdex

Evaluation Results

Real Workload Evaluation

Mobirise

CM1 SIMULATION

7x improvement

Mobirise

MONTAGE

6x improvement

Mobirise

WRF SIMULATION

7x improvement

IRIS in hybrid mode

Mobirise

LAMMPS

40-60% improvement

Mobirise

LANL ANONYMOUS APP

30-50% improvement

Publications

Our Sponsor

This work was supported by the National Science Foundation under grants no.  CCF-1744317,  CNS-1526887,  and CNS-0751200.

Mobirise