Open Science Grid - LIGO
Open Science Grid (OSG) activities at LIGO (OSG-LIGO)
Improve the robustness of the Pegasus workflow planner on the
OSG.
HIPE ported to the OSG (status report)
Burst Q-pipeline
Test and integration of the Pegasus planning tools as supported
tools on the OSG.
On-going work with the Pegasus development team on extending
the planner for LIGO requirements.
- Developing data management techniques to constrain disk requirements to < 100 GB.
- Most OSG production sites have no more than 100 GB of space available.
- Tactic: partition Terabyte of S5 gwfs and GSIFTP only active partition needed.
- Tactic: remove the partition's gwfs when they are no longer required.
- Given the dozens of hours required to move Terabytes to the OSG
sites support the L4 RDS.
- Almost 10X reduction in data transfer accomplished.
- Binary Inspiral and HIPE codes modified to support the L4 RDS.
- Almost 10X reduction in data transfer accomplished.
- Increase access to the OSG for LSC by developing an application execution framework.
- Developed infrastructure component that aggregates OSG site metadata
- VORS DB enhanced to collect location of $OSG_GRID and $OSG_DATA for sites.
- pegasus-get-sites utility upgraded with the ability to query the
VORS DB.
- Executed HIPE on 25 OSG sites.
- Created a Pegasus DAX that moves the MatLab R13 libs to an OSG site.
- Tested the Pegasus R13 DAX with a simple Burst Q-pipeline
script on LIGO-CIT-ITB site.
- Validation of Binary Inspiral application on several recent releases of the OSG middleware.
- Developing HIPE to use as a validation tool for future releases of OSG middleware.
- On-going work with the Virtual Data Toolkit (VDT) team to integrate the Pegasus planning tools in the VDT.
- On-going work with the OSG integration team and the deployment team to integrate the Pegasus planning tools in the OSG release process.
- Limit use of shared file system on OSG sites not using NFS or a Global File System.
- modify Pegasus planner to support use of $OSG_WN_TMP (local scratch directory).
- Automatically remove gwfs when data is no longer needed on a
per-DAG basis.
- Reduce the total number of GRAM job submissions by bundling jobs.
- Long-term effort with Pegasus development team to extend the planner for use with Storage Resource Managers (SRM).
- SRM version 1 (deployed at several OSG sites).
- SRM version 2 (under development).
- Prototype work with SRM version 1 at osg-gw-2.t2.ucsd.edu
- 25 LIGO VO slots to be provided.
- data movement scripts for srm to be provided by ucsd team.
- Milestones for Success:
- 4 month effort: HIPE workfow with 25 VO slots continuously for one week at UCSD.
- 8 month effort: HIPE workflow with 100 VO slots continuously for one week at UCSD.
- 12 month effort: HIPE workflow with 1000 VO slots peak
(measured in MonALISA) on OSG.
$Id: OSG-LIGO.html,v 1.3 2008/05/07 15:07:21 bdaudert Exp $