LIGO Data Grid

Available Data Web Page Information

Information about how this page is generated and details about the software will appear here this week. (4 Oct 06)

The available data web page is produced nightly in a cron job. It is currently generated on draco.phys.uwm.edu but will be moved to nemo-dataserver.phys.uwm.edu shortly.

How It Works

There are 2 steps in producing the available data page. First, segment data is collected from each site from their respective LDRdataFindServer and put into an SQLite database. Then a web page is generated from the database. Both of these steps are performed by invoking the program datasiteTool.

Each data site is queried for segment information:

   for each run (S3, S4, S5):
      for each frametype associated with this site:
         for each observatory associated with this site:
            get the segments
            store them in the database

The frametypes are configured on a per site basis in a config file. They may also be queried for from the dataFind servers themselves, but this leads to many unnecessary queries. This is because currently the data collector is not smart about knowing not to ask about L1_RDS_R_L4 frames from observatory G, for instance.

Segment information is obtained directly from the LDRdataFind server at data sites which support a SEGMENT query. Currently, this query is only supported at the data sites which support DiskCache. For sites which do not support DiskCache (PSU, UWM) a more lengthy query is performed for all LFNs of a given frametype. The segments are then generated from information in the LFN names themselves.

All data sites are queried in parallel with each other, but only one dataFind request is generated per server at one time to avoid overload.

Once all the information has been collected, the database is scanned and a web page is generated. There is no fancy system to do this, it is just a hacked together hardcoded web page generator which produces the main web page, the segment list pages and the coverage images. Coverage images are produced using the Python Image Library.

Requirements

1. Python v2.2 or later.
2. PyGlobus.
3. Glue v1.7 or later.
4. SQLite. Available for download at the SQLite web site or from the Fedora extras yum repository. yum install sqlite
5. PySQLite. Avaliable for download at the PySQLite web site or from the Fedora extras yum repository. yum install python-sqlite
6. The Python Image Library (PIL). (optional) To produce coverage graphics. It is available from the PythonWare web site or from the Fedora Extras yum repository. yum install python-imaging

Downloading

You may download a tarball via your browser or with wget:

 wget http://www.lsc-group.phys.uwm.edu/lscdatagrid/downloads/DatasiteTools-0.1.0.beta.tar.gz

The software is also available through CVS.

1. Login to the CVS archive:

   $ cvs -d ':pserver:USER@gravity.phys.uwm.edu:/usr/local/cvs/ldr' login
   Logging in to :pserver:login@gravity.phys.uwm.edu:/usr/local/cvs/lscdatagridweb
   CVS password:
   

2. Checkout the module:

   $ cvs -d ':pserver:USER@gravity.phys.uwm.edu:/usr/local/cvs/ldr' co DatasiteTools
   

You may also browse the CVS archive.

Installation

Installation is a standard python installation. Choose a directory to install the software, lets call it INSTALLATION_DIRECTORY. If you omit the --prefix option, installation defaults to /usr. Then do:

tar xzvf DatasiteTools-0.1.0.beta.tar.gz
cd DatasiteTools-0.1.0.beta
python setup.py --prefix=INSTALLATION_DIRECTORY

To create the API documentation, do make apidoc. Documentation will be in ./doc/api.

Configuration

Basic information can be garnered from the config file, etc/DatasiteTool.ini and from bin/datasiteTool --help.

This section is a bit sketchy today. Please check later for more details.

Supported by the National Science Foundation. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF)

---

$Id: availabledatapage.html,v 1.5 2006/10/26 08:10:22 patrick Exp $