Search


DASWG LAL Doxygen

Docs

How-To's
Technical
Software Docs
Minutes

Download

Browse CVS, Git, or SVN
Software Repositories
OS Security Updates
LIGO software virtual machine
VMware SL6 Install

Participate

SCCB - Software Change Control Board
Edit these pages
Sub-committees
Mailing List
Telecon

Projects

DMT
DQSEGDB
Glue
GraceDB
gstlal
LALSuite
LDAS Tools
LDG Client/Server
LDR
ligoDV
LIGOtools
LVAlert Administration
LVAlert
MatApps
Metaio
NDS Client
PyLAL
LSCSOFT VM

Legacy Projects

geopp
LDAS
LDM
LIGOtools
LSCGIS
Onasys
OSG-LIGO

Overview of the segment database design and implementation

Introduction

The implementation of the LSC segment database can be divided into 4 parts

  1. The IBM DB2 database and associated table structure which store the segment metadata.
  2. The replication mechanism and associated control tables which implement peer-to-peer replication of the information in the database.
  3. The server programs that provide an interface for LSC users, DMT programs or publishing scripts to insert or retrieve segment information.
  4. The client tools that provide users the ability to insert and retrieve segment information and.

This page documents how the LSC segment database is designed, how replication between databases works, how the server programs work and how the clients communicate with the servers.

State and Data Quality Segments

There are two types of segment stored in the database, state segments and data quality segments:

  1. State segments are derived from the state channel in the frames and describe when the interferometer is in Science mode, Injection mode, or some other condition as specifed by the interferometer state vector.
  2. Data quality segments are created by either the DMT or the detector characterization team and describe when a particular stretch of data should be flagged as either good or bad.
A segment is a pair of GPS times corresponding to the half open interval
[start,end)

State Segments

A state segment is defined by its version and value. The version is an integer that corresponds to a particular meaning of the interferometer state vector and should not be changed unless the meaning of the bits in the state vector changes. The value is the numeric integer value of the state vector. For example, all bits on, or 0xffff, which is value 65535. In order to present a more human friendly interface, each version and value pair is given a segment definition consisting of the fields

State segments are always active, there are no inactive or undefined state segments.

Data Quality Segments

A data quality flag is uniquely specified by the following fields

For a given GPS time interval, a DQ flag can be DQ segments corresponding to active and inactive flags are stored in the database. If there is no information in the database for a given interval, it is assumed that the flag is undefined for that interval.

Database Engine and Table Design

As its name implies, the segment database uses a relational database engine to store and retrieve all segment data. The segment database used the IBM DB2 database to provide this service. The table structure using in the database is based on the orginal LDAS table definition, with some modifcations to accomodate the needs of the segment database.

The SQL used to create the database tables is stored in the glue CVS archive under src/conf/db2. The script create_base executes the necessary SQL scripts to create a segment database and the script createall runs the necessary SQL scripts to create a trigger database.

Database Replication

The three laboratory segment databases at CIT, LHO and LLO are connection together by low latency peer-to-peer database replication. Any segments inserted at one of the three databases will be replicated to the other two databases with a latency of a few seconds to a couple of minutes. Replication time varies depending on the load on the databases. The replication used is called Q replication and is provided by the IBM products DB2 and WebSphere MQ.

IBM WebSphere MQ is used as the transport layer for replication between the databases. Message queues are set up between each of the servers that take part in the replication and these are used by the replication programs to send and recieve data and control messages. The following links to the IBM documentation give an overview of WebSphere MQ and it's use in database replication:

IBM Q replication is used to implement peer to peer replication. Briefly, a capture and apply program run at each site, with the capture program recording all transactions with the database and sending them, via the message queues, to the other two database. Apply programs running at the remote databases recieve messages from the queue and replicate the transaction on the other databases. The following links give an overview of Q replication, as used by the segment database.

Server components used by the segment database

All the segment servers programs are written in Python and are part of Glue. The server APIs LSCsegFindServer.py and LDBDServer.py are implemented as python modules which are loaded and executed by the server program ldbdd.

LSCsegFindServer.py

Responds to user requests for segment and data quality information. It constructs the SQL needed to service the users request, executes the query on the database and returns the results to the user. Communication between the client and server is performed over a GSI authenticated socket connection. The server runs on the same machine as the DB2 database, and can queries can be issued by remote clients.

LDBDServer.py

A generic interface between the database and LIGO lightweight XML representations of table data in the database. The LDBDServer can parse the contents of a LIGO LW document containing table data and insert it into the database. It can also execute SQL queries from a client and return the results as LIGO LW XML data. Data quality information is inserted via the LDBDServer. Communication between the client and server is performed over a GSI authenticated socket connection. The server runs on the same machine as the DB2 database, and can queries can be issued by remote clients. LDBDServer.py is also capable of inserting LFN to PFN maps into an RLS server, if desired.

Client programs and Communication Protocols

There are two types of client program used by the segment database:

  1. Those that communicate directly with the relational database engine, such as the publishing scripts and the segwizard client. There must run on the machine on which the relational database engine is installed.
  2. Those that communicate over secure socket connections with servers running on the database hosts. These clients can be run from any machine with a TCP/IP route to the servers. There are programs such as LSCsegFind, LSCdqInsert, dmtdq_seg_insert and ldbdc.

glue.statedb

The glue module statedb provides an API for frame publishing scripts to publish the interferometer state segment information from the raw frames into the segment database. Programs using the statedb API must run on the same machine as the DB2 database, as communication is via direct ODBC connections to the database.

The class that that programs can use to insert state segments provided by the statedb module is StateSegmentDatabase. Documentation for the class and its the methods is available from the statedb documentation in the glue online documentation.

To use the class to publsh the segments contained in a frame, an instance of StateSegmentDatabase should be created to connect to a specific database, the LFN of the frame should be registered, and the segments within the frame published. The following pseudo code shows the procedure:

# create a connection to publish S5 segments into the LHO database
sdb = statedb.StateSegmentDatabase( 'S5', 'seg_lho' )

# register the frame file logical file name
sdb.register_lfn( 'H-RDS_R_L3-822329600-256.gwf', 822329600, 822329856 )

# publish some segments from the file
sdb.publish_state('H1', 822329600, 000000000, 822329709, 437500000, 3, 65535, None)
sdb.publish_state('H1', 822329709, 437500000, 822329821, 437500000, 3, 65532, None)

# close the database connection
sdb.close()

LSCsegFind

LSCdqInsert

dmtdq_seg_insert

segpagegen utility

$Id$