Search


DASWG LAL Doxygen

Docs

How-To's
Technical
Software Docs
Minutes

Download

Browse CVS, Git, or SVN
Software Repositories
OS Security Updates
LIGO software virtual machine
VMware SL6 Install

Participate

SCCB - Software Change Control Board
Edit these pages
Sub-committees
Mailing List
Telecon

Projects

DMT
DQSEGDB
Glue
GraceDB
gstlal
LALSuite
LDAS Tools
LDG Client/Server
LDR
ligoDV
LIGOtools
LVAlert Administration
LVAlert
MatApps
Metaio
NDS Client
PyLAL
LSCSOFT VM

Legacy Projects

geopp
LDAS
LDM
LIGOtools
LSCGIS
Onasys
OSG-LIGO

Description of the tools and process used to insert data quality flags

Introduction

Insertion of data quality flags can be performed in two ways, which are documented on this page

  1. Manual insertion of data quality flags from a text list of GPS start and stop times using the program LSCdqInsert
  2. Insertion of XML data created by the DMT using the program dmtdq_seg_insert

Manual Insertion of Data Quality Flags

Overview

The LSCdqInsert program is used to insert a list of data quality segments into a specified segment database. Segments can be inserted at Caltech, Hanford or Livingston. There is no need to insert the same segments at all the sites, database replication will ensure that the segments are copied from any one site to the other two. LSCdqInsert reads the segment intervals from a plain text file containing GPS start and end times, and obtains the metadata about the segments from command line arguments.

Recall that in the segment database, a data quality flag is uniquely specified by the following fields

For a given GPS time interval, defined by the half open interval
[start,end)
a DQ flag can be Segments corresponding to active and inactive flags are stored in the database. If there is no information in the database for a given interval, it is assumed that the flag is undefined for that interval.

To insert segments with LSCdqInsert, the user should create a plain text file containing the GPS start and end times of the active segments. A time range from which the information about these segments is obtained is given on the command line. The client will then insert active segments for all the intervals given in the text file and inactive segments for the difference between the interval given on the command line and the list of active segments. For example if the user specified the total time range on the command line as

--gps-start-time 800000000 --gps-end-time 800001000
and the active segments file contains the intervals
800000100 800000200
800000500 800000550
the client will automatically generate inactive segments for the intervals
800000000 800000100
800000200 800000500
800000550 800001000

In addition to the run, ifos, name and version defining the flag, the user must also specify an explanation and a comment.

Usage Syntax

The program LSCdqInsert can operate in two modes, create and append as described below. Create is used to initialize a new data quality flag and insert active and inactive segments corresponding to that flag. Append is used to add segments to an existing DQ flag.

The usage syntax for the client is:

Usage: LSCdqInsert [OPTIONS]

  -s, --server HOST:PORT   connect to ldbd on HOST:PORT (default port 30020)
  -j, --identity SUBJECT   the subject of the servers service certififcate

  -A, --append             add segments to an existing data quality flag

  -r, --run RUN            the name of the run for this flag (S4, S5, etc.)
  -i, --ifos IFOS          the interferometers to which this flag applies
  -n, --name NAME          the name of the data quality flag (e.g. DUST)
  -v, --version VERS       the numeric version of this data quality flag

  -e, --explain EXPLN      a string explaining WHAT this DQ flag means
  -c, --comment COMM       a string explaining WHY this DQ flag was inserted

  -g, --gps-start-time T1  start of interval checked for this flag
  -G, --gps-end-time T2    end of interval checked for this flag

  -S, --segment-file FILE  a file containing the gps start and stop times

  -p, --ping               ping the server
  -h, --help               print this message

Creating a new Data Quality Flag

To create a new data quality flag for an interval, the definition of the flag must not already exist in the database. If it does and you want to change the values of the segments, you must create a new data quality flag with a higher version number than the one that already exists. You cannot go back and change the segments corresponding to a DQ flag without changing the version number. If you want to add segments to a specific data quality flag, and those segments do not overlap any existing active or inactive segments, see the instructions for append below.

The following steps illustrate creating a new data quality flag:

  1. Determine the time interval for which you want to create segments, i.e. the range which you have analyzed to determine the information you are inserting. In this case, we have checked data from Jan 22 2006 13:19:46 PST to Jan 26 2006 00:39:46 PST, so the interval is [822000000,822300000)
  2. Create a text file containing the segment intervals where the flag should be active. These active segments should be entirely contained in the total interval, described above. In this case we create a file called segments.txt containing the segments
    822000100 822200200
    822000150 822100677
    822202629 822252827
    Notice that the first and second segments overlap. This does not matter, as the segments will be coalesced when inserted.
  3. Determine the run, ifos, name, version and explanation of the flag. In this example, we are inserting segments that correspond to bad calibration in H1 during S5. This is the first version of the flag, so we use version 1 and we set the explanation to Calibration line dropouts.
  4. Determine the comment, i.e. where this data came from. In this example, we choose Created by scanning excitation channel.
  5. Next, determine the server to insert the flags into. In this example, we use the development server ldas-sw-backup.ligo.caltech.edu but in practice, this will be one of the gateway machines at CIT, LHO or LLO.
  6. Finally run the client with the information above to insert the data into the database:
    LSCdqInsert --run S5 --ifos H1 --name "BAD_CALIBRATION" --version 1 
    --explain "Calibration line dropouts" --comment "Created by scanning excitation channel" 
    --segment-file segments.txt --gps-start-time 822000000 --gps-end-time 822300000 
    --server ldas-sw-backup.ligo.caltech.edu

You can use the following LSCsegFind query to retrieve the active segments that you have inserted:

LSCsegFind --server ldas-sw-backup.ligo.caltech.edu --interferometer H1 
--type BAD_CALIBRATION --gps-start-time 800000000 --gps-end-time 900000000
This will return
822000100 822200200
822202629 822252827
Notice that the first and second segments in the file segments.txt have been coalsced into one segment.

Common error messages are described below.

Appending Segments to an Existing Data Quality Flag

To append segments to an existing DQ flag, the procedure is the same as above, except that the total time interval cannot overlap with any existing segment, active or inactive. This is to prevent a user changing the history of a DQ flag. To correct or change exiting segments you must create a new DQ flag with a higher version number.

This example shows how to append segments to the BAD_CALIBRATION flag created above.

  1. Determine the interval which has been analysed. In this case, the interval is [822300000,824000000) picking up where the insert above left off.
  2. Create the segments file. In this example the file segments.txt contains the segments
    822300500 822300600
    822301700 822301977
    822332629 822452827
    822454729 822457827
  3. The run, ifo, name and version fields should be the same as those of an existing segment. The --explain option does not need to be given, as an explanation for this segment already exists in the database.
  4. Determine the comment, i.e. where this data came from. In this example, we choose Created by scanning excitation channel.
  5. Next, determine the server to insert the flags into. In this example, we use the development server ldas-sw-backup.ligo.caltech.edu but in practice, this will be one of the gateway machines at CIT, LHO or LLO.
  6. Finally run the client with the information above to append the segments:
    LSCdqInsert --run S5 --ifos H1 --name "BAD_CALIBRATION" --version 1 
    --comment "Created by scanning excitation channel" --segment-file segments.txt 
    --gps-start-time 822300000 --gps-end-time 824000000 --append 
    --server ldas-sw-backup.ligo.caltech.edu

You can use the following LSCsegFind query to retrieve the active segments that you have inserted:

LSCsegFind --server ldas-sw-backup.ligo.caltech.edu --interferometer H1 
--type BAD_CALIBRATION --gps-start-time 800000000 --gps-end-time 900000000
This will return all the segments inserted above:
822000100 822200200
822202629 822252827
822300500 822300600
822301700 822301977
822332629 822452827
822454729 822457827

Common error messages are described below.

LSCdqInsert Error Messages

The following are the error messages returned by the client when it encounters an error. This is not an exhausive list, but it should cover the most common occurences.

Insertion of Data Quality Flags Created by the DMT

Overview

The client program dmtdq_seg_insert is used to insert XML files created by the DMT trigger manager into the database. This program is designed to by run automatically by the DMT SeqInsert program.

The client program contacts LDBD server on the host and port specified either by the command line option --server or the environment variable LDBD_SERVER. A properly formatted XML file created by the DMT trigger manager should be passed with the --file option. Only one file can be given at any one time. If the segment data in the file is sucessfully inserted into the database, the client program returns exit code 0 and the XML file is deleted. If the client fails to insert the segment data into the database, the clien returns exit code 1 and the input file is not removed.

Usage Syntax

The useage syntax for the client is:

Usage: dmtdq_seg_insert [OPTIONS]

  -r, --server HOST:PORT  connect to ldbd on HOST:PORT (default port 30020)
  -f, --file              file to insert (deleted upon success)
  -p, --ping              ping the server
  -h, --help              print this message

Format of the XML file

The XML file created by the DMT should be a properly formatted LIGO_LW XML document containing the following tables

  1. process table containing
  2. segment_definer
  3. segment
  4. segment_def_map
$Id$