LSC Data Analysis Software Working Groups

Introduction

Insertion of data quality flags can be performed in two ways, which are documented on this page

1. Manual insertion of data quality flags from a text list of GPS start and stop times using the program LSCdqInsert
2. Insertion of XML data created by the DMT using the program dmtdq_seg_insert

Manual Insertion of Data Quality Flags

Overview

The LSCdqInsert program is used to insert a list of data quality segments into a specified segment database. Segments can be inserted at Caltech, Hanford or Livingston. There is no need to insert the same segments at all the sites, database replication will ensure that the segments are copied from any one site to the other two. LSCdqInsert reads the segment intervals from a plain text file containing GPS start and end times, and obtains the metadata about the segments from command line arguments.

Recall that in the segment database, a data quality flag is uniquely specified by the following fields

• run the name of the run for this flag (S4, S5, etc.)
• ifos the interferometer(s) to which this flag applies
• name the name of the data quality flag (e.g. DUST)
• version the numeric version of this data quality flag

For a given GPS time interval, defined by the half open interval

[start,end)

a DQ flag can be

• active meaning that this interval has been checked and the DQ flag is applicable (i.e. the data is bad)
• inactive meaning that this interval has been checked but the DQ flag is not applicable (i.e. this data is known to be OK)
• undefined meaning that this interval has not been scanned for data quality.

Segments corresponding to active and inactive flags are stored in the database. If there is no information in the database for a given interval, it is assumed that the flag is undefined for that interval.

To insert segments with LSCdqInsert, the user should create a plain text file containing the GPS start and end times of the active segments. A time range from which the information about these segments is obtained is given on the command line. The client will then insert active segments for all the intervals given in the text file and inactive segments for the difference between the interval given on the command line and the list of active segments. For example if the user specified the total time range on the command line as

--gps-start-time 800000000 --gps-end-time 800001000

and the active segments file contains the intervals

800000100 800000200
800000500 800000550

the client will automatically generate inactive segments for the intervals

800000000 800000100
800000200 800000500
800000550 800001000

In addition to the run, ifos, name and version defining the flag, the user must also specify an explanation and a comment.

• The explanation is attached to the name of the data quality flag, and is the same for all segments with the same definition. It should be a human readable string explaining what this DQ flag means, for example PD_Overflow has the explanation Lsc photodiode overflows and the Dust flag might have the explanation Elevated dust levels in ISCT4.
• The comment is attached to the particular segments inserted by an invocation of LSCdqInsert and should be a human parse-able string explaining why these data quality flags where inserted, or where they came from. For example when inserting PD_Overflow version 1, the comment might be Segment generation from an OSC condition, meaning that the segments were generated online by the DMT. However, for PD_Overflow version 2, the comment might be Updated DMT flags by additional scanning of trend data.

Usage Syntax

The program LSCdqInsert can operate in two modes, create and append as described below. Create is used to initialize a new data quality flag and insert active and inactive segments corresponding to that flag. Append is used to add segments to an existing DQ flag.

The usage syntax for the client is:

Usage: LSCdqInsert [OPTIONS]

  -s, --server HOST:PORT   connect to ldbd on HOST:PORT (default port 30020)
  -j, --identity SUBJECT   the subject of the servers service certififcate

  -A, --append             add segments to an existing data quality flag

  -r, --run RUN            the name of the run for this flag (S4, S5, etc.)
  -i, --ifos IFOS          the interferometers to which this flag applies
  -n, --name NAME          the name of the data quality flag (e.g. DUST)
  -v, --version VERS       the numeric version of this data quality flag

  -e, --explain EXPLN      a string explaining WHAT this DQ flag means
  -c, --comment COMM       a string explaining WHY this DQ flag was inserted

  -g, --gps-start-time T1  start of interval checked for this flag
  -G, --gps-end-time T2    end of interval checked for this flag

  -S, --segment-file FILE  a file containing the gps start and stop times

  -p, --ping               ping the server
  -h, --help               print this message

Creating a new Data Quality Flag

To create a new data quality flag for an interval, the definition of the flag must not already exist in the database. If it does and you want to change the values of the segments, you must create a new data quality flag with a higher version number than the one that already exists. You cannot go back and change the segments corresponding to a DQ flag without changing the version number. If you want to add segments to a specific data quality flag, and those segments do not overlap any existing active or inactive segments, see the instructions for append below.

The following steps illustrate creating a new data quality flag:

1. Determine the time interval for which you want to create segments, i.e. the range which you have analyzed to determine the information you are inserting. In this case, we have checked data from Jan 22 2006 13:19:46 PST to Jan 26 2006 00:39:46 PST, so the interval is [822000000,822300000)
2. Create a text file containing the segment intervals where the flag should be active. These active segments should be entirely contained in the total interval, described above. In this case we create a file called segments.txt containing the segments

   822000100 822200200
   822000150 822100677
   822202629 822252827

   Notice that the first and second segments overlap. This does not matter, as the segments will be coalesced when inserted.
3. Determine the run, ifos, name, version and explanation of the flag. In this example, we are inserting segments that correspond to bad calibration in H1 during S5. This is the first version of the flag, so we use version 1 and we set the explanation to Calibration line dropouts.
4. Determine the comment, i.e. where this data came from. In this example, we choose Created by scanning excitation channel.
5. Next, determine the server to insert the flags into. In this example, we use the development server ldas-sw-backup.ligo.caltech.edu but in practice, this will be one of the gateway machines at CIT, LHO or LLO.
6. Finally run the client with the information above to insert the data into the database:

   LSCdqInsert --run S5 --ifos H1 --name "BAD_CALIBRATION" --version 1 
   --explain "Calibration line dropouts" --comment "Created by scanning excitation channel" 
   --segment-file segments.txt --gps-start-time 822000000 --gps-end-time 822300000 
   --server ldas-sw-backup.ligo.caltech.edu

You can use the following LSCsegFind query to retrieve the active segments that you have inserted:

LSCsegFind --server ldas-sw-backup.ligo.caltech.edu --interferometer H1 
--type BAD_CALIBRATION --gps-start-time 800000000 --gps-end-time 900000000

This will return

822000100 822200200
822202629 822252827

Notice that the first and second segments in the file segments.txt have been coalsced into one segment.

Common error messages are described below.

Appending Segments to an Existing Data Quality Flag

To append segments to an existing DQ flag, the procedure is the same as above, except that the total time interval cannot overlap with any existing segment, active or inactive. This is to prevent a user changing the history of a DQ flag. To correct or change exiting segments you must create a new DQ flag with a higher version number.

This example shows how to append segments to the BAD_CALIBRATION flag created above.

1. Determine the interval which has been analysed. In this case, the interval is [822300000,824000000) picking up where the insert above left off.
2. Create the segments file. In this example the file segments.txt contains the segments

   822300500 822300600
   822301700 822301977
   822332629 822452827
   822454729 822457827

3. The run, ifo, name and version fields should be the same as those of an existing segment. The --explain option does not need to be given, as an explanation for this segment already exists in the database.
4. Determine the comment, i.e. where this data came from. In this example, we choose Created by scanning excitation channel.
5. Next, determine the server to insert the flags into. In this example, we use the development server ldas-sw-backup.ligo.caltech.edu but in practice, this will be one of the gateway machines at CIT, LHO or LLO.
6. Finally run the client with the information above to append the segments:

   LSCdqInsert --run S5 --ifos H1 --name "BAD_CALIBRATION" --version 1 
   --comment "Created by scanning excitation channel" --segment-file segments.txt 
   --gps-start-time 822300000 --gps-end-time 824000000 --append 
   --server ldas-sw-backup.ligo.caltech.edu

You can use the following LSCsegFind query to retrieve the active segments that you have inserted:

LSCsegFind --server ldas-sw-backup.ligo.caltech.edu --interferometer H1 
--type BAD_CALIBRATION --gps-start-time 800000000 --gps-end-time 900000000

This will return all the segments inserted above:

822000100 822200200
822202629 822252827
822300500 822300600
822301700 822301977
822332629 822452827
822454729 822457827

Common error messages are described below.

LSCdqInsert Error Messages

The following are the error messages returned by the client when it encounters an error. This is not an exhausive list, but it should cover the most common occurences.

• Unable to connect to LDBDServer ldas-cit.ligo.caltech.edu:30020

  The server listening for inserts on the machine is down or otherwise inacessible. Inform the system administrator to resolve this error.

• Error inserting segments: globus_xio: System error in read: Connection reset by peer

  This cryptic error means that the user does not have access to the server, typically caused by the users DN being missing from the gridmap file. Contact the system administrator to resolve this error.

• Unable to connect to LDBDServer ldas-sw-backup.ligo.caltech.edu:30020
  Less than 5 minutes left for proxy certificate.
  Run 'grid-proxy-init' to generate a new proxy certificate

  Your grid certificate proxy has expired. Re-generate it using the command grid-proxy-init.

• Error inserting segments: Error executing insert on server 1:Error inserting metadata into database: 
  [IBM][CLI Driver][DB2/SUN] SQL0803N  One or more values in the INSERT statement, UPDATE statement, 
  or foreign key update caused by a DELETE statement are not valid because the primary key, unique 
  constraint or unique index identified by "2" constrains table "LDBD.SEGMENT_DEFINER" from having 
  duplicate rows for those columns.  SQLSTATE=23505

  The user is akbdempting to perform a new on a data quality flag which is already defined. For example the user, is trying to create the DQ flag S5:H1:DUST:1, but this already exists in the database.
  1. change the type of the segment, for example by incrementing the version number, or
  2. use the --append option to append segments to an existing data quality flag.

• Error inserting segments: trying to append to non exsting segment.
  Cannot find segment definition in database for type

  The user is trying to append to a segment definition that does not already exist in the database. Check the run, ifo, name and version specified, or omit the option --append to create a new segment definition with the specified parameters.

• Error inserting segments: overlap with existing segment times for this version

  When using the --append option, the segments being inserted overlap with segments that already exist in the database. The overlaping segments will be printed out with this error message. Check the list of active segments and the start and stop times specified on the command line, and rectify as neccessary.

Insertion of Data Quality Flags Created by the DMT

Overview

The client program dmtdq_seg_insert is used to insert XML files created by the DMT trigger manager into the database. This program is designed to by run automatically by the DMT SeqInsert program.

The client program contacts LDBD server on the host and port specified either by the command line option --server or the environment variable LDBD_SERVER. A properly formatted XML file created by the DMT trigger manager should be passed with the --file option. Only one file can be given at any one time. If the segment data in the file is sucessfully inserted into the database, the client program returns exit code 0 and the XML file is deleted. If the client fails to insert the segment data into the database, the clien returns exit code 1 and the input file is not removed.

Usage Syntax

The useage syntax for the client is:

Usage: dmtdq_seg_insert [OPTIONS]

  -r, --server HOST:PORT  connect to ldbd on HOST:PORT (default port 30020)
  -f, --file              file to insert (deleted upon success)
  -p, --ping              ping the server
  -h, --help              print this message

Format of the XML file

The XML file created by the DMT should be a properly formatted LIGO_LW XML document containing the following tables

1. process table containing
2. segment_definer
3. segment
4. segment_def_map

$Id: dq_insertion.html,v 1.10 2006/01/26 23:51:57 duncan Exp $