LDR Meeting at Caltech, January 18, 2007

Attendees:

  • Kevin Flasch
  • Brian Moe
  • Patrick Brady
  • Warren Anderson
  • Stuart Anderson
  • Ben Johnson
  • Dan Kozak
  • Duncan Brown
  • Murail Ramsunder
  • Sam Finn
  • Tom Nash

Rather rough notes from the meeting:

  • The old planning document from the April 23, 2004 meeting was updated.
  • Ben's talk (concerning LSCdataFind, publishing...)
    • People generally ask "I want channel X between [A,B)". We have a file of type Y between [A,B). LDR answers what files of frame type Y in [A,B).
      Can we extend dataFind to answer channel requests? (frame types contain channel X in THESE segments)
    • On the use of the diskCacheServer/API...
      Race condition... publishing script may put LFNs into LDR & SegDB before LDRdataFindDiskCache server sees it. (diskCacheAPI hasn't been parsed since file was published into LDR/SegDBs)
      Why do this then? What are the advantages of the diskCacheServer?
      • RLS command line & API is poor. Ben can crash RLS at will.
      • LHO/LLO are sources. Everything is always under /archive and file names have meaning
      • diskCacheAPI utility is a solved problem
      • Solves request that is a sub-segment of a single file
      • Will return results for long (more than 10 min) queries
      • Performance: quite a bit faster than standard dataFindServer
    • Solutions to "channel X for times [A,B)
      • MatApps (LSCdataFind under hood)
      • LAL cache, LSCdataFind
      • text files, find/ls/LSCdataFind
      • Network Data Server (NDS) (framebuilder)
      • direct query of MySQL
      • LiDAX, DTT
      • LDAS, getFramecache
    • A nice thing -- ask the "channel X..." question, get back a file descriptor (or s/t) that has a stream with just that channel.
    • PKI authentication is expensive. Have a non-auth server, only accept connections from local net or specific addresses. Support login/session to speed up multiple requests.
    • Operator in control room problem.
  • diskcacheAPI should be integrated into LDR as a distinct choice and documented.
  • We should again investigate using lsync for LDRVerify.
  • Should there be a channel browsing service? Add channel information to LDR metadata db (seperate tables? db?) LDRVerify would want to verify this information across the 'channel file'.
  • An "admin toolkit" should be assembled: LDRrm, LDRcp, LDRmv, etc. Have a 'read from file' capability.
  • Simplify the publication of data into LDR (Dan Kozak)
  • Monitoring of LDR is very important (Dan, Stuart, Kevin, .....)
  • Display transfer rates from sites
  • Archive transfer rate information
  • On the fly configuration change ...... signal handler, etc. Currently have globus signal handling problems, once done can have everything responding properly to signals
  • separate log file for critical messages (Murali, Stuart, Dan, ...... ) no consensus on what to do. Dan suggested send to syslog .
  • LDRVerify .... wanted, needed, should be a high priority
  • Some notes on bugs:
    • dataFind 'show observatories' includes "None" in the list... why?
    • #89 is of interest to data available page
    • #68 file transfer works, but LDR throws it away because of bad md5. This is very hard to find from looking at LDR logs.

Sam Finn also took some notes at the meeting.


$Id: meeting-011807.html,v 1.3 2007/03/06 11:15:38 kflasch Exp $
LDR Logo
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.