Search


DASWG LAL Doxygen

Docs

How-To's
Technical
Software Docs
Minutes

Download

Browse CVS, Git, or SVN
Software Repositories
OS Security Updates
LIGO software virtual machine
VMware SL6 Install

Participate

SCCB - Software Change Control Board
Edit these pages
Sub-committees
Mailing List
Telecon

Projects

aLIGO Planning
BAYESTAR
DMT
Glue
GraceDB
gstlal
LALSuite
LDAS
LDG Client/Server
LDM
LDR
ligoDV
LIGOtools
LIGOtools
LVAlert Administration
LVAlert
MatApps
LSCGIS
Metaio
NDS Client
OSG-LIGO
PyCBC
PyLAL
LSCSOFT VM

Legacy Projects

geopp
Onasys

LSCSoft Repository Maintainer HOWTO

Last Update: 05/15/2013

Overview

Pre-built software required for LSC data analysis is distributed from the DASWG software repository packaged for a variety of operating systems. This document explains in a general way how a package is put into the repository for distribution to the LSC. More detailed steps can be found in the Wiki web page LSCSOFT_Notes.

Red Hat based OS's Repository: Scientific Linux, Scientific Linux, Fedora, RedHat

The Scientific Linux 6.1 [SL6.1] repository will serve as our reference OS, but this instructions are vaild also for Red Hat based OS's: SL6.1, Scientific Linux, Fedora, RedHat*, etc. The tarball used to build the software, the rpms built for the OS, and the srpms are all published into the repository. More information is accessible in the Wiki page LSCSOFT_Notes.

To improve functionality, each repository directory maintains a XML file, yumgroups.xml, in which all existing and new software must be defined and gathered in groups. This way, YUM can be used to download groups of packages. So, it's important to review this issue first.

I- Maintaining the yumgroups.xml file

The Maintainer can choose between modifying this file with the YUM tools (yum-groups-manager), or by hand. We strongly recommend to do it by hand to get familiar with the xml structure of the file and after a while switch to use the more sophisticated YUM tools.

"LSCSOFT-ALL" is a "YUM-defined" group which is used to yum-download the LSC software bundle from the repository, and this group is defined in yumgroups.xml for each release. This file is mostly self-explanatory, it basically defines a set of groups and associated rpms. Its estructure is similar to

<comps>

 <group>
    some identification tags...
    <packagelist>
      <packagereq type="mandatory">mypackage_1</packagereq>
      ...
      <packagereq type="mandatory">mypackage_n</packagereq>
    </packagelist>
 </group>

 <group>
 ...
 </group>

</comps>

All approved rpms must have an entry of the form
<packagereq type="mandatory">mypackage</packagereq>
in the "LSCSOFT-ALL" group. If a new package is added to the repository and is approved for release as part of LSCSoft, then an entry should be added for each rpm associated with the package.

More concretely, here's an example of how it really looks

<?xml version="1.0"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
  <group>
    <id>lscsoft-all</id>
    <uservisible>true</uservisible>
    <name>LSCSOFT-ALL</name>
    <description>Packages for the full installation of all LSCSoft for LIGO</description>
    <packagelist>
      <packagereq type="mandatory">dol</packagereq>
      <packagereq type="mandatory">dol-debuginfo</packagereq>
      <packagereq type="mandatory">glue</packagereq>
      <packagereq type="mandatory">glue-debuginfo</packagereq>
      <packagereq type="mandatory">lal</packagereq>
      <packagereq type="mandatory">lal-devel</packagereq>
      <packagereq type="mandatory">lal-debuginfo</packagereq>
      <packagereq type="mandatory">lalapps</packagereq>
      <packagereq type="mandatory">lalapps-debuginfo</packagereq>
      <packagereq type="mandatory">libmetaio</packagereq>
      <packagereq type="mandatory">libmetaio-devel</packagereq>
      <packagereq type="mandatory">libmetaio-utils</packagereq>
      <packagereq type="mandatory">libmetaio-debuginfo</packagereq>
      <packagereq type="mandatory">compat-libmetaio</packagereq>
      <packagereq type="mandatory">libframe</packagereq>
      <packagereq type="mandatory">libframe-devel</packagereq>
      <packagereq type="mandatory">libframe-utils</packagereq>
      <packagereq type="mandatory">libframe-debuginfo</packagereq>
      <packagereq type="mandatory">libframe-matlab</packagereq>
      <packagereq type="mandatory">compat-libframe</packagereq>
      <packagereq type="mandatory">lscsoft-user-env</packagereq>
      <packagereq type="mandatory">root</packagereq>
      <packagereq type="mandatory">root-debuginfo</packagereq>
      <packagereq type="mandatory">framecpp</packagereq>
      <packagereq type="mandatory">framecpp-devel</packagereq>
      <packagereq type="mandatory">framecpp-debuginfo</packagereq>
      <packagereq type="mandatory">python-pyRXP</packagereq>
      <packagereq type="mandatory">python-pyRXP-debuginfo</packagereq>
      <packagereq type="mandatory">cfitsio</packagereq>
      <packagereq type="mandatory">cfitsio-devel</packagereq>
      <packagereq type="mandatory">numpy</packagereq>
      <packagereq type="mandatory">R</packagereq>
      <packagereq type="mandatory">R-coda</packagereq>
      <packagereq type="mandatory">R-xtable</packagereq>
      <packagereq type="mandatory">spr-devel</packagereq>
      <packagereq type="mandatory">spr</packagereq>
      <packagereq type="mandatory">python-matplotlib</packagereq>
      <packagereq type="mandatory">scipy</packagereq>
      <packagereq type="mandatory">pylal</packagereq>
      <packagereq type="mandatory">python-sqlite2</packagereq>
      <packagereq type="mandatory">xorg-x11-server-Xvfb</packagereq>
      <packagereq type="mandatory">ImageMagick</packagereq>
      <packagereq type="mandatory">ipython</packagereq>
      <packagereq type="mandatory">gsl-static</packagereq>
      <packagereq type="mandatory">java-1.6.0-openjdk</packagereq>
      <packagereq type="mandatory">ddd</packagereq>
      <packagereq type="mandatory">cyrus-sasl-gssapi</packagereq>
      <packagereq type="mandatory">gds-core</packagereq>
      <packagereq type="mandatory">gds-crtools</packagereq>
      <packagereq type="mandatory">gds-debuginfo</packagereq>
      <packagereq type="mandatory">gds-devel</packagereq>
      <packagereq type="mandatory">gds-monitors</packagereq>
      <packagereq type="mandatory">gds-runtime</packagereq>
      <packagereq type="mandatory">gds-web</packagereq>
    </packagelist>
  </group>

  <group>
   ...other groups / repeat similar structure
  </group>
   ...

</comps>
The complete file can be reached logging in a UWM machine, say for example, gc.phys.uwm.edu, in the directories
# Remember that LDG supports only 64bit packages, ergo, all 32bit (*i386,*i686) references does not exist any more

/home/htdocs/uwmlsc/secure/daswg/download/software/scientificlinux/6.1/x86_64/repodata/yumgroups.xml

If you modify the yumgroups.xml file, you should regenerate the header and repository information. For example, if yumgroups.xml is updated in the Scientific Linux 6.1, then must do (assuming you've access to the repository machine and have the permissions; read ahead for more details)

export LSCSOFT_REPOSITORY=/home/htdocs/uwmlsc/secure/daswg/download
cd $LSCSOFT_REPOSITORY
./lscsoft-publish-software.py -p scientificlinux -r 6.1 --arch x86_64

II- Building packages to release into the LSCSOFT repository

These instructions assume you have installed in your build machine all the development packages to build RPMs (gcc compilers, make, autoconf, automake, development libs, rpm, rpm-build, redhat-rpm-config, etc., which in turn belong to the YUM groups "Development Tools" and "Development Libraries" from the standard Scientific Linux repositories) AND a released software package for which you have a tarball (*.tar.gz), or the *.src.rpm which contains a spec file to build rpms.

  • First, log into your build machine as 'root' and create an auxiliary account that will be the responsible to build the rpms; let's say the 'rpmbuilder' account and switch to it,
    useradd -m rpmbuilder
    su - rpmbuilder
    
    # this will switch the root user to the  /home/rpmbuilder  directory of the Maintainer account
    pwd  
    # you must be now in  /home/rpmbuilder
    
  • Create tree building directories:
    mkdir -p rpmbuilding/{SOURCES,SRPMS,RPMS/{noarch,x86_86},SPECS,BUILD}
    
  • If only have the tarball, put it into the SOURCES directory and build it
    mv mypackage-X.YY.tar.gz /usr/src/redhat/SOURCES/
    rpmbuild -ta $HOME/rpmbuilding/SOURCES/mypackage-X.YY.tar.gz
    
  • If have the mypackage-X.YY.src.rpm, then install it and build it
    rpm -Uhv mypackage-X.YY.src.rpm
    rpmbuild -ba /usr/src/redhat/SPECS/mypackage.spec
    
  • If all goes well, you will have a set of rpms in $HOME/rpmbuilding/RPMS/x86_64 and source rpms in /usr/src/redhat/SRPMS.

    III- Publishing released software

    The current procedure to publish the software is already described in the other wiki pages listed in the Official Repositories page, so we will describe here the old procedure that use a python script to put all the files inside the relevant directories and rebuild the YUM database of the repositories.

    To "publish" mypackage-X.YY.rpm, you must put all the package *tar.gz, *rpm, *src.rpm files living in your building machine, into the physical repository machine (in this case is gc.phys.uwm.edu). In other words, the source tarballs must go inside gc.phys.uwm.edu:/path/to/source/tarballs, the source rpms must go inside gc.phys.uwm.edu:/path/to/SRPMS/, and the binaries must go inside gc.phys.uwm.edu:/path/to/x86_64/. After this, the YUM database must be recreated with the createrepo command. Once the database is re-created, all the rpms will be automatically available via Internet: the packages are now published.

  • First, from your building machine copy all the packages to your personal account in gc
    scp /your/path/*tar.gz  /your/path/*rpm /your/path/*src.rpm youridaccount@gc.phys.uwm.edu:~/.
    

    Then log into gc.phys.uwm.edu (the UWM repository machine), and move the software to the /home/htdocs/uwmlsc/secure/daswg/download/software/Scientific Linux/incoming directory. According with the nature of your building (if the packages are for production or for testing before using them in a production environment), you can choose to publish for production or for testing. Assuming you use the bash shell, use commands like
    export LSCSOFT_REPOSITORY=/home/htdocs/uwmlsc/secure/daswg/download
    mv mypackage-X.YY.tar.gz   mypackage-X.YY.rpm  mypackage-X.YY.src.rpm   $LSCSOFT_REPOSITORY/software/Scientific Linux/incoming/
    

  • Then change to this directory ($LSCSOFT_REPOSITORY) and run the publishing script (named lscsoft-publish-software.py) for the appropriate repository.

    This example publishes your packages to the SL6.1 production repository:
    ## IMPORTANT NOTE:  as of 15/May/2013, the python script depicted below does not work (it worked for old CentOS packages);  modifications need to be done...
    
    cd $LSCSOFT_REPOSITORY
    ./lscsoft-publish-software.py -g $LSCSOFT_REPOSITORY/software/Scientific Linux/incoming/ -p Scientific Linux -r 6.1 --arch x86_64
    

    To publish packages to the LSCSOFT testing repo add  -t  at the end of the lscsoft-publish-software.py command.
    ./lscsoft-publish-software.py -g $LSCSOFT_REPOSITORY/software/Scientific Linux/incoming/ -p Scientific Linux -r 6.1 --arch x86_64  -t
    

    Note that the machine on which lscsoft-publish-software.py is run must have both createrepo and repoview installed (please read LSCSOFT repository maintainance: RPMs building & YUM groups) since the python script uses these two commands to build the content and webpages of the repository directory you're working on.

    This other example publishes your packages to the Scientific Linux 6.1 repository run the following:
    ./lscsoft-publish-software.py -g $LSCSOFT_REPOSITORY/software/Scientific Linux/incoming/ -p Scientific Linux -r 6.1 --arch x86_64
    


    Now, the software should be available for users using the instructions at Download > Repositories. Go to the link and check this!


    IV- Announcement E-mail

    Send an Announcement E-mail to DASWG (daswg+announce@gravity.phys.uwm.edu) notifying the release(s) of the software package via LSCSOFT yum groups. The Maintainer is free to redact the content of the email; however, below is sample text for announcing package updates to the lscsoft repository. Make the appropriate changes (if the software is published in the Testing or in the Production repositories) and send the email to the Daswg mailing list.

    Subject:  SL6.1 LSCSOFT Testing repositories updated: package_name-1, package_name-2, package_name-3
    
    Respected DASWG members
    
    This is to announce that the Scientific Linux 6.1 LSCSOFT Production repository [64bit only]
    has been updated with the following release(s)
    
           package_name-1     version
           package_name-2     version
           package_name-3     version
           ...
    
    
    
    More info
    http://www.lsc-group.phys.uwm.edu/daswg/download/repositories.html
    https://wiki.ligo.org/SCCB/WebHome
    
    # If the package is deployed in the Production repos, include the URL
    https://wiki.ligo.org/LdgSysAdmin/SoftConfigRollOut
    
    
    Best Regards
    UWM IT Team
    
    


    Debian Repository

    This task is overtaken by Hannover team. See LSCSoft Debian Lenny.

    Mac OS X

    Mac OS X Universal Binaries can be built for distribution using the script create-lscsoft-mac-os-x. You will also need to download portable-libtool into the same directory. To use this script you must have installed the latest XCode tools. You will also need Universal MacPython2.4 and various associated packages. Download and install lscsoft-macpython2.4.dmg.

    The script will download the supported LSCSoft packages (and some needed non-LSC packages) and build them.

    curl -O https://www.lsc-group.phys.uwm.edu/daswg/download/create-lscsoft-mac-os-x 
    curl -O https://www.lsc-group.phys.uwm.edu/daswg/download/portable-libtool
    chmod +x create-lscsoft-mac-os-x
    ./create-lscsoft-mac-os-x --fresh
    

    Once the packages are built, you need to package them. Download lscsoft-packages.tar.gz to get the PackageMaker files and associated resources. Untar this in the same directory that you ran create-lscsoft-mac-os-x. This will create a directory Packages. Within this directory open the following .pmproj files, check the version numbers to make sure that they are up-to-date, and build the packages:

    curl -O https://www.lsc-group.phys.uwm.edu/daswg/download/software/darwin/lscsoft-packages.tar.gz
    tar zxvf lscsoft-packages.tar.gz
    cd Packages
    open FFTW.pmproj
    open Frame\ Library.pmproj
    open GNU\ Scientific\ Library.pmproj
    open Glue.pmproj
    open LSCSoft\ User\ Environment.pmproj
    open MetaIO\ Library.pmproj
    open Pkg\ Config.pmproj
    
    Now open the LSC Software Suite.pmproj and rebuild this package. Make sure that the package gets built in the subdirectory LSC Software Suite.
    open LSC\ Software\ Suite.pmproj
    
    Finally, make the disk image with the following commands (change the version as appropriate!)
    hdiutil create -srcfolder "LSC Software Suite" lscsoft-1.2.dmg
    hdiutil internet-enable -yes lscsoft-1.2.dmg
    
    That should be it!
  • $Id: lscsoft-maintainer.html 4437 2013-05-15 21:26:02Z xavier $