LIGO ELabs Data Relay Server Disk Requirements. Software Requirements Network Requirements User account Requirements Procedures rsync server Minute Trends via cron Minute Trends by hand Second Trends on demand Apache web server configuration Trimming the /data partition

LIGO ELabs Data Relay Server

• PEM: Physics Environment Monitoring; seems to be a catch-all label for environmental background monitoring at LIGO. Source of raw data.
• DMT: Data Monitoring Tool, LIGO data processing software. Source of filtered data.
• GDS: Global Diagnostic System, similar to DMT but more specific. Monitors for earthquakes.
• GW: Gravitational waves
• LHO: LIGO Hanford Observatory

The purpose of the ELabs Data Relay Server is to collect data for ELabs from the LIGO datastream as frame files and pass them on to the ELabs data server (at Argonne National Laboratory -- ANL). At present ELabs uses PEM channels from seismometers, magnetometers, weather stations and tilt meters, along with the DMT seismic channels and the GDS Earthquake channels. There are no GW channels in ELabs frame files.

When LIGO initially became involved in ELabs a machine at LHO named tekoa.ligot-wa.caltech.edu was used to collect and store the ELabs frame files, and tekoa also hosted the web-based analysis tool, called Bluestone. Hosting the ELabs dataset and analysis tool has been moved to the ELabs cluters at ANL, while the task of collecting ELabs data frames files is to be moved to a machine at Caltech, called terra.ligo.caltech.edu, where it will also have access to Livingston data.

The purpose of this page is to describe the requirments and configuration for the machine at Caltech (based in part on the existing configuration of tekoa at LHO).

The basic function of the relay machine is to use an NFS mount of ldas:/archive/frames to obtain access to full frame files, for either minute-trends, second-trends, or raw data, to use the FrCopy program to extract just the ELabs PEM/DMT channels into smaller frame files, and then make those files available via either htttp (ie from a web server) or rsync (which is more efficient). These frames will then be pulled down by the ELabs data server at ANL, either by http or rsync.

As of 4 January 2009 tekoa was turned off, and as a result no new data have been available for I2U2. Before that, the data were being pulled from tekoa by data2.i2u2.org using rsync over ssh, using a private user account. The new machine at Caltech could serve the data via rsyncd and thus not have to grant ssh login access to an outside user.

Disk Requirements.

The relay machine needs access to ldas:/archive/frames, including ldas:/archive/frames/dmt.

The relay machine needs a data storage area, prefereably a separate disk partition, in which to save frame files until transfered to Argonne. On tekoa this is the /data partition, with ELabs data stored under /data/ligo/frames. On tekoa we have 173GB available, but so far have only used about 40GB. We likely do not even need this much space on the new machine, because unlike tekoa, which was the initial data server, the data are now going to be relayed to another machine at Argonne and served to ELabs from there. But we will need some extra storage to generate frames at least back to 4 Januray 2009.

In what follows I will refer to this staging area as /data/ligo/frames, but it could be called something else. Adjust as appropriate.

It is probably a good idea to make /data or /data/ligo/frames a separate disk partition, if that is possible. If it happens to fill up then this can be dealt with, but it won't cause problems for any other functions hosted on the machine. A cron job will be set up to trim the /data/ligo/frames area to remove the oldest frame files, if there are concerns about available storage.

Software Requirements

The following software is needed:

• FrCopy from ligotools. This is used to extract the ELabs channels and write new frame files for I2U2. Between S5 and S6 LIGO started using the version 8 frame format, so the version of FrCopy used must understand this newer format.
• update-rds.php is a PHP script which basically serves as a wrapper to FrCopy. It determines the GPS timestamps for frame files and then iterates over input frames, invoking FrCopy to create output frames. This script is run from the command line, or can be invoked from cron.
• PHP, which is a scripting language commonly used to generate dynamic web pages. PHP is used to create the web-based LIGO Analysis Tool, Bluestone, and it's also used to generate dynamic content such as forums and status pages on Einstein@Home. But for this relay machine we will use PHP as a CLI scripting language, like perl or Python. Any recent version of PHP should do.
• rsyncd, a server which can provide access to the /data/ files using the efficient rsync protocol (port 873). This is commonly run via xinetd. In ths short run this is probably the easiest and most efficient way to serve the data from Caltech to Argonne.

In the future we may also want:

• Apache web server - the relay machine could run Apache httpd for several reasons:
  1. To serve the data via HTTP (port 80) and eventually HTTPS (port 443).
  2. To accept requests for on-demand generation of second-trend and raw frame files
  3. To run a private copy of Bluestone for use in monitoring the RDS.

Network Requirements

• This machine needs an NFS mount of the LDAS /archive/frames filesystem, including /archive/frames/dmt, so that it can get both PEM and DMT data.
• This machine needs to be accessible from "outside" on port 873 (for rsync), and possibly

port 80 (HTTP), and/or port 443 (secure HTTP) if we set up the on-demand system described later in this page, and/or want to run a private version of Bluestone.

Whoever is maintaining the machine will need ssh access, but if we use rsyncd then we won't need ssh access to use rsync. If security is a concern, we should be able to limit access to rsyncd by IP address range using tcpwrappers

User account Requirements

In day-to-day operation nothing has to be run as 'root'. There needs to be at least one user account to perform the tasks, and this account needs to be able to run tasks via cron. If more than one user is to be able to work on this system they could be joined together in the same Unix group and granted appropriate group permissions.

Procedures

rsync server

See the page LIGO data Rsync Server on how to configure the rsync server.

Minute Trends via cron

The update-rds.php script is now running on tekoa. It is invoked every hour by a cron task to generate new minute-trend frames for I2U2. The script can be obtained via CVS with

 cvs -d :pserver:anonymous@spy-hill.net:/usr/local/cvsroot/i2u2 co ligo/tla/bin/update-rds.php

You may need to edit this script, though Eric will try to make adjustments to it to make that unnecessary. Here are the things that might need adjusting:

1. If the data storage area is something other than /data/ligo/frames then edit the script update-rds.php and change the definition of the variable $RDS_data_path to the appropriate path.
2. If the data source is not mounted at /archive/frames then edit the script to change occurances of the variable $source_dirs[] to the appropriate path.
3. If FrCopy is not at /ligotools/bin/FrCopy then edit update-rds.php to change the path as appropriate.

Finally, to generate new frame files every hour, add the following line to the non-root user's crontab to run update-rds.php once per hour to obtain the latest minute-trends from Hanford:

31 * * * *  php $HOME/bin/update-rds.php -t M -i H

Add a similar line with "-i L" instead to get the latest minute-trends from Livingston.

Minute Trends by hand

The procedure just described invokes update-rds.php to generate the latest set of minute trend frames, which is what we want for continuous operation. By default, the script generates the most recent 12hrs of frame data, and will skip backward past up to 25hrs of missing data. Command line options allow you to change the time interval over which frames are generated, which can be useful for filling in a gap after an outage. For example, the command

update-rds.php  -r -s 899164814  -e 896572814  

Would regenerate minute trends (by default) for Hanford (by default) for that GPS time interval. Use the "-t" option to specify trend type (either "M" for minute-trends, "T" for second-trends, or "R" for raw data) and the "-i" option to specify the interferometer (either "H" or "L"). Other command line options are "-r" or "-x" or "-p" which are documented in the script:

 *   -r     rebuild/replace - will overwrite existing frame files
 *           in the destination directory instead of stopping  
 *
 *   -x     exists - skip over existing destination files, continue
 *           work with the next           
 *
 *   -p     partial data OK - will use whatever frames are available
 *           instead of skipping the frame (eg missing DMT frames)
 *

The "-x" flag is useful for a quick "fill in the gaps", while "-p" should be used if DMT frames were lost in a given interval (really lost, not just delayed), so that we can at least build frames containing PEM data. Using the "-r" flag to rebuild the entire RDS on tekoa last took a couple of days (and almost 40GB).

Second Trends on demand

The instructions above should be sufficient to get minute-trend frame files for ELabs from Caltech to Argonne. Eventually, we will also want to provide second-trends and raw frame files for the ELabs RDS.

There is not enough disk space to store second-trends and raw data, even though ELabs frames are smaller than regular LIGO frames. And it's likely that they would not be used in any case. So we will need to generate these frame files on demand. The original method of lauching this "data prep" task was via ssh, but Eric is setting up a mechanism to trigger the script via a web form. A task running at Argonne would then fill in and submit the web form to start this data prep step. The frame files would be generated in the appropriate place under /data on the machine at Caltech, and then these data would then automatically copied to the ANL cluster.

The same will apply to raw data at full sampling rate, again just for the ELabs channels.

Apache web server configuration

See the page LIGO data Web Server about how to configure the web server to serve the data files. Instructions on setting up Bluestone will be on a separate page.

Trimming the /data partition

After data have moved through the relay machine we don't need to keep them around anymore, though we might as well keep a copy for a while, to allow the Argonne server time to pull down a copy, even in the event of an extended outage at either end. To make sure we always have enough working space we can trim the /data partition by running a find task from cron. This can be run under a non-root account which has write access to the /data partition. Simply add these lines to that user's crontab:

20 04 * * * find /data/ligo/frames  -type f -mtime +90 -exec rm -f {} \;
40 04 * * * find /data/ligo/frames  -depth -mindepth 1 -ctime +1 -type d -empty -exec rmdir {} \;  

The first line, run at 4:20AM, deletes all existing files which have not been modified within the last 90 days. Adjust the time length accordingly to make sure that there is enough space on the partition for new files to be generated, but keeping old files around long enough that they can be pulled down by the ELabs machine even after a long outage. The second line, run at 4:40AM, deletes empty directories, a day after the files in those directories have been removed. You can of course change the times as needed.