LIGO Data Formats

Notes

All data is big endian. Floating point data is IEEE representation. Booleans are byte sized and anything ! = 0 is true.

Frame files

The .gwf file format was developed by the LIGO collaboration and is the standard for LIGO/VIRGO data. You can find more information on it in the specification document issued by LIGO.

LIGO provides a suite of tools called ~LIGOTools for working with the experiment's data; these are already installed in the directory /usr/local/ligotools/ligotools/ on i2u2-data.

LIGOTools is similar to a Linux software package, though it's not maintained in a repository. You can download a copy of the ligotools_init_2.4.tar file from LIGO, and there's also a copy in quarkcat's home folder on i2u2-data. Eric's instructions for installing and managing LIGOTools seems to be still good as of 2017.

The LIGOTools utility FrDump can be used to examine frame files. The basic command is

$ /usr/local/ligotools/ligotools/bin/FrDump -i framefile.gwf

For example,

$ sudo /usr/local/ligotools/ligotools/bin/FrDump -i /disks/i2u2/ligo/data/frames/trend_after23April2013/minute-trend/LLO/L-M-1060/L-M-1060002000-3600.gwf

(remember that many of the data directories have owner-only permissions, so sudo is often necessary). This command outputs

-----------Parameters used--------------
  Input  Files: /disks/i2u2/ligo/data/frames/trend_after23April2013/minute-trend/LLO/L-M-1060/L-M-1060002000-3600.gwf
  First frame  : 0 0 (GPS=0.0)
  Last  frame  : 2147483647 2147483647 (GPS=2147483647.0)
  Debug level  : 1
 Dump all Frame info
----------------------------------------
/disks/i2u2/ligo/data/frames/trend_after23April2013/minute-trend/LLO/L-M-1060/L-M-1060002000-3600.gwf   1060002000.000000 3600  0.000000 0.000000 
File(s) summary:
 1 Frames in the requested time range (0 to 10000000000 (GPS))
     First frame start at:1060002000 (UTC:Thu Aug  8 12:59:45 2013) length=3600.00s.
        Last frame end at:1060005600 (UTC:Thu Aug  8 13:59:45 2013) length=3600.00s.
 ADC :   300 type of AdcData :
       L1:PEM-CS_MAG_VERTEX_X_OUT_DQ.max    L1:PEM-CS_MAG_VERTEX_X_OUT_DQ.mean    L1:PEM-CS_MAG_VERTEX_X_OUT_DQ.min    L1:PEM-CS_MAG_VERTEX_X_OUT_DQ.n    L1:PEM-CS_MAG_VERTEX_X_OUT_DQ.rms
       L1:PEM-CS_MAG_VERTEX_Y_OUT_DQ.max    L1:PEM-CS_MAG_VERTEX_Y_OUT_DQ.mean    L1:PEM-CS_MAG_VERTEX_Y_OUT_DQ.min    L1:PEM-CS_MAG_VERTEX_Y_OUT_DQ.n    L1:PEM-CS_MAG_VERTEX_Y_OUT_DQ.rms
       L1:PEM-CS_MAG_VERTEX_Z_OUT_DQ.max    L1:PEM-CS_MAG_VERTEX_Z_OUT_DQ.mean    L1:PEM-CS_MAG_VERTEX_Z_OUT_DQ.min    L1:PEM-CS_MAG_VERTEX_Z_OUT_DQ.n    L1:PEM-CS_MAG_VERTEX_Z_OUT_DQ.rms
       L1:PEM-EX_MAGX_OUT_DQ.max    L1:PEM-EX_MAGX_OUT_DQ.mean    L1:PEM-EX_MAGX_OUT_DQ.min    L1:PEM-EX_MAGX_OUT_DQ.n    L1:PEM-EX_MAGX_OUT_DQ.rms
       L1:PEM-EX_MAGY_OUT_DQ.max    L1:PEM-EX_MAGY_OUT_DQ.mean    L1:PEM-EX_MAGY_OUT_DQ.min    L1:PEM-EX_MAGY_OUT_DQ.n    L1:PEM-EX_MAGY_OUT_DQ.rms
       L1:PEM-EX_MAGZ_OUT_DQ.max    L1:PEM-EX_MAGZ_OUT_DQ.mean    L1:PEM-EX_MAGZ_OUT_DQ.min    L1:PEM-EX_MAGZ_OUT_DQ.n    L1:PEM-EX_MAGZ_OUT_DQ.rms
       L1:PEM-EX_SEISX_BLRMS_10HZ30.max    L1:PEM-EX_SEISX_BLRMS_10HZ30.mean    L1:PEM-EX_SEISX_BLRMS_10HZ30.min    L1:PEM-EX_SEISX_BLRMS_10HZ30.n    L1:PEM-EX_SEISX_BLRMS_10HZ30.rms
       L1:PEM-EX_SEISX_BLRMS_10HZ30LOG10.max    L1:PEM-EX_SEISX_BLRMS_10HZ30LOG10.mean    L1:PEM-EX_SEISX_BLRMS_10HZ30LOG10.min    L1:PEM-EX_SEISX_BLRMS_10HZ30LOG10.n    L1:PEM-EX_SEISX_BLRMS_10HZ30LOG10.rms
       L1:PEM-EX_SEISX_BLRMS_1HZ3.max    L1:PEM-EX_SEISX_BLRMS_1HZ3.mean    L1:PEM-EX_SEISX_BLRMS_1HZ3.min    L1:PEM-EX_SEISX_BLRMS_1HZ3.n    L1:PEM-EX_SEISX_BLRMS_1HZ3.rms
       L1:PEM-EX_SEISX_BLRMS_1HZ3LOG10.max    L1:PEM-EX_SEISX_BLRMS_1HZ3LOG10.mean    L1:PEM-EX_SEISX_BLRMS_1HZ3LOG10.min    L1:PEM-EX_SEISX_BLRMS_1HZ3LOG10.n    L1:PEM-EX_SEISX_BLRMS_1HZ3LOG10.rms
       L1:PEM-EX_SEISX_BLRMS_300M1HZ.max    ...
 Ser :     0 type of SerData :
 Proc:     0 type of ProcData:
 Sim :     0 type of SimData :
 Detector: 1 type of Detector: LLO_4k
 StatData: 0 type of StatData:
 Event   : 0 Types of event in the file
 Simulated Event   : 0 Types of event in the file

Stream files

Stream files are found in /disks/i2u2/ligo/data/streams/, accessible from i2u2-data. Each stream represents the continuous output of an individual seismic detector channel.

Each stream has three associated files in the streams/ directory; for example,

• L1:PEM-CS_SEIS_LVEA_VERTEX_Y_BLRMS_0MHZ30.bin
• L1:PEM-CS_SEIS_LVEA_VERTEX_Y_BLRMS_0MHZ30.index.bin
• L1:PEM-CS_SEIS_LVEA_VERTEX_Y_BLRMS_0MHZ30.info

Files ending with .bin are the main stream data files; you'll note that they are much larger than the others in the set. Files ending with .index.bin are index files. Neither of these are readable. Files ending with .info are small, readable metadata files.

Reading stream index files

Using StreamTool

StreamTool is a java program that converts stream index files into human-readable format. It was written by Mihael and is not in the project's source code. See the StreamTool setup page for instructions on getting it working.

StreamTool can convert index files into readable format:

$ java gov.fnal.elab.ligo.data.engine.StreamTool H1:PEM-CS_SEIS_LVEA_VERTEX_Z_BLRMS_10HZ30.bin index > output-filename

(the output volume will be large, so I recommend redirecting to a file).

The output of this conversion will look something like

Channel: H1:PEM-CS_SEIS_LVEA_VERTEX_Z_BLRMS_10HZ30
Range: 2013-04-21 05:59:59.984 (1050573600) - 2016-09-27 14:29:58.983 (1159036199)+
Average sample rate 4.5469e-07 samples/s
         # |       Time |             GPS Time    |         Period (1/s)
------------------------------------------------------------------------
         1 |          0 | 1980-01-05 19:00:00.000 |      1050573600.0000
         2 | 1050573600 | 2013-04-21 05:59:59.984 |              60.0000
         3 | 1050695941 | 2013-04-22 15:59:00.984 |            1858.0000
         4 | 1050697799 | 2013-04-22 16:29:58.984 |               1.0000
...

Each line of the index records one of the frame files used to construct the stream. The columns are

#: The record index. A primary key to navigate the file.

Time: The timestamp of the frame file, measured in seconds from 7:00pm on Jan 5, 1980. This reference time should always be the first entry in the index. Despite the column header, this is what's usually referred to as "GPS Time."

GPS Time: A more human readable form of the frame's timestamp. Despite the header, this is what's usually referred to as "UTC Time." This seems like a good opportunity to link to LIGO's handy converter tool for translating between GPS and UTC timestamps.

Period (1/s): This is the time interval covered by this frame file. Despite the header, these numbers are in units of seconds. For every record, the Time value plus the Period value should equal the Time value of the next record.

Using stream index files

So if you wanted to calculate what record index a certain TIME would have, you would do something like this:

recordsSkipped = 0

indexRecord = lastIndexRecord = readIndexRecord()

while TIME < indexRecord.time {

indexRecord = readIndexRecord()

// Rate = Time / Records => Records = Time / Rate

recordsSkipped += (indexRecord.time - lastIndexRecord.time) / lastIndexRecord.rate

lastIndexRecord = indexRecord

}

// add offset with current rate

recordsSkipped += (TIME - lastIndexRecord.time) / lastIndexRecord.rate

Now that you have that, you can read records from the streams. The stream records are:

<record> ::= <valid> <time> <sum> <ssq>

<valid> ::= <boolean>

<time> ::= <double> // gps time

<sum> ::= <double>

<ssq> ::= <double>

This is 25 bytes total (1 + 8 * 3).

So if you wanted to calculate the mean value between T1 and T2, you would get the index of the records in the stream, R1, R2, using the above procedure, and then:

mean = (sum[R2] - sum[R1]) / (time[R2] - time[R1])

There are tools to do it. Reading various data types can be done with EncodingTools.java.

Here's an example of how to do it using the classes that the web site uses:

// make sure you have DataServer.py started in the directory with the streams

ServiceLIGOFileReader.Factory f = new ServiceLIGOFileReader.Factory("<a href="http://localhost:8100" target="_blank">http://localhost:8100</a>");

LIGOFileDataEngine de = LIGOFileDataEngine.getEngine(&lt;directoryWithTheStreams&gt;, f);

// Select a relevant DataPath from de.getPaths() (these are channel names essentially)

// but unfortunately there doesn't seem to be a method to find a DataPath by name, so

// iterate until DataPath.getName() is, say "L1:PEM-EY_SEISY_BLRMS_300M1HZ.mean"

DataSet ds = de.get(dp, dp.getTotalRange(), new Options().setSamples(&lt;number of samples&gt;));

for (int i = 0; i &lt; N; i++) {

  System.out.println(ds.getX(i) + " -&gt; " + ds.getY(i));

}

-- Main.JoelG - 2016-06-15

Comments

• T970130-v1.pdf: LIGO frame format specification document