LIGO plotting problems

On Mon25Apr I noticed that LIGO data cuts off around March 15 in the plotter.
On Mon2May I noted that frame files drop to zero between Mar15 and Mar26 (LLO-M, LLO-T).

I noted that no data is plotted at all after this time. Previously, when data was absent, the plotter would still plot a flat line. Now, there's no line at all.

On Fri20May, Dale replied that flow should have been restored the previous week. Indeed, LHO-M showed nonzero frame files starting 6May

Still no plotting.

Reconstruction

Examining i2u2-data (/disks/i2u2/ligo/data/frames/trend_after23April2013/)

Last-written frame files before flow stopped:

H-M (minute-trend/LHO/H-M-1142/) has

  -rw-r--r-- 1 quarkcat quarkcat 588924 Mar 20 22:10 H-M-1142550000-3600.gwf
  -rw-r--r-- 1 quarkcat quarkcat 586612 Mar 20 23:10 H-M-1142553600-3600.gwf
  -rw-r--r-- 1 quarkcat quarkcat 586472 Mar 21 00:10 H-M-1142557200-3600.gwf

H-T has

-rw-r--r-- 1 quarkcat quarkcat 2675329 Mar 20 23:30 H-T-1142553600-600.gwf
-rw-r--r-- 1 quarkcat quarkcat 2673588 Mar 20 23:30 H-T-1142554200-600.gwf
-rw-r--r-- 1 quarkcat quarkcat 2673948 Mar 20 23:30 H-T-1142554800-600.gwf

L-M has

-rw-r--r-- 1 quarkcat quarkcat 333528 Mar 20 21:20 L-M-1142546400-3600.gwf
-rw-r--r-- 1 quarkcat quarkcat 334104 Mar 20 22:20 L-M-1142550000-3600.gwf
-rw-r--r-- 1 quarkcat quarkcat 333284 Mar 20 23:20 L-M-1142553600-3600.gwf

L-T has

-rw-r--r-- 1 quarkcat quarkcat 1514359 Mar 20 23:40 L-T-1142554200-600.gwf
-rw-r--r-- 1 quarkcat quarkcat 1515791 Mar 20 23:40 L-T-1142554800-600.gwf
-rw-r--r-- 1 quarkcat quarkcat 1514921 Mar 20 23:40 L-T-1142555400-600.gwf

So data flow stopped at about 10 minutes after midnight on 21 March.

On the plotter, however,

At approximately 19:30 (7:30pm) on March 14, the data plotline goes to zero.

At approximately 19:50 (7:50pm) on March 14, the data plotline terminates completely.

-- Main.JoelG - 2016-06-01

Comments