Cosmic Ray e-Lab

Features

Plotting of extra data alongside flux:

There has been discussion about the ability to allow users to plot other variables alongside flux in order to look for different correlations. A few of these variables include temperature and pressure, but could include a variety of things. The development for this has two different stages. The first stage involves using the card's ST command to output temperature and pressure data into the raw data:

ST 1008 +273 +086 3349 180022 021007 A 05 C5ED5FF1 106 6148 00171300 000A710F

With the 6000 series cards, you can set this command to run every n minutes. Since this just outputs what we want right into the raw data, it's not that difficult to strip it out, save it somewhere, and use it when users want to see how the flux is affected by temperature and pressure. Here are some examples of the graphs it outputs for temperature and pressure:

Temperature

Pressure

The second stage is more complicated. It involves a lot of decisions about how we want the data to be formatted, how/where we store it, the upload UI, etc. I'll list the issues we'll have to address, then finish with my proposal.

1. The format of the data that we'll use. This is the format that Plot.pl will need to receive for it to plot:

month/day/year hour:minute:second variable

We can either force users to give us a file of space delimited lines, or create a file of space delimited lines from whatever they give us. One possibility is having them give us a CSV file like mass-registration.jsp does. Tom mentioned in a telecon that he wanted to stay away from "spreadsheet" documents because they could just use excel to make a plot instead of loading it into our system. Users would only be able to plot one variable at a time with flux. However, we could generalize it so that they could have a file with lines like so:

month/day/year hour:minute:second variable1 variable2 variable3

In the UI, we could have them specify which column of data (which variable) they would like to plot, then it would be much easier to instead plot a different variable. If we force the users to have headers at the top of each column, then we could parse that and have them select which column based on it's header. This would probably be more useful for a file that is permanently loaded into the system to be used repeatedly, rather than just loaded at analyze-time and discarded.

2. The lifetime of the data that we'll use. Do we have users permanently upload it into our system so they (and others) can access it? Or do we just have them load it on-analyze and then discard it? The latter is done in mass-registration.jsp, and that would be the easiest to implement because of the lack of further decisions. A whole bunch of other issues come up if we decide to store the data permanently. I'll discuss those further down. It would be much more useful to have users permanently save it, so that others would be able to reproduce that data (both from their own studies and from re-running plots that are saved with using that data).

3. Assuming we store the data permanently: 3.1 Where do we store it? It would either be in the user's directory or data directory. 3.2 Where does the UI go for uploading it? I would think the most obvious place would be under the Upload tab 3.3 What does the UI look like for using it? I think a pulldown list so the user can select what they want to do.

My recommendation is to store the data permanently, and save the file in the data directory in the file's ID folder (we could ask for the ID of the detector the file is for on upload like we do for regular data). We could just use our current upload.jsp page, and have a way for users to indicate whether this data is raw data or some other user-defined data used for plotting with flux. Another version of Split.pl could run on it, and still split it into different days, but also put it into the format Plot.pl needs to pass to gnuplot (as indicated about in issue 1.). We should save it with a file extension like ".extra" or something to identify it easily later (i.e. 999.2007.0927.0.extra). We should probably give it it's own file type (other than Data, Plot, Uploaded Image) so that it can have it's own set of metadata. This metadata could contain which variables are in the file (which could be grabbed from the header of each column, if we force the user to have column headers) and which columns those variables live in in the data. In the UI for analysis, we could have a pulldown list that looks at which files in the data/ID/ directory have the ".extra" extension (and correlate with the filenames of the raw data), then read the metadata for those files, and load the variable names into the pulldown list so users will be able to select what to plot based on the variable. Then those files can be collected, passed to the VDL (or Swift), combined using Combine.pl, and passed to Plot.pl, which will output a graph akin to what I linked to above. Here is a link to my idea for the UI in the analysis page:

Here

-- Main.NickDettman - 11 Jan 2008