From: Thomas Jordan Date: 16 February 2010 0807:03 CST To: "QuarkNetGrid@fnal.gov Grid" Subject: Fwd: Data Blessing Design II Hi everyone, I wanted you to see this before the meeting on Thursday, my most recent set of notes are below. Data blessing is meant to replace (update?) performance study. The TR that currently runs will need to be modified so that it creates the additional, new plots for blessing (additional because users have said that they still want the old ones as well.) This creates an option. The changes that I have made creates the files needed to draw the plots for blessing at upload time. We could: A. Save those files in the data directory and point gnuplot at them (uses lots of disk.) B. Not create those files at upload time, but create them on demand by moving the code that creates them out of Split.pl and into ThresholdTimes.pl There are other very small details here. I think that we should do B, we already have to run ThresholdTimes at performance study time. What I hope to do on Thursday. 1. Finalize the blessing interface and process for a user There's a lot here, I think that this looks like. 1.0. Uploading, verifying and tagging a "golden file." This means uploading a file, creating a "use case", inspecting the resulting plot, and tagging a file as the "golden file." This first upload puts the user in "learner mode" where she will have to visually, manually inspect the next nnn files in order to bless the data. 1.1 Subsequent uploads in learner mode The user will upload files, identify the use case for those files and manually bless the data until she has blessed (or not) nnn files. (NB nnn may be fewer than the number of uploads as each upload can spawn more than one file.) The point is that the user should understand what good data looks like and doing this several times should help with that. Once that is done, we can ask if they want their uploads to go to automatic, This will mean that all they have to do is choose a use case and we will determine the blessing. Creating a new use case will re-start this process for new files that get tagged with that new test case. 1.2 Further uploads in automatic mode. We check values in the file (averages) and compare to those same values in the golden file. If they all fall within a standard deviation, the file is blessed. If there is some difference, we display the difference and ask the user what they want to do. Certain tasks Choose A or B above Modify Plot.pl to make the plots needed for blessing Design and implement interface (that will do 1.0, 1.1, 1.2) Create tables to hold blessing meta (includes use case.) Write code to do the comparison Create materials about what good data look like and why this is important I hope to meet for 60-90 minutes and discuss this. We can then decide who will do what and set some milestones and dates. I think that there will be time on Thursday to actually get started on some of it. I don't want to spend the day talking and think that we can make good progress if we devote the day to discussing and working on blessing only. See you Thursday. I'll be at the lab on Wednesday. t. -- Yes. We. Can. Thomas Jordan voice: 352.392.3124 fax: 352. 392.8863 mail: UF-Department of Physics PO Box 118440 Gainesville, Florida 32611-8440 shipping: University of Florida Department of Physics B100, Corner Museum Rd. & Gale Lemerand Dr. Gainesville, Florida 32611-8440 Begin forwarded message: From: Thomas Jordan Date: 6 October 2009 14:19:55 EDT To: "QuarkNetGrid@fnal.gov Grid" Cc: Marjorie Bardeen Subject: Data Blessing Design II Notes on my progress on data blessing. I need help with reading the new tuples out of the .meta file and inserting them into the database. 13 new fields. I can do the rest that is here. We need to think about how to fold this into the performance study page. We need to change the workflow associated with that page so that it creates the plots here (as well as the existing ones.) Feedback and open revolt are both welcome. The latter requires that you buy me a drink or two. t. Date: 2009-10-06 13:00:27 -0400 Topic: Data Blessing Design II I have made major modifications to my local Split.pl in order to gather all of the information listed in the first section (calculations) of the notes entitled Dat Blessing Design in this (Code) section of my logbook. The output plots could look like this: In deriving these values, I have learned that the output of the DAQ's ST command contains useful information that decreases the input required of the user at upload time. We still need this information, but the user doesn't have to provide it. I'll make a table to explain: Provided by: User Script Value   X Threshold Voltages for channels 0-3 (four fields)   X The contents of the four control registers--C0, C1, C2, C3 (four, two digit hex numbers)   X The contents of the four TMC registers--T0, T1, T2, T3 (four, two digit hex numbers) X A "use case." (more about this below) (There was an redundant entry in the list in the original doc. The output of TL and n.b. We can get this information if the user enters "TL" and "ST 3 (or 2) 1" Bob may also have them introduce some other useful commands. We still need metadata for these fields for each file that comes out of the split process. Split.pl can add these tuples to the metadata file that it is already creating (it holds things like counts in channel N, start time, etc.) We'll need to read these new values out and insert them into new tables in the existing database. So, the new meta look like this: chan0Thresh int chan2Thresh int chan3Thresh int chan4Thresh int C0 int C1 int C2 int C3 int T0 int T1 int T2 int T3 int use string The user provides the use case. We need to think about how to do this. I'm worried that they may use similar (but still different) words to describe _the same_ case. This would be bad. I imagine that we could have a pick list of their previously entered uses cases along with a field for "other" present on the upload page. They'll still goof it up. This looks like 13 new metadata fields for each file. I also need to work on Plot.pl so that it will draw a version of the plot above. -- Yes. We. Can. Thomas Jordan voice: 352.392.3124 fax: 352. 392.8863 mail: UF-Department of Physics PO Box 118440 Gainesville, Florida 32611-8440 shipping: University of Florida Department of Physics B100, Corner Museum Rd. & Gale Lemerand Dr. Gainesville, Florida 32611-8440