Bluestone Update Procedure Preparation of a new branch Updated Release Grid Nodes Troubleshooting Rolling Back

Bluestone Update Procedure

This page describes the proceedure to follow to release ("roll out") a new revision of the Bluestone software. It also describes how to fall back to a previous revision, should that become necessary.

Bluestone code is currently kept in the CVS reposistory on spy-hill.net, and we roll-out from what has been checked in and tagged there. Eventually, when Bluestone is mature enough, we will use the SVN repository for ELabs instead. Even so, the main steps will be the same, just using SVN branch tags instead of CVS tags. (Note for the future that all tags in SVN are branch tags.)

The current convention is that branches of the Bluestone source code will be named version_M_mm where "M" and "mm" are the major and minor version numbers. Snapshots of the production code are named prod_M_mm, while snapshots of the test branch/release are named test_M_mm. Snapshots are taken immediately before the update, so that we can roll-right back to what was working if we have to. If you have to take different snapshots from different installations, pre-pend the hostname to the snapshot tag.

A list of the current release versions may be found here.

Preparation of a new branch

A developer who is preparing to create a new branch for release needs to verify that everything is working, check the code into CVS, and then tag the code as a branch in CVS. The one complication is that Bluestone makes use of some PHP files which are a part of BOINC. You can tag all files for the branch, including those from BOINC, if they are up to date in CVS, or you can break the link to the BOINC part of the code, tag the branch, and then restore the soft link.

The BOINC part of the code is used in common by the production, test and development releases of Bluestone, if they are on the same machine, as well as the discussion forums and other BOINC components (on www15 and on i2u2.spy-hill.net). Regardless of whether you tag the BOINC code along with the rest of Bluestone, you must remember to break the link before you roll-out a new version of any Bluestone code. If you do not, checking out from the branch will delete all the BOINC code which is not tagged for the branch, which will break the rest of the BOINC site.

When you tag the branch be sure to include the -b flag to the cvs tag command. For example:

      cd /home/i2u2/ligo/tla
      cvs tag -b version_0_72

If you then intend to continue using the existing set of files to do further development (on the trunk) then you need to immediately check out the CVS head (trunk), like so:

      cvs update -A

Updated Release

Right before you roll out a new release of the code, you want to tag the existing code (as a snapshot, not a branch), so that you can easily roll back to that snapshot should there be problems. Then you want to check out from the branch for the newer release. The one complication is again that Bluestone currently makes use of some PHP files which are a part of BOINC, and these are not necessarily checked in or out with the same CVS tags. You therefore need to unlink to the BOINC code, perform the snapshot checkin and branch checkout, and then restore the link to the BOINC code. In the future the dependence on the BOINC code will hopefully be removed, and then we'll just have to do the checkin/checkout.

Before you roll out a new release, you should also verify that the existing release works, by running a complete analysis. That way if there is a problem after the roll-out you will know that it was not there before the roll-out.

As a concrete example of a roll-out, consider the steps required to update from Bluestone v0.70 in production, to v0.71, which is the current "test" version. Here are the steps I followed:

1. First, go to the production directory, check any changes into CVS on the branch, and then tag it as a snapshot:

          cd i2u2/tla
          cvs commit
          cvs tag tla_prod_0_70
   

2. Also go to the test directory and tag that as a snapshot as well:

         cd ../tla_test
         cvs commit
         cvs tag tla_test_0_71
   

3. Now break the link to the supporting BOINC files, which are in separate subdirectories:

         cd ../tla/html 
         rm boinc_html 
   

4. Now go to the top of the production area and check out the branch for the newer release:

         cd .. 
         cvs update -C -d -r version_0_71
         cvs status | more
   

   This last step just verifies version numbers, so you might want to add `grep version` to the pipeline. Make sure all the files show that they are a part of the correct branch.

5. Restore the soft link to the BOINC code:

         cd html
          ln -s ../../boinc/html  boinc_html
   

   The separate directories for the BOINC code all go through the soft link boinc_html, so this should be enough. If not, there is shell script called create_html_links in the html subdirectory which will set up the links correctly.

6. Check the file html/tla/config.php to make sure that the constant CVS_TAG has the proper version tag. There is a bug in CVS where this is no set properly except on a fresh checkout of the file, so you may have to fill this in by hand. This constant is used to display the release version number at the bottom of every page generated by Bluestone. It seems, though, that using cvs update -C gets the version tag correct.

After you roll out a new release you should always test the analysis tool to verify that it works properly. Use the checklist on the BluestoneRegressionTests page

Be sure to log off and log in to test guest authentication still works too, with both an individual user account and with a guest account (via HTTP Basic authentication).

Grid Nodes

Bluestone consists of two general components, the "Front Panel" or Graphical User Interface (GUI) which is provided by the web site, and the "Internals" which implement data transformations or workflows, run ROOT, manage tasks, etc. Both need to be rolled out to a site which runs Bluestone (on the web). But to run tasks on a remote host, as via Grid execution, you also need to roll-out the "Internals" (at least) to the remote host.

Right now the easiest way to do this is to just roll-out everything, as described above, and ignore the GUI code, which is all in the html/ subdirectory. In the future we may (or may not) modify the procedure to not even roll-out the GUI code, but that may actually be more complicated and therefore less useful.

Troubleshooting

• Directory listing is disabled to prevent people hunting around in the TLA directories. If you get a 403 Forbidden error check that index.php is pointing to the correct default file.
• Bluestone makes use of a small subset of files from the BOINC forums. This code is considered stable and won't change, but there is a slight possibility of a problem, perhaps due to a function name conflict. The BOINC files are maintained in their own separate CVS repository. It may be useful to update these from CVS from time to time. Keep in mind, however, that with the soft link described above in place, all versions of Bluestone that have the same soft link depend on the same set of BOINC files.
• If you forget to break the soft link and check out from a new branch you may find that the BOINC files disappear (since they are not a part of the Bluestone branch). The solution is to cd to the BOINC directory and check the files out from CVS (from the head, no branch). Then put the soft link back.

Rolling Back

In the event that there is a problem with the software currently in place, it is straightforward to roll-back to the previous snapshot, which was taken just before the release. You can roll back to the snapshot you took before the last release, or you can roll back to the previous branch. (These are likely to be the same code, but there are cases where they could be different.) You need to remember to break the boinc_html soft link, perform an update using the previous tag, and then restore the soft link. For example, to roll the production version back to the branch for v0.62 I would say

      cd /home/i2u2/ligo/tla/html
      rm boinc_html
      cd ..
      cvs update -C  -r version_0_62
      cd html
      ln -s ../../boinc/html  boinc_html

If you want to go back to the exact snapshot, simply replace version_0_62 with prod_0_62.

Be sure to verify that the tool works completely after you roll-back, or you may make things worse than they were.