Troubleshooting Cookbook

Tomcat restart on www18: sudo service tomcat6 restart

Tomcat restart on www13: sudo /etc/init.d/i2u2tomcat restart

We're still running on the Tomcat 6 stack on Java 6. We have not successfully tested on Tomcat 7 and/or Java 7. Strange errors occur that we haven't yet nailed down and we have a lot of older components that may never be updated. Be warned. The Tomcat6/Java6 transition also took some work to get right

Database restart (dangerous!): sudo service postgresql-<version> restart

You must restart tomcat6 on the server after database restart. Our database layer can't handle misconnects (and predates most modern layers). Eventually look at jdbc-pool to replace our custom work? Might not be that hard. Have to look at it.

We also use mysql for a few things and hope to deprecate it. CMS eLab uses it as a persistent/fast datastore; we're working on moving that to Postgres where it belongs. Newer databases tend to be properly relational; older ones aren't. Lots of screwy data - unfortunately in use - that will prevent proper use of indexes, keys, etc. frown, sad smile

Deployment:

sudo ./deploy-from-svn branches/<version>
sudo ./deploy-from-svn tags/<version>
sudo ./deploy-from-svn trunk

We are still running off the branches/2.0 version. Trunk currently has major issues (notification framework is screwy)

Nightly rsync from LIGO on the data server + processing work done in a cronjob under the quarkcat account. The password for the sync is in ~quarkcat . To see the commands, run sudo crontab -e -u quarknet

The system attempts to generate new stuff every night and append it to the existing binary files. We then serve it via RESTful python application to our intranet to avoid NFS round-trip latency. We may replace the text-based marshalling format with something like Thrift in the future.

Argonne has a nightly backup of our systems.

Password lists (long and needed - how to securely do this?). Right now the best way may be to use GPG/PGP-encrypted emails

Connecting to ANL machines from offsite

Your SSH key needs to be loaded to ANL MCS' account portal to access their login proxy.

Add these lines to your ~/.ssh/config file (create it if it doesn't exist): (The first set of lines is to use the Argonne proxy machine automagically. The second set of lines is to forward SQL server port voodoo)

Host *.i2u2.org
User
ForwardAgent yes
PubkeyAuthentication yes
ProxyCommand ssh login1.mcs.anl.gov nc %h %p

Host data1.i2u2.org
LocalForward 5433 data1:5432
LocalForward 3307 data1:3306

Central source control uses svn. I prefer using git with the git-svn bridge but your preferences may vary.

There are multiple wikis with some custom code. It is messy. There's the fellows wiki and the library; some of our eLabs transcludes content from the latter wiki. Getting rid of said wikis will require moving content back to our system and altering the transclusion system. Unfortunately, much of the old custom code was targeted at a long-obsolete version of MediaWiki and is problematic. Some of our authentication still relies on it frown, sad smile

We use java6 and tomcat6 to run our stack; minor testing with java7 has been done and there can be strange and unknown side-effects. Don't use it (yet). Compilation takes place via an ant script to deploy everything to proper directories. There are myriad number of config options in build.xml

Apache is our web server frontend (and also handles php code). It, too, has a bunch of configuration parameters that are slightly black magic. Production-server ones should be stored in SVN (and if not, will be shortly).

Bugzilla: holds our bugs. Big list! Some of them very old. In theory we should also be tracking to a milestone but do not yet do so.

Trac: Easier way of tracking changes in SVN (colored diff, etc.)

Custom caching: we have our own caching layer to store results from our metadatabase. It is not very reliable. Need to replace with something like memcached or redis. We also need to properly expire on insertions, which we do not do and sometimes leads to confusion on our users.

-- Main.PhongNguyen - 2012-11-17
Topic revision: r2 - 2013-08-07, forero
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback