Status for Andrew DeFaria

• Worked with Zoran to resolve merge issues from RISS15_Integration -> osaka_strm
• Documented how to build development views on Tiburon for the L2 and Base components
• Assisted engineer on problems with view and installer_cmp
• Changed RISS15_Integration to not do testing all the time
• Worked more on osaka_strm build scripts

I've made some progress converting the osaka_strm* scripts from .bat -> .sh. There are still some problems:

• I'm getting permission denied for several files when attempting to write to the E drive (from //pacifica/devenv) and also to TheCity (i.e. //thecity/releaseArchive/...). Here are the error messages:

  Could not open E:\/build-osaka_strm.log for appending: Permission denied
  

  In the past we fixed this by doing chmod on the files/directories over on Pacifica however:

  Pacifica:ll build-osaka_strm.log 
  -rw-rw-r--  1 build hp-cc 3707 May 25 10:30 build-osaka_strm.log
  Pacifica:ll -n build-osaka_strm.log 
  -rw-rw-r--  1 10000 213 3707 May 25 10:30 build-osaka_strm.log
  

  Yet from build-win1:

  Build-win1:id
  uid=1151(defaria) gid=213(hp-cc)
  groups=0(root),544(Administrators),545(Users),039186($22DV00-RLFVIG81PFDK),999028($4R5U00-TTIJJ61TLD20),1156949(CC-ADMIN),213hp-cc),525633(ESSDCC),1156950(HP-CC),457005(ORG-NSS-SOFTWARE),1143096(SERVICE-RS-MSRA)
  
  

  I seem to be in the hp-cc group (213) on build-win1 and should have write permissions on that file.

  More concerning is the following:
  

  Unable to open
  \\thecity\releaseArchive\osaka\install\build-CCInstaller-osaka-20060329_0928\install/tools/patches/active/patches/patches.info.new
  for write: Permission denied
  No file written
  

  Even though:

  Thecity:cd
  /mnt/hp/releaseArchive/osaka/install/build-CCInstaller-osaka-20060329_0928/install/tools/patches/active/patches
  
  Thecity:ll patches.info 
  -rw-rw-rw-  1 build hp-cc 362 Mar 29 12:41 patches.info
  

• Running of osaka_strm_build.sh calls osaka_Domino_build.sh which does:

  verbose "Performing build..."
  mkdir Domino
  cp -rp platform/mailserver/domino/* Domino
  

  But this fails because in the view it is using (rollet_osaka_snap_int) there is no platform/mailserver/domino to copy from. I double checked this by running osaka_Domino_build.bat and it fails the same way. I think the solution is to simply ct update -add platform/mailserver/domino to the rollet_osaka_snap_int view.

• I've converted osaka_strm_build_D2R.bat -> osaka_strm_build_D2R.sh. It used to call /home/build/buildD2R.sh which simply did a perl streamBuild.pl so I eliminated that call and just put the call to perl in osaka_strm_build_D2R.sh. Part of this script attempts to ct update -overwrite d2riss however as we seen before d2riss is not in Clearcase. Again I double checked this by running the .bat file and it likewise fails. There is much more work to be done here as there were a ton of set statements and a very long and odd PATH setting so I'm not sure if my translation worked yet.

Posted by Andrew DeFaria at 05:58 PM | Permalink

I don't see how my tinkering with things could have messed up IPSEC. I did mess with the Local Computer Polices and Default Domain Policies attempting to add rights to the local user sons-sc-cc\sshd_server. I'm not sure how this effects IPSEC. This user is used by the sshd and inetd Cygwin services. Since these services need to "switch user" to the user ssh'ing or rsh'ing they need some elevated rights. Prior to 2003 Server the Local System Account (known as SYSTEM in Cygwin) had enough rights to do this. But with 2003 Server Microsoft lessened the rights that the Local System Account has. Cygwin's answer to this is to create a new user ID, sshd_server, and assign it the necessary rights to be able to switch user. Then this user would only be used for Cygwin services. In fact they have a script /bin/ssh-host-config which creates a local sshd_server user for you (as well as sets up the hosts ssh key and adds a service for sshd). However even after running that script ssh was not working.

The inetd service, which is the super server, provides services like rsh, telnet, ftp, etc. that also needs to switch user. Similarly with cron. You see the service is running as an executable under some users credentials (normally the SYSTEM user) and needs to become the requesting user. So I was trying to run these services by using the local sshd_server user that ssh-host-config creates and adds the necessary rights to switch user.

Since I was having troubles I was removing the locally created sshd_server user and having the ssh-host-config script recreate it. At one time I decided to run mmc and add the Group Policy snap in then go under Local Computer Policy: Computer Configuration: Windows Settings: Local Policies: User Rights Assignment and make that the local sshd_server user had the following rights:

1. Create a token object
2. Logon as a service
3. Replace a process level token
4. Increase Quota

One time, when adding the Group Policy Object Editor I decided to click on the Browse button and saw a Default Domain Policy and thought perhaps there was something in there overriding the Local Computer Policy. So I added that and again I made sure that sons-sc-cc\sshd_server had the above rights. I have put a copy of this mmc polices thing under C:\Polcies.msc in case Ron wants to look at it. I notice a number of SIDs in there, probably from my past creations and recreations of sons-sc-cc\sshd_server.

As I don't know what Ron did to get sons-sc-cc running again and as I see the local sshd_server as having only deny network login right I don't want to mess with anything. Right now rsh is still broken as inetd cannot switch user since it's using the sshd_server user and that user doesn't have enough rights.

Here's an exert from /usr/share/doc/Cygwin/openssh.README:

Important note for Windows 2003 Server users:

2003 Server has a funny new feature. When starting services under SYSTEM account, these services have nearly all user rights which SYSTEM holds... except for the "Create a token object" right, which is needed to allow public key authentication :-(

There's no way around this, except for creating a substitute account which has the appropriate privileges. Basically, this account should be member of the administrators group, plus it should have the following user rights:

1. Create a token object
2. Logon as a service
3. Replace a process level token
4. Increase Quota

The ssh-host-config script asks you, if it should create such an account, called "sshd_server". If you say "no" here, you're on your own. Please follow the instruction in ssh-host-config exactly if possible. Note that ssh-user-config sets the permissions on 2003 Server machines dependent of whether a sshd_server account exists or not.

Posted by Andrew DeFaria at 10:28 PM | Permalink

Regarding sons-sc-cc's crontab

I copied (and modified) the crontab for ccadmin@sons-clearcase -> ccadmin@sons-sc-cc changing sons-clearcase -> sons-sc-cc where appropriate. I had also commented out the ctmerge lines. We don't want to be doing ctmerges on sons-sc-cc just yet as sons-sc-cc doesn't yet master the branches that sons-clearcase does. If we did we'd have to establish a whole new branching structure like sc_1.0, etc. That would be doable but since sons-clearcase will be decommissioned I think it's cleaner to simple transfer mastership of everything over to sons-sc-cc. However before we can do that we need to get all the users off of sons-clearcase and using sons-sc-cc.

The only thing currently left in ccadmin's crontab on sons-sc-cc are:

00 02 * * * //sons-sc-cc/Views/official/Tools/bin/update_view -q official

00 00 * * * /usr/bin/find /tmp -type f -ctime +3 -exec rm -f {} \;

01 03 * * Sat //sons-sc-cc/Views/official/Tools/adm/bin/dum -s /dev/d/Views/* > /tmp/viewspace.log

These three things are mostly administrative, updating the view official, cleaning out old files in /tmp and doing a report of the size of the snapshot views -> /tmp/viewspace.log. Sons-clearcase also does that. The report has the disk usage of all of the folders in the Views share sorted by the biggest users.

Regarding checking in files/directories and cleaning views

As we move off of sons-clearcase and onto sons-sc-cc you should instruct the engineers in Santa Clara to do the following:

1. Check in all files and directories in your views
2. Check for any view private files you may wish to keep and save them somewhere
3. For each view you own and that you have cleaned out with #1 and #2 above, remove it. If you have problems then see the Clearcase Administrator

This goes for both snapshot views on the server (//sons-clearcase/Views) and local snapshot views on your desktop as well as any dynamic views you may have. It is not required that you remove local snapshot views however by removing the snapshot views on the server (//sons-clearcase/Views) we get a better picture of what views remain.

When all of your elements have been successfully checked in, all your view private files saved and all of your views removed you can switch to the new server by executing /view/official/Tools/bin/switch-sons-sc-cc at a Cygwin prompt. Close any of your Clearcase tools (e.g. Clearcase Explorer) first. Then start Clearcase Explorere and create new views and proceed with your work. If you saved view private files and wish to restore them then copy them from the saved locations into a newly created view.

What are view private files and why are they possibly important?

We all know what file/directory elements are and what a checked out file/directory is. View private files are files in your view that have not been added to source control. These can be the result of a compile (e.g. foo.o AKA a derived object because they have been derived and can be re-derived if necessary) or they can be simply a file that you created (e.g. file.txt). Sometimes you may create a view private file that you want to keep, possibly wanting to add it to source control at some later time. You generally don't need to worry about derived objects (IOW you can regenerate foo.o provided foo.c is in source control). But if you say created some new files that you use in the build but have not yet added them to source control, but intend to, then you obviously want to keep them. Perhaps it's time to add them to source control...

To list view private files use:

$ ct lsco

In your view.

Checking what's still checked out

Most people know how to Find Checkout. But to find all checkouts easily from a command line you can do:

$ ct lsco -avobs

Again this needs to be done in a view context (cd /view/official for example). This way you can see what's left to check in for everybody. We're aiming for 0!

Posted by Andrew DeFaria at 09:56 PM | Permalink

• Backed up production database again
• Shutdown Clearquest database and cqd on sons-clearcase
• Changed several scripts that used to refer to sons-clearcase to refer to sons-sc-cc. For example, cqc now defaults to sons-sc-cc to find the server instead of sons-clearcase
• Added setting of Clearquest connection parameters via a reg file to upgrade clearquset
• Checked in changes to scripts as well as added the switch-sons-clearcase and switch-sons-sc-cc scripts and their assoicated reg files into Clearcase
• Fixed up sons-cc, its albd_server process was stuck, and made sure that the changes multisited over to sons-cc.
• Checked out to insure that cqc on sons-cc contacts and works with sons-sc-cc

Time Spent: 3 hours

Well I have migrated the Clearquest database to sons-sc-cc and I believe I've tidied up the necessary scripts and other things so that this transition will be relatively smooth.

You should announce that you are now on a new Clearquest server. Users using the web interface (i.e. China) should point there browsers to http://sons-sc-cc/cqweb/login and login as usual. Although the new Clearquest web is a total redesign most of it is pretty intuitive and I don't think current Clearquest web users will require much hand holding.

Meantime I made sure that cqd is working and cqc can talk to it. Additionally I had to get sons-cc to work. Seems it was kinda stuck with the albd_server process running at 50% and multisite getting stuck. I believe I cleared that up. cqc appears to work from sons-cc now.

For any Clearquest client users (worse case they can use Clearquest web) they need to update to the new version of the Clearquest Client. They should be able to simply do "upgrade clearquest" in a Cygwin shell. If, however, there are problems with that here would be roughly the equivalent steps from Windows:

• Uninstall the old version of Clearquest (Control Panel: Add/Remove Programs: Rational Clearquest... etc.) You may need to reboot.
• Install new version of Clearquest. Browse to \\sons-sc-cc\Rational\Clearquest and double click on the Install Clearquest shortcut. This shortcut runs setup with the sitedefs.dat as a parameter. Follow instructions - reboot when requested.
• Start: Programs: Rational Software: Rational Clearquest:Clearquest Maintenance Tool.
• Select File: Import Profile.
• Click on the ellipsis button (...) and browse to \\sons-sc-cc\Clearquest (Note this is not the same as \\sons-sc-cc\Rational\Clearquest where the installation lives) and select the 2003.06.00.ini file. This connection profile has all of the connection parameters for the new database.
• Select Next then Finish

That should set up your connection to the database. Then try the Clearquest Client.

I will try to make myself available Monday in case there are problems.

Posted by Andrew DeFaria at 05:37 PM | Permalink

• Created trlock to transfer locks from old to new server

Time Spent: 1 Hour

Posted by Andrew DeFaria at 11:42 PM | Permalink

I feel we've reached a time where we can migrate users over to the new system. I've worked out how to migrate the CQ database, build issues for both the old (salira) and new (salira2) vobs, multisiting is working, by and large scripts work as well as CQD. The following is a plan for migrating Salira to the new server. Please let me know your comments and concerns.

Time Spent: 2 hours

1.0 Migrating Clearquest

This migration is a little more critical because not only does the database need to be remigrated since bugs have probably been added since my test migration but clients must also update to the latest version of Clearquest in order to access the new database. Also, CQD (or rather CQD clients) need to ask the new server for information. Luckily the later involves a simply change to a library however timing remains important. This has to be done in one swoop.

The following needs to take place:

1. Shutdown access to the database on sons-clearcase
2. Backup the database
3. Use installutil to copy the user db to the new server.
   
4. Export and import user accounts (since you've made changes)
5. Convert the database to SQL Anywhere 8.0 format
6. Test new database
7. Switch cqc.pm library to default to new server
8. Have client upgrade to the new Clearquest

I suspect that the database conversion will take at most 4 hours so doing this on say Saturday or Sunday might be a good idea. As for desktop installations the users should be able to simply type "upgrade clearquest" in a Cygwin window. It will prompt the user to proceed then it will uninstall the old Clearquest, install the new Clearquest and reboot the users desktop. The user will then need to import a Clearquest connection profile and they should be set.

Bear in mind that is anything goes wrong the users should be able to use the CQWeb interface by pointing to http://sons-sc-cc/cqweb/login. China users will need to be made aware of the new server location in the URL but should otherwise be unaffected as I doubt that China users use the Clearquest client directly (way too slow!).

2.0 Migrating Clearcase

In general users need to stop using views that are sons-clearcase oriented. By that I mean they need to start checking in their work and generally getting their views to the state where they can remove them. Then they need to switch-<server> to sons-sc-cc and create new views there. As we are currently multisiting between sons-clearcase and sons-sc-cc they can migrate at their leisure however bear in mind that sons-clearcase should be going away. See also Mastership Issues below.

Users don't necessarily need to upgrade their Clearcase software as the old software will work with the new server, however you should urge them to do so. Again I plan on updating the upgrade command to likewise allow desktop users to simply type "upgrade clearcase".

One thing that users should do is change the symlink for /etc/site_parms from /view/official/Tools/adm/etc/site_parms.US to /view/official/Tools/adm/etc/site_parms.SC. Most of my scripts rely on this site_parms file to set environment variables to describe particulars about the site. This can be done with the following:

$ rm /etc/site_parms
$ ln -s /view/official/Tools/adm/etc/site_arms.SC /etc/site_parms

3.0 Decommissioning sons-clearcase

Sons-clearcase has served you many years as the hub of both Clearcase/Clearquest services as well as a build machine. It has performed many duties and has done them well. Decommissioning such a server so near and dear to Salira's heart will take some doing, some finesse. The overriding concern is to untangle sons-clearcase from day to day operations and assign those duties over to sons-sc-cc in a proper manner before decommissioning sons-clearcase. Than have a nice going away party!

As users migrate to using the new server and once all elements are checked in and views are no longer needed they should be removed. We need to get to a place where all Clearcase activity on sons-clearcase is stopped, checked in and removed.

There are some ancillary functions that sons-clearcase is still performing that you may be unaware of such as licensing. These too will need to be transfered to the new server.

Finally sons-clearcase can be powered down and if there are any things we've missed it could be easily powered back up if need be.

3.1 Lock issues

One thing that is not multisited is locks on label/branch/etc types. Salira relies on these logs. I will need to write a script to evaluate the locks in the vobs on sons-clearcase and to issue the same locks on sons-sc-cc. I will work on this this week.

3.2 Mastership Issues

With multisited vobs comes issues of mastership. As you know a branch can only be mastered by one replica. The current vobs' main branch is mastered on sons-clearcase as are branches such as REL_1.0, etc. Mastership should be transfered to sons-sc-cc before sons-clearcase gets decommissioned.

There is a command called chmaster that luckily has a -all option to change the mastership of all objects to the new server. But it wants to have everything checked in and nothing locked.

3.3 Replica removal

With the vobs replicated to sons-sc-cc and all Clearcase views removed and mastership moved to sons-sc-cc, the final act would be to remove the US replica of the vobs from the vob family. This way sons-sc-cc and sons-cc in China would no longer attempt to update the US replicate on sons-clearcase (The replica on sons-sc-cc is called the SantaClara replica and remains in the vob family hosted by sons-sc-cc).

4.0 License Issues

Sons-clearcase also served as a license server for both Clearcase and Clearquest. Salira will need to contact IBM/Rational in order to obtain new licenses for these as they are host based. This can be delayed however it will need to be done before sons-clearcase can be decommissioned.

5.0 Miscellaneous Issues

There are several scripts which use some magic to determine where to find Perl library code (here or in China). That magic hard codes sons-clearcase as the server here to use. That'll need to change to use sons-sc-cc before a complete shutdown of sons-clearcase is doable.

Posted by Andrew DeFaria at 02:42 PM | Permalink