Status for Andrew DeFaria

• Tried to figure out why -stgloc is not working in Richmond nor why I can't seem to create a view in a share that is in an area where .stgloc exists
• Worked with Vin Ton and Rob Swindell regarding a problem with CQ Web and Code Pages
• Worked on other code page issues
• Added CharacterSetValidation package to PQA

Stgloc problems

Why does the following happen:

Local:mkdir \\\\fs-rmna-01\\viewstoreQA\\adefaria_test
Local:rmdir \\\\fs-rmna-01\\viewstoreQA\\adefaria_test
Local:ct mkview -tag adefaria_test \\\\fs-rmna-01\\viewstoreQA\adefaria_test
cleartool: Error: Attempt to get location information on "\\fs-rmna-01\viewstoreQA\adefaria_test"
failed: Invalid argument.
Local:ct mkview -tag adefaria_test \\\\ltsjca-adefaria\\views\\adefaria_test
Created view.
Host-local path: ltsjca-adefaria:C:\Views\adefaria_test
Global path:     \\ltsjca-adefaria\views\adefaria_test

I suspect that the problem is that this area is on a filer so Clearcase cannot determine where the view_server process should run. I believe that the only way Clearcase can figure out which server should serve the view_server process for this area is via the stgloc and that the stgloc for this area is not properly configured. Attempting to create views either through the command liine or the GUI using -stgloc viewstore or viewstoreQA also fail.

Posted by Andrew DeFaria at 04:48 PM | Permalink

• Updated create_dev_snapview to support -tag, -dir and -load_rules.

Posted by Andrew DeFaria at 05:10 PM | Permalink

• Investigating issues with bin_merge
• Discussed with Andrew about extending script to create snapshot views
• Worked with two people who have had issues with accessing new PQA Clearquest database

bin_merge

Spoke with Shivdutt about this. We decided that bin_merge should copy the binary file from the branch to the current view's checkedout copy and draw a merge arrow. Also, it should just draw an merge arrow if the user elects to use the from branch version. This mimicks a parent/child relationship that is assumed here

Still need to look into the Rebase to parent operation.

Posted by Andrew DeFaria at 06:02 PM | Permalink

• Continued to help several users with connecting to the new Controller Clearquest Database for Vinh's users
• Added Version Tree button to the bin_merge prompt dialog box.
• Resolved problem with forking from PerlTk

Forking in PerlTk

The issue here is that calling fork(2) in general on Windows just doesn't work well. Clearcase's Perl is based on ActiveState Perl and ActiveState Perl is not the best implementation of Perl. Cygwin's Perl is by far a much better implementation but Cygwin's Perl works from a true Posix environment which is specifically what Cygwin is all about. However Cygwin's Perl does not support PerlTk (I posted on Cygwin's mailing list about this - it would be great if Cygwin would finally support PerlTk) and Cygwin would need to be present on all Windows systems.

In general, while fork(2) does work under ActiveState is does not work if you have Tk objects created. Chris had suggested to use "start <program>" which is fine - for Windows - but would fail in Unix. I like to wrote my code such that it works on both Windows and Unix. So I tried in vain to get fork to work. However, ActiveState's fork(2) call is horribly broken so I had to code around it like this:

sub VersionTree {
  my $file = shift;

  my $cmd =  "cleartool lsvtree -graphical $file";

  if ($^O =~ /mswin|cygwin/i) {
    system "start /b $cmd";
  } else {
    my $pid = fork;

    return if $pid;

    system $cmd;
    exit;
  } # if
} # VersionTree

Posted by Andrew DeFaria at 01:59 PM | Permalink

• Worked with Vinh's group and adjusting to life with the new Controller CQ Database. Wrote instructions for how to unsubscribe to some Broadcom email distribution lists
• Documented how to install the new Clearquest Client Software
• Coded stripmime filter for GNATS.

Stripmime

The script described above basically takes as input an email message. It then extracts the header portion and the body portion. It uses some CPAN modules (MIME::Base64 and MIME::Tools as well as MIME::Parser) to analyze the message and obtain the message "parts". Some transformations are performed on the header information and then it is written to stdout. For the plaintext part (if present) it will again write out the message to stdout Other, non-textual parts (AKA attachments) are written to a subdirectory based on the message name/number and a note is included to stdout indicating were to find the attachment.

The stripemime script placed in /tools/gnats/4.0/bin does the same thing except that attachments are disregarded and no mention of them are written to stdout. However there is an implicit assumption above that there does exist at least one plaintext version of the mail message. This need not be the case. I know, for example, when using Thunderbird I have the option of sending email as HTML, plain text or both. I tend to send both, which are epresented in the raw email as MIME multipart with the first part being a plaintext version of the HTML, which is sent as the second multipart as HTML. However, if I were to select HTML Only I would not get the plaintext part. Stripemime, as currently coded would generate a blank email message.

Also, on gnats-irva-3, could my shell be set to bash? I've tried to set it but I get permission denied. Also, I don't appear to have a home directory set to /home/adefaria for the gadefaria user?

Finally, MIME::Parser is not installed on gnats-irva-3:

gnats4@gnats-irva-3[+1035] bin/stripmime
Can't locate MIME/Parser.pm in @INC (@INC contains:
/tools/perl/5.004_04/SunOS/lib/sun4-solaris/5.00404
/tools/perl/5.004_04/SunOS/lib
/tools/perl/5.004_04/SunOS/lib/site_perl/sun4-solaris
/tools/perl/5.004_04/SunOS/lib/site_perl .) at bin/stripmime line 47.
BEGIN failed--compilation aborted at bin/stripmime line 47.

I had managed to install MIME::Parser on my Cygwin system using "perl -MCPAN -e 'install MIME::Parser'"

Posted by Andrew DeFaria at 05:38 PM | Permalink

• Worked with Vinh on small problem with Gating_Item_SW.
• Backed up database to \\p4test\Cont as CQ_Controller_Prod.db.backup
• Copied backup up to \\pcirvd-dshah\CQ

Gating_Item_SW

Vinh noticed an abnormality whereby some Gating_Item_SW items were not correctly transfered. I'm not sure how pqamerge messed this up as it never changes the value of the data, just the field name. We can look at this later, after the merge is completed. Meantime VInh use his SQL prowness to fix the problem.

Posted by Andrew DeFaria at 02:47 PM | Permalink

This blog entry is to track the PQA merge process. See extended entry for full details...

• Received backups from Dave Shah
• In Clearquest designer I did Database: Delete Database for Cont and ContT
• In Enterprise Manager:
• Deleted 5 databases: CQ_Controller_Test, CQ_Controller_Prod, CQSchema03, CQ_TO_Prod and CQ2001AProd
• Created those 5 databases anew
• Restored CQSchema03, CQ_TO_Prod and CQ2001AProd
• Changed server field in CQSchema02: master_dbs from production server to p4test
• Ran SIDMap on all 5 databases
• Made sure that the user CQ03_Admin was added to CQSchema03, CQ_Controller_Test and CQ_Controller_Prod as a DB owner
• Back in Clearquest Designer I found that Cont and ContT were not deleted in the newly restored schema so I deleted ContT
• Next I created a new database for ContT
• Tried to delete Cont but received an error: Invalid object name "dbglobal". Worked around this problem but setting is_deleted to 1 using Enterprise Manager
• Created new Cont database
• Imported userinfo.txt (user/group info given to me by Vinh) using Clearquest Designer: User Administration. In looking at the subscriptions I see that groups appear are properly subscribed to databases but the users, particularly admin was not subscribed to any database! I will attempt to apply these user/groups and see if I can get into the db using Clearquest. Nope, as I suspected admin cannot login. Subscribing admin to all databases...
• Upgraded Cont and ContT with imported user/group info. By accident I closed Clearquest Designer and now I can no longer login as admin! Ended up hacking the encrypted password from userinfo.txt back into CQSchema03:master_users:admin:encrypted password! I can now run Clearquest Designer. Subscribed admin to all databases. Also changed my user, adefaria, to no longer attempt to authenticate via LDAP (that never worked anyway) and to be on par with admin (be a superuser, subscribe to all databases and groups, etc.). I can now access Cont via Clearquest client.
• Started pqamerge...
• pqamerge finishes in 7 hours 2 minutes 5 seconds
• Running check_attachments just to be sure
• A quick spot check doesn't show any abnormalities. Vinh can check the data more thoroughly

Posted by Andrew DeFaria at 08:21 AM | Permalink

• Looked into create_dev_snapview.pl.
• Tested speeds of FTP vs. SMB

create_dev_snapview.pl

From: Andrew Feltham
Sent: Tuesday, November 15, 2005 3:58 PM
To: Shivdutt Jha
Subject: creating snapshot views

Hi Shivdutt,

so I've tried to use Naga's tool for creating a snapshot view of a particular stream using a variety of incremental baselines.

I've done this using a file generated by Phil's nightly builds (see attached) which lists the specific baseline which was current for each vob.

First of all, it does not seem to work (see image below). I'm not sure I'm handling the composite baseline label properly

Secondly, I do find the tool a bit awkward as it expects me to input several incremental baselines for vobs which are not included in my load rules.

It would be nice if I could specify the vobs on the cmd line, and only provide those labels.

Actually the script is working as designed. The arguments for create_dev_snapview.pl are:

create_dev_snapview.pl <project_name> [ baseline_selector ]

project_name is mandatory but baseline_selector is optional. If you don't specify the baseline_selector then it appears that the script takes all the recommended baselines for all components in the project. Or you can specify a list of baselines if you want different ones. The script checks the number of baselines you specified and compares them to the number of modifiable components in the project. These two numbers must agree.

After painstakingly typing in the command line (wish you would have posted the command line in text! :-( ) and running it through Perl's debugger I came to the realization that you typed in 11 (very long) baselines and there are only 10 modifiable components in the ldx_2.3 project.

In thinking of how to do this better I can think of two methods:

• Assuming the buildconfig.txt file is a standard file and contains all the information needed, add a -f <buildconfig> option to create_dev_snapview.pl. When present the script would open and parse that file, extracting out the required information. This would help cut down on typing and errors.
• Modify the script to present a series of dialog boxes prompting for the baseline to use for each component in the project. Note that in this instance this would mean a total of 10 dialog box prompts!

Either of these options can be added to the existing script via options that cause it to operate in a different way if present or to operate like it used to operate if those options are not present.

(Question: When you type "perl" at your command line, what Perl are you running? Because Perl isn't normally present on Windows. Normally people using Clearcase run ccperl...).

FTP vs. SMB

Well this runs contrary to my previous experience but it turns out that a regular file copy using SMB shares is faster than FTP. I tested copying the old 2.7 Gig CQProd2001A database to my desktop, once with FTP and once with SMB. SMB won! Which is surprising to me because last time I tested something like this FTP was up to 10 times faster:

Local:time ftpget -u vobadm -p4sjadm p4test . "/dev/d/CQBackupsfromIrvine/0916/CQProd2001A_db_200509160213.BAK"
CQProd2001A_db_200509160213.BAK           2.66 GB    60.35 kB/s

real               769m10.858s
user               1m29.418s
sys                4m36.387s

Versus:

Local:time cp //p4test/CQBackupsfromIrvine/0916/CQProd2001A_db_200509160213.BAK /tmp

real              532m7.457s
user              0m16.714s
sys               1m11.362s

Posted by Andrew DeFaria at 12:39 PM | Permalink

• Continued work on clearprompt replacement in PerlTk.

Posted by Andrew DeFaria at 09:13 AM | Permalink

• Experimented with PerlTK in an effort to come up with a better Clearprompt for bin_merge
• Vinh found a bug in pqamerge where Prod: Open_Close_Status was not getting transferred to Cont: Active_Deferred_Status

Posted by Andrew DeFaria at 05:19 PM | Permalink

• Identified a few more bugs with pqamerge
• Changed to handle Open_Close_Status -> Active_Deferred_Status for Prod.
• Changed to correctly handle Gating_Item_SW. This is set from GatingItem from both TO and Prod. Gating_Item_HW is a new field and is set to "No".
• Worked with Vinh finalizing steps for pqamerge

Posted by Andrew DeFaria at 09:15 AM | Permalink

• Added Exit Sub/Function to several validation and initialization functions in order to allow TransferState to work
• Fixed some bugs with Fixed_In_SW_Version and OS values of "Novell 6 Pack Beta 3 ". Not sure how this got by before with the trailing space but it raised it's ugly head in TransferStates
• Change pqamerge to treat the Submit state like the Assigned state
• With implementation of ID equality, I have to constantly regenerate the database from scratch. Experienced some problems with this and had to change the schema by allowing admin to submit. This may have been a different problem in that I now need to subscribe all users and groups to Cont database and Upgrade the database each time I create it a new
• Transitioning through states have reveal some fields that where not initially tranfered from the TO database. The fields missed were ResolvedBy, ResolveNote and VerifyNote

Additional Schema Action Hook Changes

In order to set State properly pqamerge needs to transition through states to get to the desired end state. For eample, many defects in Prod are in the Closed state. But when pqamerge first creates the defect in Cont it will be in the Assigned state. In order to get to the Closed state pqamerge need to apply the Resolve, Verify and Close actions. As such it triggers state change action hook code.

For example, in order to transition to the Verified state the Verify action must be applied. When the Verify action happens the action hooks for Verify are run. Part of what they do is initalize owner to blank. It is expected that the user exectuting the Verify action will fill that in. But that messes us up since we are not a person and we already have the "correct" data. Investigating this further reveals other places where, for example, the current date is put into a field such as Resolve_Date and calculations are made in other fields such TimeFromSubmitToResolve. We don't want any of that happening!

To resolve these issues the following Action Hooks have an Exit Sub or Exit Function placed at the beginning of the subroutine or function call so as to avoid the incorrect updating of data fields and so that email is not send out (the other thing many of these Action Hooks do):

• Submit: Validation
• Assign: Notification
• Resolve: Initialization; Validation; Notification
• Verify: Initialization; Validation; Notification
• Re-Open: Notification
• Close: Initialization; Notification
• Modify: Notification
• Unassign: Notification
• DoesNotVerify: Initialization; Validation; Notification
• VerifiedPendingCustVerify: Notification
• CustomerVerified: Notification
• Data_Pending: Notification

I should have just done all of them or perhaps tried to change the Actions table from use Basic script to script None but the above set seems to be working.

Data Issue

Ran the merge and now it's taking 6 hours 54 minutes 39 seconds. This is due to running through the various states to obtain the appropriate state and "burning" IDs so that the IDs match. One remaining problem that I don't know how to fix: Defect Prod00012546 is in the closed state yet lacks any VerifyNote. When transfered to Cont pqamerge tries to go through the states and gets stuck trying to transition this defect from Resolved -> Verify due to the lack of a VerifyNote. I do not know how this happened in the Prod database. The only thing I can think of is that somebody modified Prod00012546 after it passed the Verified state blanking out VerifyNote.

I'm running check_attachments now and I expect that to result is 0 differences in attachments.

vobadm@P4TEST /dev/d/PQA
$ cqperl W:/it_scm/adm/cq/check_attachments -v
Grand total (old): 2955822684
Grand total (new): 2955822684

Posted by Andrew DeFaria at 12:17 PM | Permalink

• Implemented TransferStates
• Found final problem with attachments and fixed it
• Implemented ID number equality

Posted by Andrew DeFaria at 01:54 PM | Permalink

• Fixed bug in pqamerge that caused some attachments to not transfer

I think I've figured out where that missing data went to. As you know the total of the attachment sizes of the old databases compared to the new database was still off but a relatively substantial amount (261,285,366 bytes - see https://defaria.com/blogs/Status/archives/000471.html#more).

I changed check_attachments to help me find where the missing data was. The basic idea was to read all Cont records, total up the attachment size, then use Cont: old_id to locate the old record and total it. With those two figures I could find which records didn't convert correctly.

Turns out:

 # At this point we don't have any info about whether we are
 # coming from Prod or TO, however, there are the following fields:
 #
 #          TO               Prod              Cont
 # ----------------------- ----------------------- ----------------
 # Attachments        Attachments        Attachments
 # AttachmentsBRCM        AttachmentBRCM        AttachmentsBRCM
 #
 # You may notice that Prod: AttachmentBRCM is missing the "s".
 # Therefore:
 $field_name = "AttachmentsBRCM" if $field_name eq "AttachmentBRCM";

Remerging and will run check_attachments again.

Posted by Andrew DeFaria at 11:55 AM | Permalink

• Investigated how to remove databases from a CQ Schema
• Created template files for Clearquest Web Login page to swap in for Phase I and Phase II onto the production web server (under C:\Temp\PQA)
• Discussed the email link problem with Rational. While they can't say definitively what is supported they have stated that the old email link URL will not work in the new schema

Databases contained in a Clearquest Schema Database

A Clearquest connection profile connects to a Clearquest schema database. This schema database defines many things, one of which is which user databases this schema database covers. So with the old 2001 schema all of Prod, TO and NAS is defined in there. The Clearquest Designer has Delete Database and Undelete Database to remove and re-add databases to the schema. This is a misnomer. The schema database has a table, master_dbs, which lists all the user databases that this schema database knows about. What is really happening when you Delete Database is simply that Clearquest Designer is toggling the is_deleted field in master_dbs for this database to 1 meaning it's "deleted". Undelete database merely toggles it back to 0. Being marked deleted means that it will no longer show up as a database to select for Clearquest Windows Clients and the web server. The web server, however, needs to be restarted to notice the change.

Another Email URL link issue

With the concept of having multiple Schema repositories and database you are right to be concern about the user being confused when clicking on an email link. Normally the email link is of the format:

http://<server>/cqweb/url/default.asp?id=<id>

If the user is logged into CQ Web then they will go directly to the detail defect. If not they will make a brief stop at the login screen. With multiple Schema repositories the user will have to select the proper schema repository and database during login. However you can add on additional parameters to set the default schema repository and database like so:

http://<server>/cqweb/url/default.asp?id=<id>&dbset=<dbset>&db=<db>

For example:

http://pcsjca-ccrmt03/cqweb/url/default.asp?id=Cont00009460&dbset=2005.02.00&db=Cont

Posted by Andrew DeFaria at 03:52 PM | Permalink

• Installed Clearquest 2001A.04.00 and verified that it can access Cont
• Figured out how to modify the login page (Clearquest/Web/logon/default_content.asp) and created default_content1.asp and default_content2.asp
• Wrote How to update my Clearquest Profile to access the new Cont database

  Posted by Andrew DeFaria at 07:54 PM | Permalink

• Enhanced check_attachments to log better

Well it's better and didn't take as long as I thought. This time it took 4 hours 34 minutes 49 seconds. Not too bad. And the attachments size is closer. The total attachments for Cont is still < the sum of the attachments from Prod and TO:

I'm not sure how to account for that difference.

Posted by Andrew DeFaria at 10:33 AM | Permalink

• Worked out a plan for how to perform the PQA Merge this weekend
• Concerned about the size discrepancy in the database I wrote a little utility, check_attachments, to total up the size of all attachments in all the databases. This turned up a bug in pqamerge which I fixed. Reperforming pqamerge
• There is a problem with using clearprompt to prompt for list input for bin_merge - You can only have one line of prompting text and only 50 characters in that line. It's gonna be hard to describe a binary merge situation in 50 characters or less. Investigated making a PerlTk list dialog. In theory it can be done as ccperl does support PerlTk.
• Discussed with Naga the email link issue with PQA

TransferAttachments

I knew there was a good reason why this was eating at me. There is indeed a bug. My attachments transfer routine was not getting called at all for Prod! I wrote a small Perl script to check this and it yielded:

$ cqperl check_attachments
Totaling attachments in TO...
Totaling attachments in Prod...
Totaling attachments in Cont...
Total attachment size for TO = 272253137
Total attachment size for Prod = 2683569547
Total attachment size for Cont = 272253137

Notice that the size of the attachments for TO matches the size of attachments for Cont! IOW no Prod attachments got transferred at all!

Turns out I was only calling TransferAttachments when the field name was AttachmentsBRCM (and TransferAttachments then did all the attachments - both Attachments and AttachmentsBRCM). However, in Prod the field is named AttachmentBRCM - note the singularity here! I tell ya my eyes are going.

I'm changing my code not be dependent on the field name and to just call TransferAttachments for each record. TransferAttachments doesn't need to know the field name - it just does all of them.

Unfortunately this means that I have to delete all of the records currently in the Cont database on p4test. And it also means that the approximate 4 hour running time will probably increase. However this does explain the discrepancy in the database sizes.

Out of Disk Space

Good job Andrew, another mystery is solved.

What is more telling is that we have 2.6 Gig of attachments!

In any event my merge failed with out of disk space. The pqamerge script uses the current directory to temporarily hold the attachments as the are transfered from one DB to the other. Turns out that Prod00010818 has two very large attachments, one 328485466 bytes and another 219095209 bytes. Unfortunately I was running in ~vobadm/My Documents and that disk filled. So another lesson learned - run pqamerge in a directory on a disk with lots of disk space! Alas this means I have to start this process over again (I'm trying to get a clean run and a timing of how long to expect the merge to run. pqamerge outputs how long it takes to do it work).

Posted by Andrew DeFaria at 06:54 PM | Permalink

• Almost code complete on bin_merge. Need to integrate this into UCMCustom
• Emailed IBM support regarding Clearquest Web URLs

Posted by Andrew DeFaria at 05:20 PM | Permalink

• Resolved a ticket regarding Clearquest Web (IIS) and not displaying properly. Documented under Clearquest: Getting the right JRE for Clearquest Web
• Started coding bin_merge

Posted by Andrew DeFaria at 06:49 PM | Permalink

• Changed enable_ldap to read from and save to an ldap_settings.cfg file
• Helped Ray with some Perl Regex's

Posted by Andrew DeFaria at 04:44 PM | Permalink

• Added a few users to Clearquest
• Set up IIS on pcsjca-ccrmt03
• Set up Clearquest Web on pcsjca-ccrmt03

Using the old web server to serve the new Cont database

Vinh has been wanting to have the old web server software set up alongside the new web server software for two reasons: 1) as a backup or fail over in case the PQA merge is not working out and 2) as a redirector for old style URLs (with old IDs) to the new server with new IDs. But having to wait for a new server to arrive will cause an unnecessary delay

The thought had been that perhaps the PQA merge could be done and simply use the old web server software to point over to the new database. One question was can the old web server software interact with the merged database seeing as the merged database was done under new Clearquest software. To this end I set up the Clearquest web server software on pcsjca-ccrmt03 and attempted to access the merged database on p4test. And It works! To log in go to http://pcsjca-ccrmt03/cqweb/login, set the Schema Repository to 2005.02.00 and log into the Cont database

Posted by Andrew DeFaria at 09:43 PM | Permalink