WebObjects: HP-UX Bug May Cause WebObjects Problems

1. GENERAL INFORMATION ABOUT THE BUG

This bug happens only on Hewlett Packard PA2.0 CPUs. For these CPUs, HP's library /usr/lib/libdce.sl has a bug in which a particular function corrupts a floating-point register. The function is pthread_yield(), part of the 'pthreads' package. Keep in mind that even if your code does not directly call this function, both Apple's frameworks and third-party libraries (such as Sybase's client library) may call it and trigger the bug. HP concurs that this is a bug in their library, and have assigned it bug number JAGAB25393. A patch from HP is available; please see section 4, below, for details.

It's difficult to predict the possible effects of the bug, so please read the sections below that describe how to test your system. Problems observed so far include:

• applications suddenly slow to a crawl, and sometimes recover
• calculations produce incorrect results

Keep in mind that this bug could cause a variety of other symptoms, including sporadic ones.

2. HOW CAN I TELL IF A PARTICULAR HP-UX MACHINE HAS THE BUG?

To see if your system (your HP CPU plus libraries) has the bug, run a test application:

1. Download and unpack the Register.tar file from Apple's ftp site

2. Open a shell window

3. Type: cd Register

4. Type: make

You can ignore the warnings from the compiler and linker

5. Type: ./Register

• If your system does not have the bug, the output from step 5 will be:
  10
  10
• If your system does have the bug, the output will look like:
  10
  <some number other than 10>

We recommend that you use this test app instead of looking at the CPU model (because if you receive a new library in the future, having a PA2.0 CPU will no longer cause problems).

3. HOW CAN I TELL IF A PARTICULAR APPLICATION MAY BE AFFECTED BY THE BUG?

If the above test shows that your system does have the bug, you should assume that all your applications are affected, and apply the patch or workaround described below to all applications. This will simplify your work, because it can be difficult to determine which applications are affected. The bug affects only applications that use HP's 'pthreads' package. This includes (at least) applications using 'cthreads' or Foundation's NSThread class, since both of those use 'pthreads'. Keep in mind that even though an application's code may not use threads directly, it may include libraries (such as Sybase's client library) that use threads. To see if your application uses threads, debug it under gdb, breakpoint in "pthread_create", and run the application.

If you do hit a breakpoint in "pthread_create", your application uses threads and risks encountering the bug. If you don't hit a breakpoint while running your application through typical operations, your application may not be at risk.

Example:

shell> gdb myApplication
GDB is free software and you are welcome to distribute copies of it
under certain conditions; type "show copying" to see the conditions.
There is absolutely no warranty for GDB; type "show warranty" for details.
GDB 4.15.3 (hppa1.1-hp-hpux10.10),
Copyright 1996 Free Software Foundation, Inc...
(gdb) break pthread_create
Breakpoint 1 at 0x5a58
(gdb) run
Starting program: /tmp_mnt/Net/custom/homes/ranch/morton/pthread
Breakpoint 1 at 0x7aa431f8

The following output reveals that your application uses threads, which tells you that it may encounter the bug.

Breakpoint 1, 0x7aa431f8 in pthread_create ()
(gdb) quit
The program is running. Quit anyway (and kill it)? (y or n) yes

4. HOW CAN I PATCH THE BUG?

HP has provided a patch for this problem. Apple recommends that all WebObjects users apply this patch. To get more information on the patch, point your Web browser to:

http://us-support.external.hp.com

If you don't have an account, click on the "New Users Register Now!" to start the registration process. Once you've logged in, click the "Individual Patches" link, then the "Retrieve a Specific Patch" link. The patch number for this patch is PHHS_16429. This will show a page that allows downloading the patch and also gives details as to which modules are affected. Complete installation instructions on the patch are available from the HP Web site.

The patch is about 28 MB. As with any patch, you should back up your system beforehand, and test everything afterwards.

After you install the patch, you do not need to reboot or rebuild applications affected by the register-trashing bug--your applications should begin to work immediately. (The patch also fixes some other things that do require a reboot, though.)

Apple recommends that all users install this HP patch. However, you should note this warning from HP's installation instructions:

IMPORTANT NOTE

Oracle 7.3.3 and above will need to be relinked. This patch contains changes to the libcma library.
Applications such as Oracle that use this library and are built static will need to be relinked.
For information and help for the rebuild please contact your Oracle support personnel.

Because some users may not be able to install the patch, Apple also provides an alternate workaround for this bug in Section 5, below.

5. HOW CAN I WORK AROUND THE BUG?

Apple recommends that you install HP's patch, as described in section 4. However, if you are unable for some reason to install the patch, you can work around the bug by:

• Installing an alternate library which HP ships, but does not install by default.
• Changing either your application's makefiles or WebObjects' makefiles to link with this library

You do not need to change how you build frameworks or libraries. The only thing that matters is how you link when you build executable files (applications and tools).

Because the workaround involves linking with a static library instead of a dynamic library, there are two downsides to implementing the workaround in every application:

• Your applications will use more memory. This is especially important if you run multiple copies of your application--each instance will use more memory. When linked with the static library, an executable grows by about 3 MB on disk, so the memory used while running will probably grow by a similar amount.
• Your applications will not automatically incorporate changes that may appear in the shared library when you patch or upgrade HP-UX.

Despite these potential problems, we recommend that, when in doubt, you should add the workaround to all apps. Of course, using the HP patch is preferable to either.

The alternate library in question is:

/opt/dce/lib/libdce.a

You will need to use HP's installer software, /usr/sbin/swinstall, to install the software. Details of the installation will depend on your system and configuration.

Once the library is installed, you can modify selected applications to link with it by:

1. If the source directory does not have a Makefile.preamble file, create a Makefile.preamble containing this one line:

LOCAL_LDFLAGS += -L/opt/dce/lib -ldce

If you already have a Makefile.preamble, add the line above to the end of it (unless the file already has a line defining LOCAL_LDFLAGS, in which case you should add the options "-L/opt/dce/lib -ldce" to the end of that line).

2. "Make clean" your application (since the above change will not be enough to cause "make" to rebuild) and then "make".

Or you can modify two system-wide make files that WebObjects uses to build WO applications and tools. This saves you time, and makes sure no project slips through the cracks, but it may have downsides to this approach, as discussed above. To modify the two system-wide make files:

1. Log in as or su to root and cd to the Register directory

2. Type the following commands to create the local makefiles (if they don't already exist) and patch the makefiles:

 mkdir -p /opt/Apple/Local/Developer/Makefiles/pb_makefiles
 touch /opt/Apple/Local/Developer/Makefiles/pb_makefiles/tool.make.preamble
 cp /opt/Apple/Developer/Makefiles/pb_makefiles/wo-preamble.make wo-preamble.make.bkp
 cp /opt/Apple/Local/Developer/Makefiles/pb_makefiles/tool.make.preamble tool.make.preamble.bkp
 cat preamble-additions >> /opt/Apple/Developer/Makefiles/pb_makefiles/wo-preamble.make
 cat preamble-additions >> /opt/Apple/Local/Developer/Makefiles/pb_makefiles/tool.make.preamble

3. "Make clean" any application you need to rebuild (since the above change will not be enough to cause "make" to rebuild) and then "make".

Try rebuilding the Register application to make sure you've installed and linked to the new library correctly. After making the application, use the "chatr" command to examine the application's attributes:

chatr Register

If you see a line in the output that looks like:

dynamic /usr/lib/libdce.1

then the application is still using the old library. If this line does not appear, the application is using the new library. Since the new library is statically linked, no entry will appear for it.

6. TECHNICAL DETAILS

The bug is that, on PA2.0 CPUs, floating point register 12 ($fr12) is not preserved across calls to pthread_yield(). This violates the HPPA calling conventions. Many system calls use pthread_yield(), so they may also fail to preserve the register. This bug exists in the current release of the library /usr/lib/libdce.sl. The non-shared ("archive") version of this library (optionally installed in /opt/dcelib/libdce.a) does not have the bug. We have no reason to believe that the sharing itself causes the bug.

The bug typically affects compiled code which is optimized, since optimizations causes floating-point registers to store local variables or intermediate values. Because many compiler allocate registers in ascending order, the bug will often have no effect on a method or function that declares fewer than five local floating-point values.

In summary, the bug occurs when:

• Your application runs on HP-UX 10.xx system.
• Your machine has a PA2.0 CPU.
• Your application uses HP's 'pthreads' package.
• You have five (?) or more floating-point variables in a function/method.
• You ask the compiler to optimize (that is, variables go in registers).
• You make certain system calls at a time when variables are meaningful.

This bug can trash floating-point registers at various points in your code, Apple's frameworks, and third-party libraries. One affect it has on the Foundation framework is especially problematic: At the time that NSRunLoop fires an NSTimer, $fr12 contains the current real time. If the code invoked by a timer makes any of several system calls, NSRunLoop sees an incorrect time. It may conclude that other NSTImers are "way behind", and begin firing them madly to catch up. This can cause applications to perform very slowly, sometimes recovering later.