4.0-7.1  - Fix to address RT kernel's renaming of the address_space.nrpages
           member to address_space.__nrpages.  Without the patch, "kmem -i"
           would fail with the error message "kmem: invalid structure member
           offset: address_space_nrpages".  (bwalle@suse.de)

         - For crash utility debug backtraces displayed in error conditions,
           the usage of __builtin_return_address() has been replaced with the
           backtrace() function.  This prevents crashes if the Makefile is
           modified to compile with -O2.  (bwalle@suse.de, anderson@redhat.com)

         - Fix for ia64 hypervisor backtraces when the entries in the cpu map
           are not contiguous.  (takebe_akio@jp.fujitsu.com)

         - Fix to make shell-escaped commands in a crash input file direct 
           their output properly.  Without the patch, if the output of an 
           input file was redirected to a file or pipe, the output of any 
           shell-escaped commands in the input file only went to stdout.  
           (anderson@redhat.com)

         - Fix to allow the usage of the "-i inputfile" command line option
           when operating from an init script.  Without the patch, the crash
           session would fail during initialization with the error message: 
           "crash: /dev/tty: No such device or address".  (anderson@redhat.com)
 
         - Fix for the "kmem -P <address>" option, where <address> is an 
           invalid physical address.  Without the patch, the command causes 
           a segmentation violation on an ia64; on other architectures an 
           unnecessary mem_map header is displayed prior to the error message. 
           (wency@cn.fujitso.com)

         - Fix for a potential endless cascade of SIGFPE exceptions during
           session initialization when a vmlinux and vmcore do not match,
           and a correct System.map or a non-debug vmlinux file is not supplied.
           Doing that is is allowable, but is certainly not recommended.  In 
           this case, and incorrect kernel HZ value of 0 was calculated and used
           for the initial "UPTIME:" display.  (anderson@redhat.com)

         - More gracefully handle a nonsensical "search -u <address>" command
           attempt on a kernel thread or any context with no user address space.
           Without the patch, the error message was related to a failed user 
           virtual address translation attempt; with the patch it now indicates: 
           "search: current context has no user address space".
           (anderson@redhat.com)

         - Reworked the "search" command for usage with the Xen Hypervisor. 
           When attempted on a Xen hypervisor, and depending upon the arguments 
           used, a segmentation violation, a nonsensical error message, or if 
           neither of the aforementioned, the command could appear to work but
           not necessarily find the search target value even though it was there
           in the specified memory range.  To address the various shortcomings,
           the following restrictions have been put in place for usage with the 
           Xen hypervisor:  

             (1) A starting virtual address must be supplied either symbolically
                 or with the "-s <address>" option.
             (2) The (nonsensical) "-u" option is no longer accepted. 
             (3) The "-k" option is no longer accepted. 
             (4) When cycling through virtual memory, as soon as an address 
                 cannot be read, the search will be quietly suspended.  To 
                 determine where the search was suspended, enter "set debug 1", 
                 and then re-run the command.

           (anderson@redhat.com)

         - Fix for initialization-time segmentation violation due to a module
           allocating and creating an exported symbol list outside of its own
           virtual address space, and then overwriting its own symbol list 
           pointer.  (anderson@redhat.com)

         - Implementation of a "--minimal" command line option, which brings 
           up a crash session that is restricted to the "log", "dis", "rd", 
           "sym", "eval" and "exit" commands.  This option may provide a way to 
           extract some minimal/quick information from a corrupted or truncated 
           dumpfile, or in situations where one of the several kernel subsystem 
           initialization routines, which are not called, would abort the
           crash session.  (sharyath@in.ibm.com, sachinp@in.ibm.com)

         - 4.0-6.3 to 4.0-7.1 incremental patch

           (8/19/08)
 

4.0-7     - License tag in crash.spec changed from "GPL" to "GPLv2"; otherwise
            identical to 4.0-6.3.  (spot@fedoraproject.org)

          - Available only in Fedora Rawhide devel branch:
             build: dist-f10,devel:crash-4.0-7
             http://koji.fedoraproject.org/koji/buildinfo?buildID=56268

          (7/15/08)
  

4.0-6.3  - Support for Fedora FC9 kernels containing the linux-2.6.utrace.patch,
           which removes the task_struct.parent member.  Without the patch, the
           crash session fails during initialization with the error message: 
           "crash: invalid structure member offset: task_struct_parent".  
           (anderson@redhat.com)

         - Available in Fedora Rawhide devel branch: 
             build: dist-f10,devel:crash-4.0-6.3
             http://koji.fedoraproject.org/koji/buildinfo?buildID=47600
           
         - Further scalability improvements to the "search -k" mechanisms.
           (anderson@redhat.com)

         - Changed ppc64 manner of determining the number of cpus to first check
           the cpu_present_map, and only if that doesn't exist, continue to use
           the cpu_online_map.  Without the patch, depending upon which cpus
           were offline, crash sessions could fail during initialization with
           the error message: "crash: cannot determine idle task addresses from 
           init_tasks[] or runqueues[]".  (anderson@redhat.com)

         - Fix/workaround for the ppc64 "bt" command on panic/active tasks when
           run against dumpfiles whose kernel had crashed with one or more
           cpus offline.  Without the patch, the "bt" command could cause a 
           segmentation violation, or fail because the starting stack location
           and instruction pointer were invalid.  With the patch, an error 
           message will be displayed, indicating that the NT_PRSTATUS note for 
           that task could not be determined.  (anderson@redhat.com)

         - Added support for vtop translation of 1MB large pages available on 
           new z10 (s390x) systems. (holzheu@linux.vnet.ibm.com)

         - Prevent misleading init-time warning message for s390/s390x when 
           verifying the vmlinux file with respect to the host machine type.  
           Without the patch, this message would appear when running on s390
           or s390x machines: "WARNING: machine type mismatch: crash utility:
           S390X /usr/lib/debug/lib/modules/2.6.18-86.el5/vmlinux: (unknown)"
           (holzheu@linux.vnet.ibm.com)

         - Minor documentation fix to crash.8 man page, moving the "wr" command 
           from being munged into the "whatis" description into its own list 
           entry.  (yamato@redhat.com)

         - Support for running against an x86 xen-syms hypervisor binary based
           upon xen 3.1.2 or later.  Without the patch, the session would fail
           to recognize that it was PAE, and "bt" commands on the non-active 
           task would fail with the error messages "bt: cannot resolve stack 
           trace" and "bt: invalid structure size: task_struct".
           (oda@valinux.co.jp, anderson@redhat.com)

         - Support for running against an x86_64 xen-syms hypervisor binary 
           based upon xen 3.1.2 or later.  Without the patch, the session would
           fail during initialization with the error message: "crash: cannot 
           resolve idle_pg_table_4".  In addition, the x86_64 xen-syms 
           hypervisor is now relocatable, but the kdump vmcore does not 
           (currently) export the base physical address of the relocated 
           hypervisor text and static data.  Without that knowledge, the crash 
           utility cannot make virtual to physical address translations, and 
           therefore cannot navigate through the vmcore.  To address that 
           shortcoming, a patch is required for either the xen hypervisor code 
           or the kexec-tools package to export the value of the hypervisor's 
           "xen_phys_start" symbol to the vmcore.  Until such time, however, a 
           workaround has been put in place to pass the value with a new command
           line option that is invoked like so:

             # crash --xen_phys_start <address> xen-syms vmcore

           The value of the xen_phys_start <address> argument can be 
           determined in two ways, either from /proc/iomem on the live 
           system running the dom0 kernel that generated the kdump, or by 
           running crash on the target vmcore using the dom0 vmlinux file.
           For example, on this system, the <address> argument would be
           3ee00000:
             
             # cat /proc/iomem | grep Hypervisor
               3ee00000-3fdfffff : Hypervisor code and data
             #

           Alternatively, the vmcore file in this example indicates that the
           <address> argument would be 0x3f000000: 

             # crash vmlinux vmcore
             ...
             crash> px xen_hypervisor_res
             xen_hypervisor_res = $3 = {
               start = 0x3f000000, 
               end = 0x3fffffff, 
               name = 0xffffffff8049ab72 "Hypervisor code and data", 
               flags = 0x80000200, 
               parent = 0xffff880000001180, 
               sibling = 0x0, 
               child = 0xffff8800000000a8
             }
           
           If the --xen_phys_start command line option is not used, the session
           will fail during initialization.  However there will be a warning
           message preceding the failure indicating: "WARNING: This hypervisor 
           is relocatable; if initialization fails below, try using the 
           --xen_phys_start <address> command line option".  Eventually the 
           value of the hypervisor's "xen_phys_start" will be passed in the 
           vmcore header, obviating the need for this workaround.
           (oda@valinux.co.jp, anderson@redhat.com)
        
         - 4.0-6.2 to 4.0-6.3 incremental patch

           (4/30/08)
  

4.0-6.2  - Implemented a new "rd -S" option which, like the "-s" option, 
           displays the symbolic translation of kernel virtual addresses,
           but also recognizes the virtual addresses of slab objects, and when
           found, the address is replaced by the kmem_cache slab name string
           inside brackets.  (anderson@redhat.com)

         - Make the found address displayed by "kmem -[sS] <address>" be the
           address of the containing object if the <address> argument is 
           offset from the beginning of the object.  This only applies to 
           kernels using kernel/slab.c; CONFIG_SLUB kernels currently do display 
           the address of the containing object.  
           (anderson@redhat.com)

         - Fix for "kmem -[sS] [address]" in 2.6.25 CONFIG_SLUB kernels, which
           address changes in the kernel's per-slab free list tracking.  Without 
           the patch, error messages of the type "kmem: invalid kernel virtual 
           address: 10700  type: get_freepointer" would be seen when the full 
           list of objects in a per-cpu slab was displayed.  
           (anderson@redhat.com)

         - Fix for "kmem -[sS] <slab-address>" in 2.6.25 CONFIG_SLUB kernels,
           in which the slab structure is actually a page struct.  Some slab
           addresses would not be recognized as such, and therefore without the 
           patch, error messages of the type "kmem: address is not allocated in
           slab subsystem: <slab-address>" would be seen.
           (anderson@redhat.com)

         - Fix for an initialization-time failure with Ubuntu kernels because
           of a mismatch between the /proc/version string and the linux_banner 
           string, due to additional information appended to the linux_banner 
           string in Ubuntu kernels.  (anderson@redhat.com, asid@hp.com)

         - Fix for the "net" command in 2.6.22 and 2.6.23 kernels, where the 
           "dev_base" net_device structure was replaced by the "dev_base_head" 
           list_head.  Without the patch, the "net" command with no arguments 
           would fail with the error message: "net: dev_base does not exist!".
           (eteo@redhat.com)

         - Fix for the "net" command in 2.6.24 and later kernels where the 
           global "dev_base_head" list_head has been removed, and the network
           devices are linked from the "init_net" net structure.  Without the
           patch, the "net" command with no arguments would fail with the
           error message: "net: dev_base does not exist!".
           (anderson@redhat.com)

         - For kernels configured with CONFIG_SLUB, "kmem -S" has been updated
           to properly differentiate whether a cache's "full" slabs are tracked
           but whose full list is empty, or whether the full slabs are not 
           tracked at all.  Without this patch, a cache's full list could be 
           indicated as "(empty)" instead of the more correct indication of 
           "(not tracked)".  (i-kitayama@ap.jp.nec.com, anderson@redhat.com)

         - Fix for the "vm" command when the crash session was invoked with 
           the -s command line option.  Without the patch, if invoked prior to
           a "set", "ps" or "vtop" command, the "vm" command run against a
           task other than the initial context would mistakenly indicate that
           the task contained no virtual memory.  
           (anderson@redhat.com, baiwd@cn.fujitsu.com)

         - Fix/workaround for the "search -k" command option on relocatable
           2.6-era ia64 machines configured with CONFIG_SPARSEMEM.  Without 
           the patch, an immediate segmentation violation occurs.
           (anderson@redhat.com, yzgcsu@cn.fujitsu.com)
 
         - 4.0-6.1 to 4.0-6.2 incremental patch

           (3/31/08)
  

4.0-6.1  - Support for 2.6.25 x86_64 kernels with the x86/x86_64 merger patch.
           Without the patch, attempting a crash session would fail during
           initialization with the error message: "crash: invalid structure 
           member offset: tss_struct_ist".  (anderson@redhat.com)

         - Support for 2.6.25 x86 kernels with the x86/x86_64 merger patch.
           Without the patch, attempting a crash session on a dumpfile would 
           fail during initialization with the error message: "crash: invalid 
           structure size: user_regs_struct".  (anderson@redhat.com)

         - Fix for "bt" command when running on a live 2.6.25 x86 kernel with
           the x86/x86_64 merger patch.  Without the patch, "bt" would fail
           with the error message: "bt: invalid structure member offset: 
           task_struct_thread_eip".  (anderson@redhat.com)

         - Fix for the "timer" command in 2.6.25 kernels.  Without the patch
           the command would fail with the error message: "timer: zero-size 
           memory allocation! (called from <user address>)".
           (anderson@redhat.com)

         - Cosmetic change to the x86 "bt" command to recognize the entry point
           name change from "sysenter_entry" to "ia32_sysenter_target".  Without
           the patch, the entry point would indicate the "sysenter_past_esp"
           assembly code label.  (anderson@redhat.com)

         - 4.0-6.0 to 4.0-6.1 incremental patch

           (2/29/08)
 

4.0-6.0  - Available only as version 4.0-6.0.5 in Fedora's dist-f9/devel branch:

             http://koji.fedoraproject.org/koji/buildinfo?buildID=37614

         - When compiling within a 2.6.25-based build environment, four 
           "typedef unsigned int u32;" declarations are required due to a new
           structure declaration in "asm-x86/ptrace-abi.h" that uses u32 
           members, but u32 is only defined in "asm-x86/types.h" within an 
           #ifdef __KERNEL__ section.  I posted a patch on LKML to address the
           ptrace-abi.h problem by changing the structure member declarations
           to use __u32 typedefs, which was accepted in the -mm tree.  
           (anderson@redhat.com)

         - 4.0-5.1 to 4.0-6.0 incremental patch

           (2/20/08)
 

4.0-5.1  - Update "ps -l" to use task_struct.sched_info.last_arrival value
           on 2.6.23 and later kernels that don't have a task_struct.last_ran 
           member.  Without the patch, the option would fail with the error
           message: "ps: neither task_struct.last_run nor task_struct.timestamp 
           exist in this kernel".  (anderson@redhat.com)

         - Fix for potential initialization-time failure when running against
           2.4-era x86 netdump dumpfiles if the ebp and esp contents in the 
           ELF header's NT_PRSTATUS register dump do not contain a vestige of 
           the panic task's kernel stack address.  Without the patch, there may
           be one or more warning messages complaining about tasks not being in 
           the PID hash, followed by a fatal error message: "crash: invalid 
           kernel virtual address: <bad-address>  type: 32-bit KVADDR", where
           the <bad-address> can be any bogus kernel virtual address.
           (anderson@redhat.com)

         - Fix to make the unused do_radix_tree() function work as advertised.
           (atyson@hp.com)

         - Added zlib-devel to the crash-devel package-dependency Requires line
           in the crash.spec file.  (anderson@redhat.com)

         - 4.0-5.0 to 4.0-5.1 incremental patch

           (2/19/08)
 

4.0-5.0  - Tentatively scheduled as the baseline version for RHEL4.7 and RHEL5.2
           crash utility errata releases; also built in Fedora Rawhide:

             4.0-5.0.0 - RHEL4.7 errata version
             4.0-5.0.2 - RHEL5.2 errata version
             4.0-5.0.3 - Fedora Rawhide (devel branch)

         - Fix for a potential segmentation violation during crash session 
           initialization if a task's kernel stack has been completely overrun,
           corrupting its thread_info structure at the bottom of the stack.
           This could occur running against kernels from 2.6.8 through 2.6.18.
           With the patch, the suspect task will be reported during the task
           initialization sequence.  (anderson@redhat.com)

         - Fix for the "bt" command when run on xen x86 dom0 dumpfiles, which 
           may potentially show empty backtraces for one or more active tasks.
           (oomichi@mxs.nes.nec.co.jp)
      
         - Initial support for OpenVZ kernels.  (kshileev@sw.ru)
        
         - 4.0-4.13 to 4.0-5.0 incremental patch

           (1/17/08)
 

4.0-4.13 - If the vmlinux file or dumpfile is a machine type mismatch with 
           the crash utility binary, or far less likely, a ppc64 or ia64 
           endian mismatch, the crash session will fail during initialization 
           with the generic error message, "crash: <filename>: not a 
           supported file format".  To aid the user in understanding what
           caused the failure, this patch prepends an additional error
           message that clarifies the reason behind the mismatch.
           (anderson@redhat.com, bwalle@suse.de)

         - An update for "kmem -V" option, which currently displays the kernel's
           "vm_stat" counter values, will now also display the "vm_event_states"
           counter values, both of which were introduced in 2.6.18.  For 2.6
           kernels prior to 2.6.18, the precursor "page_states" counter values
           will be displayed.  (anderson@redhat.com) 

         - Implemented a new "kmem -z" option to display per-zone memory
           statistics.  The amount of data displayed is dependent upon the
           kernel version.  At a minimum, the size, min/low/high and free
           page counts are shown.  If the zone struct contains nr_active,
           nr_inactive, pages_scanned and all_unreclaimable members, those
           fields are shown.  If the zone struct contains a per-zone vm_stat[]
           array (identical to the system-wide vm_stat[] array), its contents
           are dumped.  For any other data in the zone, the address of the 
           zone structure is displayed.  
           (anderson@redhat.com)

         - Fix for the RSS amounts displayed by the "ps" and "vm" commands
           on 2.6 kernels prior to 2.6.13.  (anderson@redhat.com)

         - Fix for the x86 "bt" command when running a version of crash built
           on a pre-2.6.20 host against a 2.6.20 or later dumpfile, or when
           running a version of crash build on a 2.6.20 or later host against
           a pre-2.6.20 dumpfile.  Without the patch, kernel exception frames 
           would be mistaken for, and displayed as, user exception frames, and 
           parts of the backtrace above the kernel exception frame would be 
           truncated.  (atyson@hp.com)

         - Fix for FC8 xen x86 kernels (2.6.21-2952.fc8xen) that fail during
           initialization after reporting "WARNING: cannot read linux_banner 
           string", followed by the fatal error message "crash: vmlinux and 
           vmcore do not match!".  This required a change to the virtual
           address mask value used to determine the base value of the x86 
           kernel's unity-mapped virtual address region.  (anderson@redhat.com)  

         - Set a default "phys_base" value for recent fully-virtualized 
           relocatable x86_64 kernels whose text start address is not equal 
           to the __START_KERNEL_map value.  Without the patch, the crash
           session fails during initialization with the warning message
           "WARNING: cannot read linux_banner string", followed by the fatal
           error message "crash: vmlinux and vmcore do not match!".  The
           error can alternatively be worked around if the "phys_base" value
           is first determined by running a crash session on the live system
           that generated the dumpfile, by entering: "help -m | grep phys_base".
           The value shown can then be used when running against the dumpfile
           like so: "crash --machdep phys_base=<value> vmlinux vmcore"
           (anderson@redhat.com) 

         - Debug: implemented a new "--active" crash command line option, which
           will gather only the active tasks from each runqueue, skipping the
           traversal of the kernel's pid_hash mechanism.  
           (anderson@redhat.com) 

         - Debug: "help -n" formats and displays ASCII VMCOREINFO data.
           (anderson@redhat.com)

         - 4.0-4.12 to 4.0-4.13 incremental patch

           (1/11/08) 
 

4.0-4.12 - Fix for the "kmem -n" command to handle the 2.6.24 kernel replacement
           of the "node_online_map" nodemask with its appropriate entry in the
           new "node_states[]" nodemask array.  Without the patch, the per-node 
           zone data would not be displayed, and any commands depending upon 
           the node table data would be affected.  (anderson@redhat.com) 

         - Fix for "kmem -p" on 2.6.24 x86_64 kernels that are configured with
           CONFIG_SPARSEMEM_VMEMMAP,  which use a virtually-mapped page struct
           array.  Without the patch, the virtual-to-physical translation of
           each page structure was invalid, and "kmem -p" would display invalid
           data.  This would also affect other commands as well, such as the
           output of "kmem -i", and the output of a "vtop" command on a mapped
           page address.  Also, the virtual base address of the region is now 
           displayed by the "mach" command. 
           (oomichi@mxs.nes.nec.co.jp, anderson@redhat.com)

         - Fix for the "dev" command's character device name string output to 
           recognize the change of the name structure member from a pointer 
           to an embedded string.  Without the patch, 2.6.16 and later kernels
           would display "(unknown)" character device names.  
           (olivier.daudel@u-paris10.fr, anderson@redhat.com) 
       
         - Fix for the "kmem -[sS]" command to handle the 2.6.24 change to 
           the CONFIG_SLUB kmem_cache structure, which re-worked the manner
           in which the per-cpu slabs get referenced.  Without the patch,
           the command would fail with several error messages of the type:
           "kmem: page_to_nid: invalid page: ffff81003993f4b0".
           (anderson@redhat.com)
           
         - Fix for the "kmem -[fF]" command to handle the 2.6.24 kernel change 
           of the free_area struct, which replaced the singular linked list
           of pages with 5 (MIGRATE_TYPES) linked lists.  Without the patch,
           the command would fail with the error message: "kmem: unrecognized
           free_area struct size: 88".   (anderson@redhat.com)

         - Fix for the "runq" command to handle the 2.6.24 kernel change to
           the CFS scheduler that introduced per-cpu init_cfs_rq structures
           for task group scheduling.  Without the patch, no queued tasks 
           were displayed, because the rb_root of queued tasks was being
           taken from the embedded cfs_rq in each per-cpu runqueue.  
           (anderson@redhat.com)  

         - 4.0-4.11 to 4.0-4.12 incremental patch

           (12/12/07) 
 


4.0-4.11 - Fix for task-gathering to handle the 2.6.24 pid_namespace-related
           changes to the kernel pid_hash array.  Without the patch, the crash
           session fails during initialization with the message "crash:  cannot
           gather a stable task list via pid_hash (500 retries)".
           (anderson@redhat.com)

         - Fix for "kmem -f <address>" and "kmem <address>" commands on 
           x86 kernels, which may incorrectly indicate that the address is in
           the kernel's free page list.  Without this patch, if the address 
           argument is a physical address over 4GB, or a page struct address 
           referencing a physical address over 4GB, it is possible that the 
           address would incorrectly be shown as being in the kernel's free 
           page list.  (anderson@redhat.com)

         - Fix for x86 "bt" command for active tasks in Egenera dumpfiles
           based upon LKCD version 7.  Without the patch, the starting points
           for the active task backtraces were erroneous.  
           (anderson@redhat.com)

         - Fix for "kmem -S" error message if a slab object is found in both
           a per-cpu list and on a slab's global free list.  Without the patch,
           the object address and cpu number values are flip-flopped in the 
           error message.  (bob.montgomery@hp.com)

         - 4.0-4.10 to 4.0-4.11 incremental patch

           (12/6/07) 


4.0-4.10 - Fix a regression introduced in 4.0-4.9 that causes the "kmem -p"
           command to fail in SPARSEMEM kernels that that have the struct
           page.index member embedded in an anonymous union, which occurred
           when the CONFIG_SLUB-related modifications were made to the page 
           struct in 2.6.22.  Without the patch, "kmem -p" fails with the error
           message "kmem: invalid structure member offset: page_index".
           (anderson@redhat.com)

         - 4.0-4.9 to 4.0-4.10 incremental patch

           (11/21/07) 


4.0-4.9  - Fix for the "kmem -p" command in kernels configured with 
           CONFIG_SPARSEMEM, i.e., not CONFIG_SPARSEMEM_EXTREME.  Without 
           the patch, the page structure address for each physical page 
           was erroneous.  (oomichi@mxs.nes.nec.co.jp)

         - Fix for the "kmem -p" command output of MAPPING and INDEX values
           on kernels where the mapping and index members of the page structure
           are contained within anonymous unions.  Without the patch, those
           fields may be dashed-out.  
           (bob.montgomery@hp.com, anderson@redhat.com)
        
         - Fix for the "mod" command to search for module object files in the
           /lib/modules/<release>/updates directory tree before looking
           in /lib/modules/<release>.  (charlotte.richardson@stratus.com)

         - Fix for the "waitq" command for 2.6.15-era and later kernels, which  
           replaced the __wait_queue.task member with the __wait_queue.private
           member.  Without the patch, the command would fail with the error
           message: "waitq: invalid structure member offset: __wait_queue_task".
           (atyson@hp.com)

         - SIAL interpreter fix for an "operation on 'v1' may be undefined" 
           warning in sial_exeop().  (bwalle@suse.de)

         - Fix for several unpredictable failure modes when attempting
           "crash -h [command] > outputfile" from a shell command line.  
           (anderson@redhat.com)

         - Addressed compiler warnings generated by extensions/echo.c and
           extensions/dminfo.c.  (bwalle@suse.de, anderson@redhat.com)

         - Addressed compiler warnings generated by lkcd_common.c, lkcd_v8.c
           and symbols.c when using:
             -O2 -fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector
             -fno-builtin-memset -fno-strict-aliasing
           (bwalle@suse.de)

         - Fix for "kmem -p" on i386 CONFIG_SPARSEMEM kernels with greater than
           4GB of memory.  Without the patch, the physical address value wraps
           back to zero after physical page ffff0000.
           (oomichi@mxs.nes.nec.co.jp)

         - Fix to redirect SIAL script command output to pipes, files, etc., in
           the same manner as native crash commands.  
           (Robert.Denman@teradata.com, anderson@redhat.com)

         - Fix for ppc64 kernels with 64K pages whose PTE_RPN_SHIFT has changed
           from 32 to 30.  Without the patch, an initialization-time warning 
           message "WARNING: cannot access vmalloc'd module memory" would occur,
           the "mod" command would fail with the same message, and "kmem -s"
           failures could occur when attempting to read a kmem slab cache name
           string.  Translations and reads of vmalloc'd kernel virtual addresses
           and user virtual addresses would appear to work, but bogus data was
           returned because the resultant physical address that was read was
           incorrect.  (anderson@redhat.com)

         - Fix for "kmem -s" if a slab cache whose name string cannot be read 
           is encountered.  Without the patch, a fatal error message would be
           displayed and the command aborted.  With this patch, a non-fatal
           warning message is displayed, and the cache name is indicated as
           "(unknown)".  (anderson@redhat.com)

         - Fix for x86-64 SPARSEMEM kernels with CONFIG_NUMA off.  Without the
           patch, the crash session fails during initialization with the message
           "crash: invalid structure member offset: pglist_data_node_mem_map".
           (sachinp@in.ibm.com)

         - Fix to use the ia64 physical start address from the LKCD dump header 
           instead of the default value.  This was reported as bug on an SGI
           machine.  (bwalle@suse.de)

         - For s390[x] kernels the page table allocation method will be changed
           such that instead of 3 levels, it will be now possible to allocate 4
           levels.  The current implementation of the page table walk functions
           in the crash utility makes assumptions on how the page tables are 
           allocated by the kernel, e.g. 3 levels are hard coded.  This patch 
           changes that, and the page table walk is done only according to the 
           s390 architecture without assumptions on the implementation in the 
           kernel.  (holszheu@linux.venet.ibm.com)

         - Fix for LKCD dumpfile access failures that abort() the crash session
           after displaying an error message indicating a problem with physical  
           memory zones in the dumpfile.  Without the patch, the crash session
           would end immediately after displaying an error message of the sort:
           "conflicting page: zone 0, page 0: 0, 177160130 != 65536".  That
           error message will now only be displayed if the crash debug mode is 1 
           or more, a readmem() "seek error" will be displayed instead, and the
           session will return to the "crash>" prompt.  (anderson@redhat.com)

         - 4.0-4.8 to 4.0-4.9 incremental patch

           (11/20/07) 
           

4.0-4.8  - Implemented support for kernels configured with CONFIG_SLUB, which 
           completely replaces the venerable "kernel/slab.c" with the new 
           "kernel/slub.c" kmalloc() slab subsystem.  Accordingly, the 
           "kmem -s [address]", "kmem -S [address]", and "kmem <address>"
           commands will display slab-related information in a similar manner
           to what they currently do, with additional per-node information.
           It should be noted that, due to slub.c's design, the verbose 
           "kmem -S" output will be pared down slightly to not display the 
           list of all "full" slabs unless the proper kernel slub debugging 
           has been turned on.  However, given a address of an object from a 
           full slab page, or of the full slab page itself, that address
           will then be traced back to its original slab cache and its data
           displayed.  (anderson@redhat.com)

         - Change for support of LKCD dumpfile version 8 and later to determine
           the backtrace starting registers from the dumpfile header.  Increase
           (maximum) NR_CPUS for ia64 to 4096.
           (bwalle@suse.de)

         - The SIAL interpreter extension module has been updated to support
           the ia64, ppc64, s390 and s390x architectures.  Several fixes have
           been applied, and three new debug commands, sdebug, sclass and sname
           have been added.  (lucchouina@yahoo.com)

         - Fixed a bug in the CONFIG_SPARSEMEM patch (contributed in 4.0-3.22) 
           in which a static pointer variable was initializing itself with a 
           buffer that was returned from a command-time-only GETBUF() call, 
           instead of using malloc().  It would then continue to use the buffer,
           trampling on the buffer contents set up by whatever command that 
           subsequently allocated the buffer.  I only caught this during the 
           CONFIG_SLUB development, so I have no examples (if any) of how this
           would have ever manifested itself in a crash command error.
           (anderson@redhat.com)

         - Fixed the "mach" command in CONFIG_SLUB kernels which would abort
           with the error message: "mach: cannot resolve cache_cache" when
           trying to determine the value for the L1 CACHE SIZE display.  Since
           the generic manner of determining the cache size no longer worked 
           correctly anyway, the L1 CACHE SIZE display has been removed.
           (anderson@redhat.com)

         - Fix for missing NODE header in NUMA "kmem -f" output.
           (anderson@redhat.com)

         - Fix for the chronology of the contents of the kernel message buffer 
           output by the "log" command.  (atyson@hp.com)

         - Display a WARNING message if a PT_LOAD segment in an ELF-style
           dumpfile advertises a memory segment that would go beyond the end
           of the dumpfile.  (bwalle@suse.de, anderson@redhat.com)

         - 4.0-4.7 to 4.0-4.8 incremental patch

           (10/30/07) 


4.0-4.7  - Incorporation of Luc Chouinard's SIAL interpreter (Simple Image
           Access Language) as a crash extension module.  When loaded with
           the "extend" command, the sial.so module provides three commands,
           "load" to load a SIAL script, "unload" to unload it, and "edit", 
           which unloads the script, brings up an $EDITOR-based edit session
           of the script, and then loads it again.  Also, when the sial.so
           module is loaded, it will automatically load any SIAL scripts
           found in the /usr/share/sial/crash or $HOME/.sial directories.
           Therefore, by putting "extend <path-to>/sial.so" in either
           ./.crashrc or $HOME/.crashrc, all desired SIAL scripts may be
           loaded on a particular machine in a hands-off manner.  For details,
           consult the README and README.sial files in the extensions/libsial
           subdirectory.  (lucchouina@yahoo.com)

         - Removed hardwired-dependencies in the top-level and extensions
           subdirectory Makefiles for building extension modules.   Now it is 
           possible to copy an extension module's .c file into the extensions 
           subdirectory, and enter "make extensions" from the top-level to build
           it.  If the build of the module requires special handling, a .mk 
           makefile with the same prefix as the .c file may be provided, and
           and it will be automatically used to build it.
           (jmoyer@redhat.com, anderson@redhat.com)

         - When a 32-bit x86 xenU guest is run on an x86_64 dom0 host, the
           new-style xen ELF format dumpfile contains an ELF header with an
           e_machine type of EM_X86_64 (instead of EM_386).  This was getting
           gets rejected with the error message "crash: vmcore: not a supported 
           file format".  The fix simply accepts the e_machine type mismatch,
           since the new-style ELF format dumpfiles are 64-bit by default. 
           (anderson@redhat.com)

         - Enhanced the "kmem <address>" option to also search for task_struct 
           and kernel stack addresses, and report them with the "set" output.
           Also, fix for when "kmem <vmalloc-address>" was entered, the header
           for the mem_map data was not displayed.  (anderson@redhat.com)

         - Fix for determining starting rip/rsp backtrace hooks for the panic
           task in x86_64 xen dom0 kdumps; newer kernels have replaced the
           call to "xen_machine_kexec" with "machine_kexec", and without this
           patch may display back-traces with missing frames.  Also on x86_64
           non-xen kdump panic task backtraces, it is possible that the wrong
           stack instance of "crash_kexec" is used as the starting hook, which
           may also lead to missing frames.  (anderson@redhat.com)

         - Fix for ia64 LKCD dumpfiles where it is not possible to read the task
           structure of the task that follows a task which is in the task address
           "fixup list", and zeroes are returned instead.  (atyson@hp.com)

         - Fix for potential "mod -[sS]" failures with modules whose object
           files contain an unusually large number of sections; module 
           loading attempts may issue a "<segmentation violation in gdb>"
           message followed by the error message: "mod: [module name]: gdb 
           add-symbol-file command failed".
           (carl.hsieh@teradata.com, anderson@redhat.com)

         - Fix to prevent dumpfile reads beyond EOF when reading new (optimized)
           xen ELF core xendumps.  Without the patch, error messages of the sort:
           "crash: cannot read index page [number]" may occur during session
           initialization, with unpredictable run-time results.
           (yamahata@valinux.co.jp)

         - In x86_xen_kdump_p2m_create(), the same variable was being used as 
           the for-loop index in both an outer and an embedded inner for-loop.
           As a result, if debug level was equal to or larger than 7, the outer
           for-loop was repeated only once.  (nishimura@mxp.nes.nec.co.jp)

         - 4.0-4.6 to 4.0-4.7 incremental patch

           (9/25/07) 


4.0-4.6  - Also released as:

             4.0-4.6.1 - RHEL5.1 errata version (beta)
             4.0-4.6.2 - Fedora Rawhide (devel branch)

         - Implemented the "runq" command for 2.6.20 and later kernels that have
           replaced the O(1) scheduler with the CFS scheduler.  If the kernel
           was configured to use CFS, the command will display the tasks queued
           in each cpu's RT and CFS runqueues.  (anderson@redhat.com)

         - The initial support put in place for the usage of "kerntypes"
           debuginfo files only recognized files created by the LKCD 
           "dwarfextract" utility run against a -g built vmlinux kernel. 
           This version adds a new "-k" command line option that allows the 
           usage of standard -g compiled LKCD Kerntypes files.  
           (holzheu@linux.vnet.ibm.com)

         - Update of "xencrash" support to properly handle dom0/hypervisor
           kdumps taken under xen version 3.1 in addition to those taken under 
           xen 3.0.x.  Without this patch, the following warning message 
           would be displayed during initialization of a xen-syms hypervisor
           session: "WARNING: unsupported elf note format".  Fixes x86 "bt" 
           command segmentation violation when running against a xen-syms 
           hypervisor.  Fixes x86_64 session initialization failure when running
           against a xen-syms hypervisor, which would display the error 
           message "crash: invalid structure member offset: tss_struct_rsp0".  
           (oda@valinux.co.jp)

         - 4.0-4.5 to 4.0-4.6 incremental patch

           (8/27/07) 
 

4.0-4.5  - Addresses FC7/upstream x86 kernels that have been configured such
           that the vmlinux symbol values do not match their relocated values
           when loaded.  If CONFIG_PHYSICAL_START contains a value that is
           greater then CONFIG_PHYSICAL_ALIGN, then this mismatch occurs.
           Since the crash utility and its embedded gdb have always expected
           that the compiled-in kernel symbol addresses are "real", the virtual
           to physical translation fails, leading to an initialization-time
           failure with the message: "crash: vmlinux and /dev/crash do not
           match!" (/dev/mem or the dumpfile name may replace /dev/crash).
           To deal with this issue, there are several alternatives:

            1) Configure the kernel with CONFIG_PHYSICAL_START less than
               or equal to CONFIG_PHYSICAL_ALIGN.  Having done that, there
               is no problem; the resultant vmlinux file will be loaded at
               the address for which it was compiled, which has always
               been the case.
            2) Since /proc/kallsyms uses the same format as a System.map file,
               and since it reflects the relocated symbol addresses, it
               can be placed on the crash command line as if it were
               a System.map file.  (Note that the System.map file created
               by these relocated kernels contains the same "wrong" symbol
               values as the vmlinux file from which it was created.)
            3) On a live system that has /proc/kallsyms (i.e., the kernel was
               configured with CONFIG_KALLSYMS), this version of the crash
               utility will replace/patch the vmlinux symbol values with those
               seen in /proc/kallsyms.  The relocation value will be displayed
               as a WARNING message during initialization.
            4) On a dumpfile, the relocation will not be performed automatically
               as on a live system.  It will require the addition of the 
               /proc/kallsyms on the command line, or if run on a different
               host, a copy of the crashed system's /proc/kallsyms may be
               used.
            5) Alternatively on a dumpfile, a new command line option has been
               created to specify the relocation amount.  For example, if a  
               kernel was configured with a CONFIG_PHYSICAL_START value of 16MB
               and a CONFIG_PHYSICAL_ALIGN of 4MB, that results in a relocation
               of 12MB.  To specify that, enter "crash --reloc=12m ..." on the
               command line.  (Recall that if crash is run on the live system,
               a WARNING message will specify the relocation amount.)

           Using /proc/kallsyms or a --reloc=[size] as a command line argument
           is similar to using a System.map file, in that it results in the loss
           of the use of line number debug data.  (anderson@redhat.com)

         - Fix for x86 2.6.22 kernel initialization-time failure indicating:
           "crash: invalid size request: 0  type: __per_cpu_offset"
           (oomichi@mxs.nes.nec.co.jp)

         - Fix to recognize the 2.6.22 kernel's replacement of kmalloc slab 
           subsystem from the "./mm/slab.c" file to CONFIG_SLUB-configured 
           kernels that use the infrastructure in "./mm/slub.c".  Without this 
           fix, crash sessions would fail during initialization with the message
           "crash: invalid structure member offset: kmem_cache_s_c_num".
           (anderson@redhat.com)

         - Cliff Wickman sent an additional patch for the LKCD kerntypes 
           support he introduced in version 4.0-4.4, which addresses this 
           message that is seen during initialization on 2.6.22 kernels:
           "WARNING: cannot determine pgdat list for this kernel/architecture".
           (cpw@sgi.com)

         - NOTE: The CONFIG_SLUB change in the 2.6.22 kernel will require a 
           significant update in the crash utility in order for "kmem -[sS]"
           options to work again. 

         - NOTE: 2.6.20 and later kernels may have replaced the O(1) scheduler
           with the new CFS scheduler.  If configured to use CFS, the "runq" 
           command fails, which will require a crash utility update to recognize
           and display the contents of each cpu's RT and CFS run queue. 

         - 4.0-4.4 to 4.0-4.5 incremental patch

           (7/27/07) 


4.0-4.4  - Fix for kernels in which the irq_desc_t typedef is not included in 
           the vmlinux debuginfo data, by using the 2.6-era struct irq_desc.  
           Without the patch, the "irq" command fails with the error message, 
           "irq: cannot determine size of irq_desc_t".  (hugh@mimosa.com) 

         - Implemented new "irq -u" option that displays only in-use IRQs, now
           that there can be several thousand entries in the irq_desc[] array.
           (anderson@redhat.com)

         - Prevent occasional 99% cpu usage waiting for the built-in less 
           command to complete.  (anderson@redhat.com)

         - Implemented support for the use of "kerntypes" debuginfo files that
           are created by the LKCD "dwarfextract" utility, as an alternative to 
           the use of the vmlinux file.  This requires the use of the matching 
           System.map file, as in this example: 

             # crash kerntypes System.map [vmcore]
 
           This capability was written by Cliff Wickman of SGI, and he has
           generously offered to maintain its functionality.  (cpw@sgi.com)

         - Fixes, code improvement and cleanup for "crash -h [command]".
           (hugh@mimosa.com)
 
         - The output of command data exceeding a terminal page-size has been
           traditionally fed by default to "/usr/bin/less -E -X" with a prompt;
           if the /usr/bin/less command was not available on the host system,
           output would be fed to "/bin/more" instead.  Scrolling can be turned
           off with "set scroll off" or the built-in alias "sf", and back on
           with "set scroll on" or the built-in alias "sn". This release
           allows the user to specify an alternative scrolling program by
           creating a CRASHPAGER environment variable, which be used by default
           if it exists.  Also, the "set scroll [arg]" internal variable setting
           command, which until now accepted "on" and "off" as arguments, now
           accepts "less", "more" and "CRASHPAGER" as alternative arguments,
           both during runtime, or in .crashrc files.  Also, new crash command 
           line arguments have also been added to override the default and/or 
           .crashrc settings: --more, --less, and --CRASHPAGER.  Lastly, the 
           output of the "crash -h [command]" will also use the relevant scroll
           command selection.  (anderson@redhat.com)

         - Updated crash(8) man page.  (hugh@mimosa.com, anderson@redhat.com)

         - 4.0-4.3 to 4.0-4.4 incremental patch

           (7/20/07) 


4.0-4.3  - Tentatively scheduled as the baseline version for RHEL4.6 and RHEL5.1
           crash utility errata releases:

             4.0-4.3.0 - RHEL4.6 errata version
             4.0-4.3.1 - RHEL5.1 errata version 

         - Fix for "kmem -f" command on 2.6.17 and later CONFIG_DISCONTIGMEM
           kernels.  Without the patch, the command would fail with the error
           message "kmem: cannot determine zone mem_map: TBD".  
           (troy.heber@hp.com)

         - Fix for segmentation violation when using the wrong vmlinux file 
           command line argument on a live system on either the x86_64 or 
           ia64 architectures.  If attempted with this version, the normal
           "WARNING: vmlinux and /proc/version do not match!" message will
           be followed by an additional warning message that displays the 
           Linux version number from /proc/version, and then the final message:
           "crash: please use the vmlinux file for that kernel version, or 
           try using the System.map for that kernel version as an additional
           argument."  (anderson@redhat.com)

         - For all 4 types of input-file processing: 

             1) $HOME/.crashrc
             2) ./.crashrc
             3) "crash -i input"
             4) session-time "< input" 

           If a command in the input file encounters a FATAL error, the 
           remainder of the commands will be executed.  Until now, if any
           command in the input file caused a FATAL error, the processing
           of the remainder of the commands would be aborted.
           (anderson@redhat.com)

         - 4.0-4.2 to 4.0-4.3 incremental patch

           (6/22/07) 


4.0-4.2  - Fix for support of 2.6.22 kernels, which have changed the name
           of the task_struct's "thread_info" member to the "stack" member.
           This would cause the crash session to fail during initialization.
           (troy.heber@hp.com, anderson@redhat.com) 

         - Fix to account for the number of pgdata nodes being less than the
           number of cpus.  Without the patch, the crash session would fail 
           during initialization with the error message: "crash: numnodes out
           of sync with pgdat_list?"  (sharyath@in.ibm.com)

         - Implemented support for ia64 dom0/HV kdump dumpfile support, taken
           either via the traditional kdump process, or simulated via the 
           Fujitsu "sadump" facility.  (oda@valinux.co.jp)

         - Created a "--no_panic" command line option to avoid the panic-task 
           search during initialization.  (anderson@redhat.com)

         - Implmented a new "ps -r" option to display resource limits (ulimits):  

             crash> ps -r 20618
             PID: 20618  TASK: 1003cb82030       CPU: 1   COMMAND: "bash"
                   RLIMIT     CURRENT       MAXIMUM
                      CPU   (unlimited)   (unlimited)
                    FSIZE   (unlimited)   (unlimited)
                     DATA   (unlimited)   (unlimited)
                    STACK    10485760     (unlimited)
                     CORE        0        (unlimited)
                      RSS   (unlimited)   (unlimited)
                    NPROC      8180          8180
                   NOFILE      1024          1024
                  MEMLOCK      32768         32768
                       AS   (unlimited)   (unlimited)
                    LOCKS   (unlimited)   (unlimited)
               SIGPENDING      1024          1024
                 MSGQUEUE     819200        819200

           (anderson@redhat.com)

         - Implement support for the registration of CLEANUP extension commands
           that do not show up in help menu, but get called by restore_sanity(). 
           Extension modules may also register HIDDEN_COMMAND functions; and the
           "help -e" debug output has been enhanced.  (anderson@redhat.com)

         - Implemented a new symbol_value_module() primitive, primarily for use
           by extension modules to quickly access the address of a module symbol
           in cases where a name-clash may exist between the base kernel and/or
           other modules.  (anderson@redhat.com)

         - The crash-4.0-4.2.src.rpm package will create an additional package
           named crash-devel-4.0-4.2.i386.rpm, which is for use by extension 
           modules.  The -devel package installs the top-level "defs.h" file in 
           "/usr/include/crash/defs.h".  (anderson@redhat.com)  

         - 4.0-4.1 to 4.0-4.2 incremental patch

           (6/04/07) 


4.0-4.1  - Implemented dependable backtraces for the x86_64 architecture. (!!!)
           This feature builds upon the current "low_budget" backtrace function,
           and also required the fix for the BUG()/ud2a disassembly problem 
           addressed in 4.0-3.22.  It does not require kernel unwind support,
           but rather it calculates function framesizes by disassembling the 
           code from the beginning of the function to the point where it calls
           the next function, parsing for add or sub instructions on the rsp,
           and for push and pop instructions, thereby determining the framesize
           of the function at the point of the call.  This is similar to what is
           done for x86, but requires far less hackery.  You will notice a slight
           hitch the first time a "bt" is done on a task, but for each text
           return address in any backtrace, its framesize is cached for all
           subsequent instances.  It also accounts for backtrace text return 
           addresses from the .text.lock section, by appending "(via function)" 
           to the end of the frame line.  Also, because it layers on top of the 
           current backtrace code, it does not compromise the capability of 
           switching between the process, IRQ, and exception stacks.  That all
           being said, 100% accuracy cannot be guaranteed.  But for the ~30 
           sample dumpfiles I keep around for x86_64 testing, I cannot find any
           obviously invalid backtraces.  However, if there is any doubt, the
           "bt -o" option will perform backtraces using the "old" manner; and
           "bt -O" will force the old manner to always be used.  Of course the 
           "bt -t" and "bt -T" options are still available.  It's interesting to
           redirect the output of "foreach bt" to a file using this version, and
           then compare it with the output from an older version.  
           (anderson@redhat.com)

         - Fix for s390 and s390x backtrace commands to recognize the kernel 
           structure name change from "runqueue" to "rq".  
           (holzheu@linux.vnet.ibm.com)

         - Merged fourth round of "xencrash" patches, which allows a crash
           session to alternatively be brought up against the xen-syms
           binary instead of a vmlinux kernel.  This patch enhances the
           "doms" command display contents, and adds support to access the
           ia64 frame table virtual address space so that the page_info table
           can be accessed.  (oda@valinux.co.jp)

         - 4.0-3.22 to 4.0-4.1 incremental patch

           (4/27/07) 


4.0-3.22 - In kernel version 2.6.20 a "__bug_table" section has been added
           to the kernel for x86 and x86_64, which contains the encoding for
           the filename and line number information associated with each
           instance of a kernel BUG().  Prior to that, x86 and x86_64 kernels
           may have contained the filename/line-number encoding in the bytes
           following the BUG()'s "ud2a" instruction.  When disassembled, the
           output would display a series of nonsensical instructions, or perhaps
           one or more "(bad)" instruction lines, before eventually getting
           back in sync with the actual instruction stream.  Whether the 
           encoded bytes were included depends upon the kernel version,
           whether CONFIG_DEBUG_BUGVERBOSE was configured, or whether an
           "#if 1" surrounding the BUG() definition was manually changed.
           This version of crash determines whether the encoded bytes exist,
           and if so, the embedded gdb disassembler has been modified to
           skip over those bytes, resulting in correct "dis" command output.
           If necessary, a "dis -b" option has been added to override the 
           pre-calculated encoded byte count value.  (anderson@redhat.com)

         - Fix for the x86 backtrace code to also recognize the encoded
           filename and line number information potentially following 
           "ud2a" instructions generated by kernel BUG() calls.  In order
           to determine the framesize of a function, the backtrace code
           does its own text disassembly to count instances of push, pop,
           and stack register increments/decrements.  Without this patch, 
           the framesize calculation may either be too small or too large,
           depending upon the contents of the encoded data following the 
           BUG()'s ud2a instruction.  Therefore, it is possible that one or
           more bogus frames are selected and displayed, and/or one or more
           legitimate frames are skipped over.  For example, when it affected
           the framesize calculation of schedule(), backtraces of all non-active
           tasks ending up in schedule() would be invalid.  Here's an example in 
           which the schedule() framesize was miscalulated:

             PID: 1292   TASK: ed78a000  CPU: 0   COMMAND: "setroubleshootd"
              #0 [c07fdba8] schedule at c05f370e
              #1 [c07fdcb4] __journal_file_buffer at ee05126d
              #2 [c07fdcd8] __journal_file_buffer at ee05126d
              #3 [c07fdd08] ext3_mark_iloc_dirty at ee08837d
              #4 [c07fdd38] journal_dirty_metadata at ee052a13
              #5 [c07fdd80] __find_get_block at c0463f59
              #6 [c07fddac] __find_get_block at c0463f59
              #7 [c07fddf0] find_get_page at c0444294
              #8 [c07fddfc] filemap_nopage at c0446cf5
              #9 [c07fde6c] find_extend_vma at c0454132
             #10 [c07fde7c] get_futex_key at c042f9f6
             #11 [c07fde94] futex_wake at c042fe2a
             #12 [c07fdeb8] do_futex at c0430a19
             #13 [c07fdfac] sys_poll at c047254b
             #14 [c07fdfb8] system_call at c0404cf8
                 EAX: ffffffda  EBX: 09f3da18  ECX: 00000002  EDX: 00000064
                 DS:  007b      ESI: 00000064  ES:  007b      EDI: 00342ff4
                 SS:  007b      ESP: bfe76d04  EBP: bfe76d18
                 CS:  0073      EIP: 0094a402  ERR: 000000a8  EFLAGS: 00200246
             
           With the fix, it looks like this:
             
             PID: 1292   TASK: ed78a000  CPU: 0   COMMAND: "setroubleshootd"
              #0 [c07fdba8] schedule at c05f370e
              #1 [c07fdc0c] schedule_timeout at c05f3e7c
              #2 [c07fdc30] do_sys_poll at c047243e
              #3 [c07fdfac] sys_poll at c047254b
              #4 [c07fdfb8] system_call at c0404cf8
                 EAX: ffffffda  EBX: 09f3da18  ECX: 00000002  EDX: 00000064
                 DS:  007b      ESI: 00000064  ES:  007b      EDI: 00342ff4
                 SS:  007b      ESP: bfe76d04  EBP: bfe76d18
                 CS:  0073      EIP: 0094a402  ERR: 000000a8  EFLAGS: 00200246
           
           In the example above, the schedule() framesize was miscalculated
           because the post-ud2a text contained the filename pointer address 
           c060fe0b, and the "60" was decoded as a "pusha" instruction; that
           occurred twice, each time incrementing the framesize by 32 bytes.
           (anderson@redhat.com)

         - Added preparations for an upcoming version update to kdump's 
           associated makedumpfile utility, which will return an error if a 
           read attempt of a page that has been explicitly excluded is made.
           Until now, a zero-filled page was returned.  To maintain the
           current behavior of returning a zero-filled page when accessing
           an excluded page, three options are available:
            1) use the "--zero_excluded" crash command line option.
            2) during runtime, enter "set zero_excluded on".
            3) enter "set zero_excluded on" in your .crashrc file.
           (anderson@redhat.com, oomichi@mxs.nes.nec.co.jp, bob.montgomery@hp.com)

         - Implemented "help -n" debug output function for compressed diskdump
           and compressed kdump dumpfiles.  As is done for the other dumpfile 
           formats, the core file's header information along with any other 
           run-time dumpfile data is displayed.  (anderson@redhat.com) 

         - If the page-exclusion "dump_level" of a compressed diskdump, a 
           compressed kdump, or an ELF diskdump dumpfile exists and can be
           determined, its value and bitmask translation will be displayed as 
           part of the "help -n" dumpfile debug output.  Also, as has been done
           with partial ELF diskdumps, if a compressed diskdump or compressed 
           kdump can be confirmed as a partial dump, the "[PARTIAL DUMP]" 
           indicator will follow the dumpfile name during initialization and by
           the "sys" command.  (anderson@redhat.com, oomichi@mxs.nes.nec.co.jp,
           indou.takao@jp.fujitsu.com, akiyama.nobuyuk@jp.fujitsu.com)

         - Support for xendumps of fully-virtualized x86_64 relocatable 
           kernels.  Without the patch, the physical base address was not
           being determined, and the session would fail during initialization 
           with the error message: " crash: vmlinux and core do not match!"
           (anderson@redhat.com)
 
         - Fix for 4.0-3.21 "BOOKE" ppc.c patch, which failed to compile.
           (antipov@ru.mvista.com) 
           
         - 4.0-3.21 to 4.0-3.22 incremental patch

           (04/10/07)


4.0-3.21 - Introduced support for upstream xensource ELF format dumpfiles,
           which will replace the current xendump format in xen 3.0.5.  The 
           new xen format uses ELF in a non-standard manner such that memory 
           contents are defined in section headers instead of the traditional 
           manner of using program headers.  Testing has been completed on
           paravirtualized x86, x86 PAE, x86_64 and ia64 dumpfiles.  Fully-
           virtualized dumpfiles have not been tested.  (anderson@redhat.com)

         - A number of "xencrash" (where the session is run against a xen-syms
           binary) fixes have been applied: 
            1) "bt" did not switch from the ia64 MCA stack to the vcpu stack.
            2) "bt" caused an infinite loop if ar_bspstore contained an illegal
                value.
            3) "bt" shows unnecessary unwind warning message. (ia64)
            4) "man log" caused crash to fail with a segmentation violation.
            5) "man log" did not have an example.
           (oda@valinux.co.jp)

         - Fix for "vtop" on x86 PAE kernels, which could abort upon reaching
           the PTE translation section, showing the error message: "vtop:
           cannot determine the swap location".  (anderson@redhat.com)
         
         - Fix for "vm -p" or "vtop" on 2.6 x86 PAE kernels, which could show
           incorrect swap offsets, because the swap type/offset encoding was
           moved to the high word of the 64-bit PTE.  (anderson@redhat.com)

         - Fix for "vm -p" on x86_64 kernels when a PTE referenced a swap
           location, it would show "(not mapped)" instead of the swap location.
           (anderson@redhat.com)

         - In current 2.6 kernels, it is now possible to recognize ppc BOOKE 
           processors, which is the current default in crash.  If the processor 
           is confirmed to not be BOOKE, then page table translation is done 
           differently.  (antipov@ru.mvista.com)

         - Fix for live system analysis of Ubuntu kernels due to a mismatch
           between /proc/version and the linux_banner string.  This was due 
           to an appendage to the linux_banner string in Ubuntu kernels.
           (asid@hp.com)

         - Fix for 2.6.21 kernels that fail during initialization with the
           message: "crash: invalid (optional) structure member offsets:
           zone_struct_free_pages or zone_free_pages".  This was due to the
           removal of the zone struct's "free_pages" member; instead the
           zone struct's "vm_stat[NR_FREE_PAGES]" value is used.  
           (anderson@redhat.com)

         - 4.0-3.20 to 4.0-3.21 incremental patch

           (03/16/07)


4.0-3.20 - Merged third round of "xencrash" patches, which allows a crash 
           session to alternatively be brought up against the xen-syms 
           binary instead of a vmlinux kernel.  This update introduces
           support for ia64.  (oda@valinux.co.jp)

         - Verified support of live system analysis of ia64 xen kernels, and
           removed unnecessary EFI memory verification warning message during
           their initialization.  (anderson@redhat.com)

         - Added gdb's "shell" command to the prohibited gdb command list, and 
           updated the "help output" page to describe shell escape usage.
           (anderson@redhat.com)

         - Fix for the x86 "bt" command for the 2.6.20 kernel, which has added
           the "xgs" field to the pt_regs structure.  Without this patch, the
           exception frame dump in "bt" would show invalid contents for several
           registers; the fix also shows the GS register contents.  
           (anderson@redhat.com)

         - Fix for the "mount" command for the 2.6.20 kernel to recognize the
           new "nsproxy" field in the task_struct and the contents of the
           nsproxy and mnt_namespace structures, in order to find the root 
           mount namespace.  Without the patch, the command would fail with:
           "mount: invalid kernel virtual address: 69  type: first list entry".
           (anderson@redhat.com)

         - Fix for the "files" command for the 2.6.20 kernel to handle the
           removal of the fdtable "max_fdset" member.  Without the patch, the
           command would fail with: "files: invalid structure member offset: 
           fdtable_max_fdset".  (anderson@redhat.com)

         - Fix for the "net -[sS]" command options for the 2.6.20 kernel to 
           handle the removal of the fdtable "max_fdset" member.  Without the 
           patch, the command would fail with: "net: invalid structure member 
           offset: fdtable_max_fdset".  (anderson@redhat.com)

         - Fix for the "vm" command for the 2.6.20 kernel to handle the removal
           of the file structure's "f_dentry" member, and its placement inside
           the embedded "path" structure.  Without the patch the command would
           fail with: "vm: invalid structure member offset: file_f_dentry".
           (anderson@redhat.com)

         - Fix for the "swap" command for the 2.6.20 kernel to handle the removal
           of the file structure's "f_vfsmnt" member, and its placement inside
           the embedded "path" structure.  Without the patch the command would
           fail with: "swap: invalid structure member offset: file_f_vfsmnt".
           (anderson@redhat.com)

         - 4.0-3.19 to 4.0-3.20 incremental patch

           (02/21/07)


4.0-3.19 - Fix for support of paravirtual x86 xendumps that were:

            1) created on host machines with greater than 4GB of memory, and 
            2) the active guest task at crash-time had been assigned a page
               directory page (cr3) with a machine address greater than 4GB.  

           If both of the above apply, the crash session would fail with one of 
           two error messages, either "crash: cannot read/find cr3 page", or 
           "crash: cannot create xen pfn-to-mfn mapping".  (anderson@redhat.com)

         - Fix for the "kmem -p [page-struct-address]" command construct, which
           would cause a segmentation violation when run on SPARSEMEM kernels.
           (anderson@redhat.com)

         - Added a new "struct -u" option, which indicates that the subsequent
           address argument is a user virtual address in the current context.
           This option could be used, for example, if a known kernel data 
           structure exists at user virtual address in the current context, 
           or if the debuginfo data of a user program were loaded into the 
           crash session via the gdb "add-symbol-file" command.
           (anderson@redhat.com)

         - Added new "rd -f" and "struct -f" options, which indicate that the 
           subsequent address argument is a dumpfile file offset.  These options
           could be used, for example, to print a known kernel data structure 
           that exists in the dumpfile header, or to simply dump data directly 
           from the dumpfile.  (anderson@redhat.com)

         - Cosmetic fix to prevent double-printing of "kmem -p" and "kmem -v"
           headers when those commands are passed multiple address arguments.  
           (anderson@redhat.com)

         - 4.0-3.18 to 4.0-3.19 incremental patch

           (02/07/07)

4.0-3.18 - Enhancement to the "mod" command to expand the number of section
           arguments to the internal "add-symbol-file" command issued to gdb to
           load the debug data for module objects.  On most architectures,  this
           allows the usage of the command construct "p [module-symbol-name]" to
           print out the module data structure in the same way that is done for 
           kernel proper data structure names.  (castor.fu@3pardata.com)

         - Two enhancements to significantly speed up the initialization of 
           crash sessions when running against multi-gigabyte xen kernels or 
           xendumps.  The cache of mfn-to-phys_to_machine_mapping page has been
           changed from a single-mfn-to-phys_to_machine_mapping page format to 
           storing a contiguous-range-of-mfns-to-phys_to_machine_mapping format.
           This benefit is primarily seen during the "gathering module symbol 
           data" phase.  The second change simply increases the size of the
           pfn-to-xendump-page-offset cache.  (anderson@redhat.com)

         - Fix for a segmentation violation during the "gathering task table 
           data" phase of initialization if the thread_info structure of the
           runqueue-advertised active task has been freed.  This has only ever
           been seen in a xendump created by "xm dump-core -L [guest-domain]".
           (anderson@redhat.com)

         - Cosmetic fix to prepend newlines to messages that happen to be 
           generated during any of the "please wait" segments of initialization.
           (anderson@redhat.com)

         - Addressed several compiler warnings when using -D_FORTIFY_SOURCE=2.
           Some are in gdb code that is never exercised, others were legitimate
           but would require impossible code paths, but one of them could
           result in runaway "help -t" output if the kernel was built without
           IKCONFIG.  (bwalle@suse.de)

         - Fix for the s390x "bt -f" command option, which was displaying the 
           stack as a sequence of 32-bit words which were dumped "backwards",
           i.e., at the wrong offset.  (krader@us.ibm.com)

         - 4.0-3.17 to 4.0-3.18 incremental patch

           (02/01/07)


4.0-3.17 - Two fixes for "dev -p" command option: 
            1) The head entry of the PCI device list was being skipped.
            2) For systems with no PCI devices, exit gracefully rather than
               failing the command due to the use of an invalid virtual 
               address.  
           (rachita@in.ibm.com, anderson@redhat.com)

         - Fix to recognize "linux_banner" symbol type change from 'R'
           to 'r' in 2.6.20-rc2 kernels.  Without the patch, the session 
           fails during initialization with the error message " WARNING: 
           invalid linux_banner pointer: 756e694c", and then "crash: vmlinux
           and vmcore do not match!  (vgoyal@in.ibm.com)

         - Fix to recognize "__per_cpu_start" and "__per_cpu_end" symbol
           type change from 'A' to 'D' in relocatable kernels.  Without
           the patch, SMP kernels running on uniprocessor systems may fail
           during initialization with the message "crash: cannot resolve 
           init_task_union".  (sachinp@in.ibm.com)

         - Fix for the xencrash "dumpinfo -t" command to properly cycle
           through the ELF_timeval structures for each cpu.
           (anderson@redhat.com)

         - Fix for x86_64 backtraces that may end prematurely at either a
           stale "schedule" or "schedule_timeout" reference when doing a 
           "bt" on an active task in a dumpfile.  (anderson@redhat.com)

         - Fix for a possible empty panic message in 2.6 kernels both during 
           initialization and when running the "sys" command, because of
           the change of the kernel panic() string from "Kernel panic: " to 
           "Kernel panic -- not syncing: ".  If the panic message was not
           recognized in another manner, such as by an oops message, by a 
           kernel BUG message, or sysrq-generated crash, the "PANIC:" status
           would be empty.  (anderson@redhat.com)
           (01/12/07)


4.0-3.16 - Recognize new XC_CORE_MAGIC_HVM xendump magic number, which in turn
           introduces support for xendumps of fully-virtualized ia64 kernels.
           (oda@valinux.co.jp)

         - Recognize an INVALID_MFN marker in the indexed mfn list of a xendump,
           and if found, fail the read attempt on the associated pfn.
           (oda@valinux.co.jp, anderson@redhat.com)
           (12/21/06)


4.0-3.15 - Introduced support for xendumps of fully-virtualized x86 kernels
           taken while running on an x86 Xen host (32-bit on 32-bit host).
           (anderson@redhat.com)

         - Introduced support for xendumps of fully-virtualized x86 kernels
           taken while running on an x86_64 Xen host (32-bit on 64-bit host).
           (anderson@redhat.com)

         - Introduced support for xendumps of fully-virtualized x86_64 kernels.
           (anderson@redhat.com)

         - Introduced support for xendumps of para-virtualized ia64 kernels.
           It should be noted that currently the ia64 Xen kernel does not 
           lay down a switch_stack for the panic task, so only raw "bt -t" 
           backtraces can be done on the panic task.  (anderson@redhat.com)

         - Introduced support for "xm save" dumpfiles of para-virtualized ia64
           kernels, which use a completely different format than that used for
           x86 and x86_64.  (anderson@redhat.com)

         - Additional support for the current kexec/kdump patch for Xen: 

            1) Merged second round of "xencrash" patches, which allows a crash
               session to be alternatively brought up against the xen-syms 
               binary instead of a vmlinux kernel.  (oda@valinux.co.jp)

            2) Using the xencrash feature above, the pfn_to_mfn_list_list value
               of any guest domain that was running when the dom0 or hypervisor 
               crashed can be determined; that pfn value can in turn be used
               as an argument to a new "--p2m_mfn [pfn]" crash command line 
               option.  That will allow a crash session to be run against any
               guest domain.  Therefore, with a single dom0/hypervisor vmcore,
               the following types of crash sessions may be initiated:

                 $ crash vmlinux-dom0 vmcore
                 $ crash xen-syms vmcore
                 $ crash --p2m_mfn [pfn] vmlinux-guest-#1 vmcore
                 $ crash --p2m_mfn [pfn] vmlinux-guest-#2 vmcore
                 $ ...

               (anderson@redhat.com)

            3) Fixed "help -n" debug output to properly display the contents
               of the new XEN_ELFNOTE_CRASH_INFO and XEN_ELFNOTE_CRASH_REGS 
               ELF note types.  (anderson@redhat.com)

         - Turn off the LKCD dumpfile-access "spinner" when "crash -s" is used.
           (castor.fu@3pardata.com)

         - Update to MODULES_IN_CWD code segment so that it will work on 2.6 
           kernels where modules end with ".ko".  This requires that kernel.c 
           is compiled with -DMODULES_IN_CWD.  (castor.fu@3pardata.com)

         - Support LKCD "map" files in lieu of standard System.map files.
           Without this patch, crash would fail with an error message of the
           sort: "crash: map.4: not a supported file format".  (bwalle@suse.de)

         - The ia64 PR_UNALIGN_NOPRINT and PR_FPEMU_NOPRINT prctl commands have
           been moved earlier in time, in order to prevent "unaligned access"
           messages when accessing ELF header contents.  (anderson@redhat.com)

         - The dlopen() call used by the "extensions" facility has been changed
           to use the RTLD_GLOBAL flag, so that symbols from an extension object
           will be visable to subsequently loaded modules.  (asid@hp.com)
           (12/20/06)


4.0-3.14 - Tentatively scheduled for RHEL5-GA
        
         - Added support for Magnus Damm's latest kexec/kdump patch for Xen.
           The ELF header of the vmcore, which is a full memory dump of the 
           dom0/hypervisor combination, contains a XEN_ELFNOTE_CRASH_INFO note 
           that contains the pfn_to_mfn_list_list value for dom0, allowing
           pfn-to-mfn translations may be made for crash analysis of the dom0
           linux kernel.  (anderson@redhat.com)

         - Added support for recognizing the zero-fill segments in ELF vmcore
           files created by the makedumpfile command from kdump /proc/vmcore files.
           Without this patch, ELF vmcore files generated by makedumpfile could
           only be used by gdb.  (anderson@redhat.com)
 
         - Updated the 4.0-3.4 patch that addressed the bogus kernel-/proc/version
           mismatch initialization failures using recent s390x vmlinux files that 
           contain an ASCII character just preceding the Linux version string.  
           That patch fixed the problem when the vmlinux file name was placed on 
           the crash command line; this version also fixes it when "crash" is 
           entered alone on the command line, and it has to search for the vmlinux
           file.  (anderson@redhat.com)
           (12/01/06)


4.0-3.13 - Adapted the "xencrash-0.2" patch described here: 

           https://www.redhat.com/archives/crash-utility/2006-November/msg00036.html

           This functionality consists of three inter-dependent parts, all of
           while are still under development:

            1) the kexec-tools user package
            2) the kdump kernel patch for Xen  
            3) the crash utility

           The end result will be a single crash binary that can be used with 
           either the Xen dom0 vmlinux kernel, or with the xen-syms hypervisor binary, 
           with the common vmcore created when either of those two entities crash.  
           (oda@valinux.co.jp, anderson@redhat.com)           

         - Fixed the initialization-time, and "sys" command, displays of the system
           memory size when memory nodes have holes.  Without this patch, more memory
           than what is installed may be displayed.  (anderson@redhat.com)
           (11/27/06)
      

4.0-3.12 - For 2.6.14 and later ia64 kdumps, taken either as a result of the
           INIT switch, or when an MCA exception has occurred, several problems
           needed to be addressed.  First, the "pseudo-task" that handles the 
           kdump operation due to an INIT or MCA was not being recognized as 
           the "panic" task.  Secondly, the backtraces of the per-cpu INIT 
           or MCA handling pseudo-tasks only went back as far as their entry 
           onto their own per-cpu stacks, and did not show the backtrace of 
           the task that was running on that cpu when the INIT or MCA event 
           occurred.  This version recognizes the pseudo-task that handles the
           kdump operation; and for each cpu, the active tasks' backtraces now
           also show a transition back to the task that was running on that cpu
           when the INIT or MCA event occurred.  (j-nomura@ce.jp.nec.com)
           
         - To address the need to display per-cpu variables, the "p"
           command has been modified to recognize "per_cpu__xxx" arguments
           when the kernel is SMP, in order to prevent the attempt to display
           the contents of a variable whose symbol value does not represent 
           the actual location of its data.  In that case, the data type of 
           the per-cpu variable will be displayed, followed by the addresses
           of each per-cpu instance.  Given that information, a proper command
           can be utilized in order to display the data.  For example, to look
           at the per-cpu buffer_head accounting for cpu 2:
           
             crash> p per_cpu__bh_accounting
             PER-CPU DATA TYPE:
               struct bh_accounting per_cpu__bh_accounting;
             PER-CPU ADDRESSES:
               [0]: c5405a80
               [1]: c540da80
               [2]: c5415a80
               [3]: c541da80
             crash> bh_accounting c5415a80
             struct bh_accounting {
               nr = 434,
               ratelimit = 2216
             }
           
           Note that "p" on the first command line above is optional, because
           whenever a data variable is entered alone, crash will recognize it
           as such, and pass it to the "p" command by default.  I had thought
           of putting this functionality into the "struct" command, but many
           of the per-cpu variables are pointers, arrays, etc..  So for the 
           non-structure cases, the "rd" command would be more appropriate, 
           or alternatively a cobbled-together gdb print command.  
           (anderson@redhat.com)

         - A consolidated cleanup and minor fixes patch has been applied to
           the experimental x86_64 dwarf CFI unwind facility.
           (rachita@in.ibm.com)

         - Also related to the experimental x86_64 dwarf CFI unwind facility, 
           fixed a problem where if a "set unwind on" was done, and followed 
           by a subsequent "set unwind off", then the "bt" output could either
           cause a segmentation violation, or display backtrace data that was
           different from the original.  (anderson@redhat.com)
           (11/15/06)


4.0-3.11 - Tentatively scheduled for RHEL5-B2

         - Updated fix for 2.6.18 x86_64 kernels to address the change in
           the IRQ-stack-to-process stack linkage; the fix introduced in
           4.0-3.9 could fail depending upon the crash session's display
           window size, due to a behind-the-scenes gdb line-wrap of text
           disassembly.  (anderson@redhat.com)
           (11/09/06)


4.0-3.10 - [Red Hat internal -- identical to 4.0-3.9]


4.0-3.9  - Tentatively scheduled as errata version for RHEL4-U5.
 
         - The current 2.6.18 x86_64 kernel has changed the IRQ-stack-to- 
           process-stack linkage, where until now the link value was a pointer
           to the exception frame on the process stack, but has been changed
           to point to a location on the process stack above the exception
           frame.  Because of that, after displaying the trace data from the
           IRQ stack, "bt" would then display an invalid exception frame, 
           which was reported as a "possibly bogus exception frame".
           (anderson@redhat.com)
 
         - Also in x86_64 kernels, fix for the "bt" command.  When the backtrace
           started on the NMI exception stack, it was displaying the correct
           exception frame data, but was erroneously reporting that it was a
           "possibly bogus exception frame".  (anderson@redhat.com)
 
         - And again in x86_64 kernels, fix for the "bt" command.  When making
           the transition from the IRQ stack back to the process stack, when
           the IRQ stack entry was made via the relatively new "call_softirq"
           entry point.  In that case, there is no exception frame on the
           process stack, because it's essentially just a cross-stack call
           from do_softirq().  However, a bogus exception frame was being 
           displayed, along with a "possibly bogus exception frame" message;
           and if the RIP value in the truly bogus exception frame happened
           to fall in the user virtual address range, the remainder of the
           process stack trace was not displayed at all.  (anderson@redhat.com)
 
         - Fix for 2.6.18-era ia64 DISCONTIGMEM kernels, which would fail 
           during initialization with the error message: "crash: invalid 
           (optional) structure member offsets: pglist_data_node_next or 
           pglist_data_pgdat_next".  (anderson@redhat.com)
 
         - Adapted Olivier Daudel's nifty enhancement to the "struct" command, 
           which allows the single "struct.member" argument to optionally be 
           expressed in a "struct.member[,member,member] format, in order to 
           display multiple members of a given structure.  This also applies to 
           the "union" and "*" commands, as all three functions have now been
           combined into one behind the scenes.  Fixed the display for applying 
           a minus count, and given that it opened up a the door to a number of
           entry errors, I also added additional error-catching/handling to avoid 
           the display of incorrect structure data.  
           (olivier.daudel@u-paris10.fr, anderson@redhat.com)
 
         - Fixed three sources of potential segmentation violations when using  
           the "bt" command when the experimental dwarf CFI unwind backtrace 
           facility was turned on.  (anderson@redhat.com)
 
         - Added a new machdep_init(POST_VM) call, which is currently only being
           used by the x86_64 architecture; it calls init_unwind_table(), which 
           has to be done after vm_init() in order to access the unwind tables 
           of kernel modules.  (anderson@redhat.com)

         - Prevent ia64 "floating-point assist fault" and "unaligned access"
           console messages by issuing PR_FPEMU_NOPRINT and PR_UNALIGN_NOPRINT
           prctl() settings.  (anderson@redhat.com)
           (11/02/06)
 
 
4.0-3.8  - Fix for the "irq" command when run on 2.6.17 and later kernels, which
           replaced the hw_interrupt_type structure with the irq_chip structure.
           Without the patch, the command would fail with the error message
           "irq: invalid structure member offset: irq_desc_t_handler".
           (rachita@in.ibm.com)
 
         - Phased in the first stage of support for the use of dwarf CFI data to
           produce accurate x86_64 back traces, and to eventually improve the 
           reliability of x86 back traces.  The code is very much still under 
           development, and is not turned on as of yet; for x86_64 only, its 
           usage can be toggled on and off with the set command, by entering 
           "set unwind on" or "set unwind off".  It will only work if dwarf CFI
           information exists in the kernel memory, or if the vmlinux file 
           contains an .eh_frame section.  Expect multiple iterations before
           this feature is ready for prime-time.
           (rachita@in.ibm.com, anderson@redhat.com)
           
         - Prevents stream of invalid "WARNING: possibly bogus exception frame" 
           messages during initialization when run against x86_64 xendump 
           dumpfiles created with the new "xm dump-core" facility.  
           (anderson@redhat.com) 
 
         - Fix for the "struct -o" option to print structure member offsets if 
           the member type is a function pointer.  (anderson@redhat.com)
           (10/20/06)
 
 
4.0-3.7  - Support for paravirtualized x86_64 RHEL4 Xen kernels, which require
           the use of unique hardwired kernel VM addresses, as well as a new 
           user vtop function.  Without the patch, crash would report several 
           read errors during invocation, and then eventually die with this
           message: "crash: cannot access phys_to_machine_mapping page".
           (anderson@redhat.com)
 
         - Fix for accessing user space stack addresses in ia64 kernels with
           3-level page tables.  This was a reqression introduced in 4.0-3.1,
           and would cause the new "ps -a" option to fail with an error message
           such as: "ps: cannot access user stack address: 60000fffffffbe28".
           Also, if the user stack address was given an the argument to the
           "vtop" command, it would indicate "(not mapped)".  
           (anderson@redhat.com)
  
         - Implemented a new "sig -g" option, which breaks down the signal 
           information into a common per-thread group section, followed by
           the signal information relevant to each task in the thread group.
           Added the capability of using the option via "foreach sig -g".
           (olivier.daudel@u-paris10.fr, anderson@redhat.com)
 
         - Update to allow the entry of multiple "list -s struct.member" 
           arguments, in order to display multiple members from each structure.
           Added the capability of entering a single "-s" option with multiple
           members entered in a comma-separated list, i.e., using the option
           format "-s struct.member1,member2,member3". 
           (olivier.daudel@u-paris10.fr, anderson@redhat.com) 
 
         - The refresh_hlist_task_table() and refresh_hlist_task_table_v2() 
           functions now recognize when the number of running tasks exceeds their
           internal table size, and realloc's task space as required.  Without
           the patch it would be possible to not access all tasks in a live 
           system if the number of tasks increased (rather dramatically) from the 
           time that the crash session started.  (anderson@redhat.com)
 
         - Added a new hash queue tool called hq_entry_exists().  The function 
           may be helpful in an extension, or future patch, to query for the 
           existence of an entry in the current hash queue.  (jmoyer@redhat.com)
           (10/13/06)
  
 
4.0-3.6  - Workaround for pre-2.6.17 kernels whose vmlinux file does not 
           contain debug information for the "pid_hash" array.  Without this
           patch, the crash session would fail during initialization with the 
           error message: "crash: cannot determine pid_hash array dimensions".
           This problem appears to be limited to kernels built with gcc
           version 4.0.0, which had a known regression that omitted debug
           information for uninitialized variables.  (anderson@redhat.com)
           (10/05/06)
 
 
4.0-3.5  - Implemented new "ps -a" option which, when available, displays the
           complete command line and environment variables of selected, or all,
           tasks.  For example:
           
            crash> ps -a automount
            PID: 3948   TASK: f722ee30  CPU: 0   COMMAND: "automount"
            ARG: /usr/sbin/automount --timeout=60 /net program /etc/auto.net
            ENV: SELINUX_INIT=YES
                 CONSOLE=/dev/console
                 TERM=linux
                 INIT_VERSION=sysvinit-2.85
                 PATH=/sbin:/usr/sbin:/bin:/usr/bin
                 LC_MESSAGES=en_US
                 RUNLEVEL=3
                 runlevel=3
                 PWD=/
                 LANG=ja_JP.UTF-8
                 PREVLEVEL=N
                 previous=N
                 HOME=/
                 SHLVL=2
                 _=/usr/sbin/automount
   
           Individual tasks may be selected in the same manner as always;
           "ps -a" alone lists all tasks.  (anderson@redhat.com)
 
         - Implmented new "ps -g" option, which lists tasks by thread group,
           for selected, or all, tasks.  For example, to display the tasks 
           in the thread group containing task c20ab0b0:
            
            crash> ps -g c20ab0b0
            PID: 6425   TASK: f72f50b0  CPU: 0   COMMAND: "firefox-bin"
              PID: 6516   TASK: f71bf1b0  CPU: 0   COMMAND: "firefox-bin"
              PID: 6518   TASK: d394b930  CPU: 0   COMMAND: "firefox-bin"
              PID: 6520   TASK: c20aa030  CPU: 0   COMMAND: "firefox-bin"
              PID: 6523   TASK: c20ab0b0  CPU: 0   COMMAND: "firefox-bin"
              PID: 6614   TASK: f1f181b0  CPU: 0   COMMAND: "firefox-bin"
      
           The thread group leader will be shown first, with the other threads
           indented.  Individual tasks may be selected in the same manner as
           always; "ps -g" alone lists all thread groups.  (anderson@redhat.com)
 
         - Fix for "timer" display; although the timer_list entries for each cpu
           are correct, the "TVEC_BASES[cpu]" output was displaying incorrect 
           addresses for each cpu's tvec_base_t structure.  (anderson@redhat.com)
           (10/02/06)
 
 
4.0-3.4  - Implemented support for x86_64 and ia64 compressed kdump dumpfiles 
           created by the makedumpfile command, which need to pass their
           respective physical address load locations in a kdump-specific
           dumpfile sub-header.  (oomichi@mxs.nes.nec.co.jp) 
 
         - Fix for the "timer" command on 2.6.17 and later kernels.  Without this
           patch, the command would spew out error messages of the sort:
 
             timer: invalid list entry: 0
             timer: ignoring faulty timer list at index 0 of timer array
 
           This was due to the kernel's tvec_bases data structures being moved
           out of the per-cpu memory regions, and replaced with just per-cpu 
           pointers to the data.  (anderson@redhat.com)
 
         - Fix for ia64 machines whose kernel's text and static data region 5
           segment is not loaded at physical address 64MB; live systems get 
           the physical load address from /proc/iomem, while kdump dumpfiles 
           contain the load address in the ELF header.  Without this patch, 
           the crash session would fail during initialization with a "crash: 
           invalid kernel virtual address: [address] type: xtime" error message.
           The physical address may still be forcibly set using the command line
           option "--machdep phys_start=[address]"  (anderson@redhat.com)
 
         - When using the "--machdep phys_start=[address]" on an ia64 machine, 
           an irrelevant error message indicating: "WARNING: invalid vm= option" 
           would be displayed.  (anderson@redhat.com)
 
         - Updated the ppc64 page size determination from always using 
           getpagesize() on the host machine to symbolically determining
           whether 64k page sizes are in use.  (sachinp@in.ibm.com)
 
         - Enhancement of the "sig" command to display the lists of both private
           and/or shared queued signals, if any.  (olivier.daudel@u-paris10.fr)
 
         - Adapted "mount [-n pid|task]" patch, which displays the mounted 
           filesystems with respect to the namespace of a given pid or task.
           (olivier.daudel@u-paris10.fr)
 
         - Fix for running crash without parameters on a live system that does
           not have a "/usr/src" directory, which would result in a segmentation
           violation.  (holzheu@de.ibm.com)
 
         - The /proc/version check against vmlinux "strings" output needed to be
           made aware that some other character may be adjacent to the "L" in the
           "Linux version..." string.  This would lead to erroneous "vmlinux and
           /proc/version do not match!" errors during initialization.
           (holzheu@de.ibm.com)
 
         - gdb-6.1.patch update for gdb-6.1/sim/ppc/debug.c to compile in SUSE
           build environment.  (olh@suse.de)
           (9/19/06)
 

4.0-3.3  - Addressed a number of issues associated with CONFIG_SPARSEMEM
           kernels and kernels using updated manners for the linkage of
           their pglist_data structures, and pointers to their mem_map arrays.
           (anderson@redhat.com)

         - Implemented "kmem -n" for CONFIG_SPARSEMEM kernels; in addition
           to the pgdat- and zone-related data command output, it also
           displays a list of the SPARSEMEM mem_sections.  Here is an
           example from an ia64:
             
            crash> kmem -n
            NODE    SIZE      PGLIST_DATA       BOOTMEM_DATA       NODE_ZONES
              0   2359296   e000000008c00000  a000000100749b70  e000000008c00000
                                                                e000000008c02400
                                                                e000000008c04800
                                                                e000000008c06c00
                MEM_MAP       START_PADDR  START_MAPNR
            e0000001040a3f00       0            0
            
            ZONE  NAME         SIZE       MEM_MAP      START_PADDR  START_MAPNR
              0   DMA        262144  e0000001040a3f00            0            0
              1   DMA32           0                 0            0            0
              2   Normal    2097152  e0000001048a3f00    100000000       262144
              3   HighMem         0                 0            0            0
            
            -------------------------------------------------------------------
            
            NR      SECTION        CODED_MEM_MAP        MEM_MAP       PFN
             0  e00000010409ff00  e0000001040a3f00  e0000001040a3f00  0
             1  e00000010409ff08  e0000001040a3f00  e0000001044a3f00  65536
             4  e00000010409ff20  e0000001038a3f00  e0000001048a3f00  262144
             5  e00000010409ff28  e0000001038a3f00  e000000104ca3f00  327680
             6  e00000010409ff30  e0000001038a3f00  e0000001050a3f00  393216
             7  e00000010409ff38  e0000001038a3f00  e0000001054a3f00  458752
             8  e00000010409ff40  e0000001038a3f00  e0000001058a3f00  524288
             9  e00000010409ff48  e0000001038a3f00  e000000105ca3f00  589824
            10  e00000010409ff50  e0000001038a3f00  e0000001060a3f00  655360
            11  e00000010409ff58  e0000001038a3f00  e0000001064a3f00  720896
            12  e00000010409ff60  e0000001038a3f00  e0000001068a3f00  786432
            13  e00000010409ff68  e0000001038a3f00  e000000106ca3f00  851968
            14  e00000010409ff70  e0000001038a3f00  e0000001070a3f00  917504
            15  e00000010409ff78  e0000001038a3f00  e0000001074a3f00  983040
            16  e00000010409ff80  e0000001038a3f00  e0000001078a3f00  1048576
            17  e00000010409ff88  e0000001038a3f00  e000000107ca3f00  1114112
            18  e00000010409ff90  e0000001038a3f00  e0000001080a3f00  1179648
            19  e00000010409ff98  e0000001038a3f00  e0000001084a3f00  1245184
            20  e00000010409ffa0  e0000001038a3f00  e0000001088a3f00  1310720
            21  e00000010409ffa8  e0000001038a3f00  e000000108ca3f00  1376256
            22  e00000010409ffb0  e0000001038a3f00  e0000001090a3f00  1441792
            23  e00000010409ffb8  e0000001038a3f00  e0000001094a3f00  1507328
            34  e0000001040a0010  e0000001010a3f00  e0000001098a3f00  2228224
            35  e0000001040a0018  e0000001010a3f00  e000000109ca3f00  2293760
            crash>
           (anderson@redhat.com)

         - Fix for "kmem -i" failure in CONFIG_SPARSEMEM kernels that would
           typically fail with the error message: "kmem: invalid kernel virtual
           address: 0  type: node_zones free_pages". (anderson@redhat.com)

         - Fix for "kmem -f" failure in CONFIG_SPARSEMEM kernels that would 
           typically fail with the error message: "kmem: invalid kernel virtual
           address: ab8  type: node_zones name".  (anderson@redhat.com) 

         - Fix for "kmem -f" failure in 2.6.17 kernels (possibly earlier) that
           would fail with the error message: "kmem: invalid structure member 
           offset: zone_zone_mem_map".  (anderson@redhat.com)

         - Fix for "kmem [address]" failure in 2.6.17 kernels (possibly earlier)
           that would fail with the error message: "kmem: invalid structure 
           member offset: zone_zone_mem_map".  (anderson@redhat.com)

         - Fix for "kmem -i" that resulted in a bogus "CACHED" page count
           value.  (anderson@redhat.com)

         - As an result of the last "kmem -i" fix, I've added a new "kmem -V"
           option that dumps the kernel's new vm_stat[] array contents by
           their enum values:

            crash> kmem -V
                   NR_ANON_PAGES: 38656
                  NR_FILE_MAPPED: 3116
                   NR_FILE_PAGES: 141106
                         NR_SLAB: 58605
                    NR_PAGETABLE: 1059
                   NR_FILE_DIRTY: 7
                    NR_WRITEBACK: 0
                 NR_UNSTABLE_NFS: 0
                       NR_BOUNCE: 0
                        NUMA_HIT: 86475467
                       NUMA_MISS: 0
                    NUMA_FOREIGN: 0
             NUMA_INTERLEAVE_HIT: 31523
                      NUMA_LOCAL: 86475467
                      NUMA_OTHER: 0
            crash>

          Interally, a new dump_vm_stat() function has been added to access 
          any of the items in the list.  (anderson@redhat.com)

        - Implemented support for relocatable x86_64 live kernels and kdump
          generated vmcores.  Without this patch, attempts to analyze those
          kernels would fail during initialization with the error message:
          "crash: vmlinux and vmcore do not match!"  (anderson@redhat.com)

        - Support for recognizing real-time signals in the "sig" command.
          (olivier.daudel@u-paris10.fr)

        - Fix for "sys -c" display of "sys_ni_syscall" entries that showed
          different system call names that have the same (W) symbol value
          as the (T) symbol "sys_ni_syscall".  For example:

           crash> sym -l | grep ffffffff802a38b6
           ffffffff802a38b6 (W) compat_sys_ipc
           ffffffff802a38b6 (W) compat_sys_keyctl
           ffffffff802a38b6 (W) compat_sys_sysctl
           ffffffff802a38b6 (W) ppc_rtas
           ffffffff802a38b6 (T) sys_ni_syscall
           ffffffff802a38b6 (W) sys_pciconfig_iobase
           ffffffff802a38b6 (W) sys_pciconfig_read
           ffffffff802a38b6 (W) sys_pciconfig_write
           ffffffff802a38b6 (W) sys_spu_create
           ffffffff802a38b6 (W) sys_spu_run
           ffffffff802a38b6 (W) sys_vm86
           ffffffff802a38b6 (W) sys_vm86old
           crash> 

          Depending upon the kernel, one of those symbols would be displayed
          instead of sys_ni_syscall.  (olivier.daudel@u-paris10.fr)

        - Fix for "sig" command where in later 2.6 kernels, the queued signal
          list at the end of the display would loop back on itself, repeatedly
          displaying the same queued signal(s).  (olivier.daudel@u-paris10.fr)
          (09/07/06)


4.0-3.2  - Enabled CONFIG_SPARSEMEM support for ia64 kernels; tested on
           RHEL5-alpha (2.6.17-1.2519.4.5.el5).  Without this fix, crash
           would fail during initialization with error message indicating:
           "crash: CONFIG_SPARSEMEM kernels not supported for this architecture"
           (anderson@redhat.com, dwilder@us.ibm.com)

         - Moved read_in_kernel_config() to just after the internal gdb 
           module gets initialized.  Without this fix, Xen kernels built
           with CONFIG_IKCONFIG would fail during initialization indicating: 
           "crash: gdb_interface: gdb not initialized?"
           (anderson@redhat.com, moriwaka@valinux.co.jp)

         - Implemented new s390/s390x command "s390dbf" command to print out
           kernel traces from the s390 debug feature (s390dbf). The debug 
           feature is an s390 kernel trace API which uses wraparound buffers
           to store trace records in memory.  Many of the s390 device drivers
           use this feature.  There is some documentation of the s390dbf in
           the kernel sources under /Documentation/s390/s390dbf.txt.
           (holzheu@de.ibm.com)

         - RHEL5-alpha kernel modules (only x86_64 confirmed) may possibly 
           fail to be loaded with the "mod" command due to dwarf2 errors 
           associated with the the split module.ko/module.ko.debug debuginfo
           facility used by RHEL kernels.  Bugzillas have been filed to 
           address those problems, but the crash utility's error-reporting 
           mechanism has beem modified to properly reflect that the internal
           gdb module has failed to load the kernel module's debug data.
           Without this fix, the "mod -[sS]" commands would silently return
           without loading the module data because the "add-symbol-file"
           operation inside the gdb module failed, did a longjmp(), and ended
           up back at the crash prompt.  That behaviour has been changed
           to report the module name and the gdb error like so: 

             crash> mod -S
             mod: /lib/modules/2.6.17-1.2564.1/kernel/drivers/scsi/scsi_mod.ko
             gdb add-symbol-file command failed
             crash>

           Note that this problem occurs in all post-RHEL4 kernels, i.e., 
           FC4, FC5, and now FC6 and RHEL5.
           (anderson@redhat.com)

         - Fix for runaway unkillable "repeat" command output that can happen
           when scrolling is turned off and the command that was entered is 
           bogus.  (anderson@redhat.com)

         - Fix for "struct structure.member address" output when the member
           is an array; additional members beyond the array contents would
           get displayed.  (anderson@redhat.com)

         - Fix to internal gdb module to properly handle relocatable kernel
           virtual addresses; this will be required for upcoming relocatable
           RHEL5 kernels required for the kexec/kdump facility.
           (anderson@redhat.com)

         - Combined kernel_init(PRE_GDB) and kernel_init(POST_GDB) into a 
           single call to kernel_init() that is done after gdb is initialized;
           verify_version() now called by kernel_init().  This is just a code
           re-work, and does not change any functionality. (anderson@redhat.com)
           (8/23/06)


4.0-3.1  - Fix to address 2.6.18 and later Fedora 2.6.17-based kernel data
           structure name change from "runqueue" to "rq".  This would cause 
           crash to fail during initialization with a "crash: cannot determine
           idle task addresses from init_tasks[] or runqueues[]" message, 
           followed by a red herring message: "crash: cannot resolve 
           init_task_union".  (haren@us.ibm.com)

         - Added 4-level pagetable support for ia64.  Since this is based
           upon whether the kernel was built with CONFIG_PGTABLE_4, the
           determination of whether the crash utility uses 4-level page
           tables is based upon one of two possibilities: the "automatic"
           manner depends upon the kernel also being configured with
           CONFIG_IKCONFIG; otherwise it will require the commmand line
           option "--machdep vm=4l".  (troy.heber@hp.com)

         - Leveraging Troy Heber's addition of code to dig out and uncompress
           in-kernel CONFIG_IKCONFIG data, a new "sys config" command option has
           been added, which dumps all of the kernel configuration data.
           (anderson@redhat.com, troy.heber@hp.com)

         - Also leveraging the new CONFIG_IKCONFIG data access, the value of HZ
           can now be absolutely determined by reading CONFIG_HZ.  If the config
           data is not available, then the current use of the HZ #define will
           be replaced by the use of sysconf(_SC_CLK_TCK) to account for the
           upcoming removal of HZ from glibc header files.  
           (anderson@redhat.com, olh@suse.de)

         - Added a new "--cpus [number]" command line option to work around any
           situations where the number of cpus cannot be correctly determined. 
           This is unlikely to ever be needed, but it was necessary for an ia64 
           kdexec/kdump development kernel issue that has been addressed. 
           However it's been left in place as a work around in case the same 
           thing occurs due to some other circumstance.  (anderson@redhat.com)
           (8/04/06)


4.0-2.33 - Fix for possible compilation error in x86_xen_kdump_load_page_PAE()
           function in 4.0-2.32 version of x86.c.  (anderson@redhat.com)
           (7/13/06) 


4.0-2.32 - Implemented and tested code to create the Xen kdump p2m table from
           the mfn value found in the "pfn_to_mfn_frame_list_list" member
           contained within the shared per-domain "arch_shared_info" structure,
           which is contained within the architecture-neutral "shared_info"
           structure.  However, the use of this capability will require that:

             (1) the Xen kdump implementation pass this mfn value in the vmcore
                 ELF header, and
             (2) the crash utility will need additional updating to access this
                 value from the vmcore ELF header.

           The current test version of the Xen kdump code passes the dom0 cr3
           value in the ELF header, but that only works for Xen kernels with
           writable pagetables.  Using the pfn_to_mfn_frame_list_list mfn will 
           work for both writable- and shared-pagetable Xen kernels.  
           (anderson@redhat.com)
  
         - Support for kernels with no vmalloc addresses, i.e., with an empty 
           "vmlist", fixing an initialization-time session failure indicating:
           "crash: invalid kernel virtual address: 0  type: first vmlist addr"  
           (moriwaka@valinux.co.jp)
  
         - Fix that allows the "wr" command to accept at 64-bit value.
           (castor.fu@3pardata.com)
  
         - Fix for "vtop" on user/kernel virtual addresses that showed the page
           offset value on the "PAGE:" output line on x86 PAE kernels.
           (anderson@redhat.com)
  
         - Added "rd -x" option to avoid the display of the ASCII translation at
           the end of each line.  (anderson@redhat.com)
  
         - Fix for unnecessary double-printing of the "mount" command header 
           when a directory argument is referenced by two different vfsmounts.
           (harihare@vnet.ibm.com, shenlinf@cn.ibm.com)
  
         - Fix to recognize equivalent directory arguments to the "mount" 
           command, i.e., "/boot" is the same as "/boot/".  
           (shenlinf@cn.ibm.com) 
  
         - Fix for "swap" command that dropped "/dev" from swap device pathnames
           in 2.6 kernels.  (shenlinf@cn.ibm.com)
  
         - Fix for potential segmentation violation when running "bt -f" command 
           on s390 and s390x.  (holzheu@de.ibm.com)
  
         - Added a "rd -m machine-address" option to read Xen machine 
           addresses if they are accessible; also a general cleanup of the
           m2p functionality.  (anderson@redhat.com)
           (7/12/06)


4.0-2.31 - Bumped crash-internal NR_CPUS for x86 and ia64; added a warning
           message to "recompile crash" and forced an initialization failure 
           when the kernel's configured NR_CPUS is greater than the maximum
           allowed NR_CPUS value compiled into crash.  
           (maneesh@in.ibm.com, anderson@redhat.com)    

         - Fix for initialization failure indicating a kernel/memory-source
           mismatch when x86 kernel configures its physical memory start
           address higher than the traditional 1MB starting point.
           (anderson@redhat.com)

         - Fix for kernels that have replaced the "system_utsname" data 
           structure with contents of the "init_uts_ns" data structure.  
           This fixes a "crash: cannot resolve system_utsname" initialization
           failure.  (pbadari@us.ibm.com, anderson@redhat.com)

         - Fix for large LKCD dumpfiles that resulted in an initialization
           time failure indicating "fixme, need to add more zones (ZONE_ALLOC)".
           When statically-defined ZONE_ALLOC value is too small, the fix
           expands the zone size dynamically.  (indou.takao@jp.fujitsu.com) 

         - Fix for "kmem -i" failure when the "all_bdevs" block_device list 
           is empty.  Part of the command output would be displayed, followed by
           "kmem: invalid kernel virtual address: 0  type: inode buffer".
           (anderson@redhat.com)

         - First pass at supporting a Xen hypervisor kexec/kdump vmcore as the 
           dumpfile format for the dom0 vmlinux.  Developed/tested OK on an x86 
           vmlinux/vmcore set supplied by horms@verge.net.au.  Code for x86_64
           is in place, but untested.  (anderson@redhat.com)

         - Also in place, but untested, is initial support for Xen x86 PAE
           kernels.  (anderson@redhat.com)
           (6/27/06)


4.0-2.30 - RHEL4-U4 version.  RHEL3-U8 will be (indentical) version 4.0-2.29.

         - Fix for x86_64-only "vm -p" failure due to "pml page" read error
           on kernels with 3-level user page tables; regression was introduced 
           by x86_64 Xen support in 4.0-2.24.  (anderson@redhat.com)

         - Fedora, and future RHEL, build procedure requires the removal of
           the inclusion of certain kernel header files; removed inclusion(s)
           of page.h, list.h, and segment.h.  (anderson@redhat.com)
           (6/06/06)


4.0-2.24 - Fix for 2.6.17 kernels that do not use "pgdat_list" memory node
           list header, which would cause crash to fail during initialization
           with a "crash: cannot resolve: pgdat_list" error message.
           (anderson@redhat.com)

         - Fix for 2.6.17 kernels that have re-worked the kernel pid_hash
           handling, which would cause crash to fail during initialization
           with a "crash: cannot determine pid_hash array dimensions" error
           message.  (anderson@redhat.com)

         - If the vmlinux file and /proc/version do not match, and crash tries
           to find an appropriate System.map file to use for symbol addresses,
           a new "WARNING: vmlinux and /proc/version do not match!" message
           will be displayed.  Note that the System.map file that crash finds
           will be appropriate for data symbols, but may not necessarily be
           correct for text regions.  When this happens, kernel text disassembly
           may be incorrect, and this in turn leads to other problems, such as
           incorrect back-tracing.  (anderson@redhat.com)

         - Fix for recent 2.6 kernels "sys" UPTIME display and for "ps -t" RUN 
           TIME displays due to change to HZ value.  (anderson@redhat.com)

         - Continued Xen support: this version runs on live x86_64 xen0 and xenU
           kernels, on x86_64 xenU core dumps, and x86_64 xenU "xm save" files.
           As is the case for x86 Xen, this support is only for x86_64 kernels 
           with writable page tables.  (anderson@redhat.com)

         - Fix for x86_64 IS_LAST_PML4_READ() macro, which (harmlessly) never 
           worked, but caused the PML4 page to be re-read each time.  Added a 
           per-arch clear_machdep_cache() function for processors needing to do
           their own virtual-to-physical page table cache clearing; so far only 
           ppc64 and x86_64 need it for the top-most of their 4-level page table
           pages.  (anderson@redhat.com)
           (5/01/06)

4.0-2.23 - Fix for "kmem -[sS]" command in 2.6.15 kernels which introduced
           per-NUMA node slab chains.  Without this patch the command fails
           with a "kmem: invalid structure member offset: kmem_cache_s_lists"
           error message.  (sharyath@in.ibm.com)

         - Fix for this initialization error on 2.6.16 kernels indicating:
           "crash: cannot determine idle task addresses from init_tasks[]
           or runqueues[]" followed by "crash: cannot resolve init_task_union"
           error messages.  This was due to the introduction of a runqueue.cpu
           member that conflicted with an old cpu member in RHEL3-specific O(1)
           scheduler code.  (anderson@redhat.com)
    
         - Fix for "kmem -i" in newer 2.6 kernels where the new ZONE_DMA32 bumps 
           up the value of ZONE_HIGHMEM, causing a potential segmentation 
           violation.  (anderson@redhat.com)

         - Fix for "kmem -i", where the PG_slab bit determination has been fixed
           so that the correct number of slab pages is displayed. 
           (anderson@redhat.com)

         - Fix for "swap" command and "kmem -i" option on 64-bit 2.6.15 kernels 
           which could fail with a crash internal buffer dump followed by these
           messages: "swap: cannot allocate any more memory!" or "kmem: cannot 
           allocate any more memory!".  This was due to the swap_info_struct.max
           member being downsized from a long to an int.  (anderson@redhat.com)

         - Continued Xen support: this version runs on live x86 xen0 and xenU 
           kernels, on xenU core dumps, and on xenU "xm save" files.  This 
           support is for x86 kernels with writable page tables only.  Minimal
           support for running on live x86_64 xen0 kernels with writable page
           tables is also in place, but does not allow access to user virtual 
           memory as of yet.  (anderson@redhat.com)
           (4/12/06)


4.0-2.22 - Incorporated initial patch-set to implement support for kernels built
           with CONFIG_SPARSEMEM.  (dwilder@us.ibm.com)  

         - Fix for post-2.6.15 ppc64 kernels to use cpu_online_map when perusing
           the paca array for the per_cpu_offsets.  (haren@us.ibm.com)

         - Fix for ppc64 "bt" command for active tasks that were running in
           user space at the time of crash.  (haren@us.ibm.com) 

         - Fix to remove dependencies upon any kernel header files so as to 
           allow crash to build in a Ubuntu environment.  (aquynh@gmail.com)

         - Fix size of x86_64 "cpu_khz" variable to match that of the kernel.
           (sharyath@in.ibm.com) 

         - Created framework for support of Xen kernel dumpfiles and live Xen 
           kernels; this is going to be a long-period work-in-progress affair, 
           and the code added in this release is being done now primarily to aid
           in future patch integration efforts.  (anderson@redhat.com)
           (3/23/06)


4.0-2.21 - Fix to recognize post-2.6.15 ppc64 kernels moving the per_cpu_offsets
           to the "paca" structure.  Without this patch, crash fails with the 
           following error messages: "crash: cannot determine idle task addresses
           from init_tasks[] or runqueues[]" and "crash: cannot resolve 
           init_task_union".  (pbadari@us.ibm.com)

         - Incorporated a patch containing ppc64 specific changes when reading 
           kdump vmcores.  Kdump vmcores contain pt_regs for all cpus in the ELF
           header, so they are read from there rather than from the active tasks'
           kernel stacks; also, the registers contents are printed before any 
           active task backtrace.  (haren@us.ibm.com)

         - If pglist_data.node_mem_map structure member does not exist, as in a
           ppc64 kernel built with CONFIG_SPARSEMEM, print an init-time warning 
           message instead of failing with "crash: invalid structure member
           offset: pglist_data.node_mem_map" message.  (haren@us.ibm.com, 
           anderson@redhat.com)
           (2/16/06)


4.0-2.20 - Fix to recognize 2.6.16 change that removed the x86_64 cpu_pda[]
           array of x8664_pda structures and replaced it with a _cpu_pda[]
           array of pointers to those structures.  Without the patch, crash 
           failed during initiatization of 2.16.16 x86_64 kernels with a 
           "crash: cannot resolve cpu_pda" error.  (rachita@in.ibm.com)

         - Added a minor enhancement to the "list" command to allow the
           "start" argument to also be an (expression) that evaluates to the
           address of the starting list_head; previously it only allowed
           a symbol or a virtual address.  (anderson@redhat.com)
           (2/03/06)


4.0-2.19 - Fix for the "bt" command on ia64 kernels with 64K page size.
           (1/11/06)


4.0-2.18 - Fix for the "files" command for 2.6.14 and later kernels, in which 
           the files_struct data structure contains the new fdtable data 
           structure.  (rachita@in.ibm.com) 

         - Fix for an "invalid lvalue in assignment" compile-time error 
           generated from gdb-6.1/bfd/coff-alpha.c that prevents the embedded 
           gdb from building with newer compilers.  (troy.heber@hp.com)
           (1/5/06)


4.0-2.17 - Fix to resurrect LKCD version 8 support, inadvertently broken in 
           4.0-2.15. (troy.heber@hp.com)

         - Fix for "net -S" failures in certain 2.6 kernels that failed with
           "net: cannot determine what an inet_sock structure is" message;
           shows embedded sock structure instead of failing. (anonymous donor)

         - Fix for erroneous "net -s" source/destination address and port
           values in certain 2.6 kernels; added "net -s" source/destination 
           address and port values for IPv6 sockets.  (anderson@redhat.com)
           (12/16/05)


4.0-2.16 - Fix for the x86_64 backtrace code to search all of the exception
           stacks for the origin of the active tasks' backtrace when the
           information is not available in the dumpfile header.  Up until now,
           the search was made in the process stack, the per-cpu IRQ stack,
           and the per-cpu NMI exception stack; this patch looks at all 3 
           exception stacks in 2.4 kernels (NMI, STACKFAULT and DOUBLEFAULT), 
           and all 5 exception stacks in 2.6 kernels (NMI, STACKFAULT, 
           DOUBLEFAULT, DEBUG and MCE).

         - Fix to remove erroneous warning message re: the task cpu not being 
           the same as the IRQ or exception stack cpu, which was displayed when 
           doing a non-context-sensitive "bt -E" on an x86_64.
           (12/12/05)


4.0-2.15 - Applied Kurt Rader's (kdrader@us.ibm.com) patch for SUSE SLES 9 
           "bigsmp" kernel LKCD dumpfiles, to fix "conflicting page" abort
           caused by a dumpfile header that is larger than the formerly 
           hard-wired header size.

         - Fix for ppc64-only segmentation violation when running "bt" on the
           panic task when run against a dumpfile created by the diskdump 
           facility's new compressed format. 
           (12/02/05)


4.0-2.13 - Adapted Takao Indoh of Fujitsu's patch for determining proper size
           of the ia64_init_stack; fixes empty ia64 "bt -a" output for cpu 8 and
           above for diskdumps generated via OS_INIT.

         - Applied a patch to address a "net -s" error due to the inet_opt 
           structure being dropped between 2.6.10 and 2.6.11, which led to a
           "net: invalid structure member offset: inet_opt_daddr" failure.

         - Made the initialization-time rule such that if "bt -O" is contained
           in any or all of the 3 possible initialization-time input files 
           ($HOME/.crashrc, ./.crashrc, or "-i inputfile" files), the setting
           will remain idempotent.  Fixed the redundant running of $HOME/.crashrc
           and ./.crashrc files if they are the same file.

         - Added a gdb work-around/hack for ia64 initialization-time warning
           "WARNING: cannot determine unw.tables offset" on rebuilt RHEL3 ia64
           kernels that would prevent "bt" from working.

         - Backed out 4.0-2.11 x86_64 pseudo-backtrace patch to show in-kernel
           exception frame RIP and RSP values as a unique frame following the
           register dump; instead, the exception RIP address is translated
           and displayed prior to the register dump.
           (11/23/05)


4.0-2.12 - Update to diskdump page_desc struct, required for ongoing support
           of the diskdump facility's compression feature, currently under 
           development.

         - Applied patch from Ken'ichi Ohmichi of NEC to prevent a segmentation
           violation during a "bt -f" on an x86_64 task that had taken a NMI
           during cpu_idle().

         - Adapted Badari Pulavarty's patch for recognition of recent 2.6.14
           kernel structure/member name changes: mm_struct._rss to _file_rss,
           and the kmem_cache_s structure's renaming to kmem_cache.  Without
           the patch, crash sessions would fail during initialization with an
           "crash: invalid structure member offset: kmem_cache_s_num" error,
           and the "ps" command would fail with a "ps: invalid structure member
           offset: mm_struct_rss" error.
           (11/15/05)


4.0-2.11 - Adapted a number of proposed patches:

           - Badari Pulavarty of IBM's implentation of support for 2.6.14
             ppc64 kernel's use of 4-level page tables.
           - Added a new "extensions" sub-directory for collecting crash
             command extension libaries; initially populated with the sample
             "echo.c" from the extend help page, along with a device-mapper
             related "dminfo.c" module from NEC.
           - Castor Fu of 3PAR's implementation of support for LKCD version 10,
             as well the handling of single-bit errors in LKCD compressed 
             pages by trying out all possible single-bit errors.  Also his
             fixes for better recognizing -fomit-frame-pointer kernel builds,
             a stronger defense against potential bogus processor numbers
             associated with tasks in dumpfiles, and a fix to re-allow crash
             builds for gcc 2.x compilers.

         - Fix for potential "vmcore: initialization failed" fatal error during
           initializaton when using more than just a vmlinux and vmcore command
           line arguments.