• SOLARIS
• pstack core or
• /opt/SUNWspro/bin/dbx program core
• (dbx) thread -info t@1 ==> show the thread information
• (dbx) where ==> show the thread stack
• AIX
• The syscorepath utility can be used to specify a single system-wide directory in which all core files of any processes are saved. The syntax for this command is: syscorepath -p alternate_directory.
• To set the OS for full core dumps and files to unlimited
• Set the ulimit setting for core dumps to unlimited: ulimit -c unlimited.
• Set the ulimit setting for core files to unlimited: ulimit -f unlimited.
• Set Smit to use full core dumps either by starting smit and setting: System Environments -> Change/Show Characteristics of Operating System -> Enable Full CORE dump to "TRUE", or by using the command chdev -l sys0 -a fullcore='true' as root.
• Ensure that the current working directory has enough disk space available to write the core file. You can redirect AIX core files to alternative locations using a symbolic link. To do this, you must create a link from the current working directory of the process to an alternative directory where there is a file called "core". After a full core file has been generated and located, you must rename that file to prevent any other core file, that is generated in the same directory, from overwriting it.
• Run SMIT (or smitty) --> System Environments --> Change / Show Characteristics of Operating System --> Use pre-430 style CORE dump. Set the last option to true and then adb / dbx will be able to read new core files produced (the core file must be created under new settings).
• LINUX
• gdb program core
• (gdb) where
• HPUX
• adb programname core
• (adb) $c

1. If it is related to any performance or memory issue
1. vmstat 1
2. AIX
1. topas
2. prtconf
3. SOLARIS
1. prtdiag
2. prtconf/proc/meminfo
3. prstat
4. LINUX
1. top -d 1 |grep processname
2. cat /proc/cpuinfo
3. cat /proc/meminfo
4. cat /proc/version
5. HPUX
1. grep -i Physical /var/adm/syslog/syslog.log

1. Add the line set -vx in the beginning of the script. The shell script will then start debug output. You can disable the debug message by adding set +vx from that point onwards debug messages will be disabled
   
2. Or doing sh -x shellscriptfilename (best option out the lot)
   
3. This is similar to the first one except it will be applicable for the complete shell script. By adding

   #!/bin/sh -vx

   to the beginning of the shell script
   

#!bin/sh

PROCCHECKDELAY=1
RUNCHECKDELAY=2
PROCNORUN=
PAGEROWS=80
PROCNAME=$1
PROCID=

if [ $# -ne 1 ]
then
 echo give program name to watch;
 exit 1;
fi

stat_process()
{
 PROCID=
 _mypid=
 while [ true ]
 do
   _mypid=`ps -e 2>/dev/null|grep $1|head -n1|awk '{ print $1 }'`;
   if [ ! -z "$_mypid" ]
   then
     PROCNORUN=
     break;
   fi
   if [ -z "$PROCNORUN" ] ;
   then
     PROCNORUN=yes
     echo "$1 is not running"
     echo "recheck after $RUNCHECKDELAY seconds";
     sleep $RUNCHECKDELAY;
   fi
 done
 PROCID=$_mypid;
}

i=$PAGEROWS;
while true;
do
stat_process $PROCNAME;

if [ $i -eq $PAGEROWS ] ;
then
 i=0;
 UNIX95= ps -p $PROCID -o vsz,pid,ruser,rss,pcpu,pmem,args 2>/dev/null
else
 UNIX95= ps -p $PROCID -o vsz,pid,ruser,rss,pcpu,pmem,args 2>/dev/null|tail +2
fi
if [ ! -z "$PROCID" ]
then
   i=`expr $i + 1`;
fi
sleep $PROCCHECKDELAY;
done;

1. Observe the complete stack trace esp multi-threaded programs.

   $>pstack core|c++filt




2. Deploy sample programs to test the possible operation that is causing issue rather than trying to debug the complete operation.



3. Use dtrace to find out the program trace just before when the issue happens

   $dtrace -n 'pid$target:a.out::entry { ustack() }' -p PID




4. Use DBX to figure out the variable dump, will have to use debug build
   

   
              $> dbx program core
              (dbx) threads
              (dbx) thread t@3900
              (dbx) dump 
         

   
   



5. Use stubs for the suspected list of functions
   



6. Include log messages so as to print out the input and output data values
   



7. Monitor the process resource closely during the test eg. mem, cpu usage
   



8. Run the program within DBX enabled with checking of memory access, leaks, or usage

   (dbx)>check -all

   
   



9. Use coreadm to manage core file dumps. This will help you from analyzing the previous coredumps as well if configured to dump using pid as one of the file name.