1. Observe the complete stack trace esp multi-threaded programs.

   $>pstack core|c++filt




2. Deploy sample programs to test the possible operation that is causing issue rather than trying to debug the complete operation.



3. Use dtrace to find out the program trace just before when the issue happens

   $dtrace -n 'pid$target:a.out::entry { ustack() }' -p PID




4. Use DBX to figure out the variable dump, will have to use debug build
   

   
              $> dbx program core
              (dbx) threads
              (dbx) thread t@3900
              (dbx) dump 
         

   
   



5. Use stubs for the suspected list of functions
   



6. Include log messages so as to print out the input and output data values
   



7. Monitor the process resource closely during the test eg. mem, cpu usage
   



8. Run the program within DBX enabled with checking of memory access, leaks, or usage

   (dbx)>check -all

   
   



9. Use coreadm to manage core file dumps. This will help you from analyzing the previous coredumps as well if configured to dump using pid as one of the file name.
   

• Linux
  
• gprof
  
• GNU gprof
  
• Speed your code with the GNU profiler
• callgrind
• Callgrind: a heavyweight profiler
• ltrace/strace/mtrace
• These are low level system calls, library calls and memory tracers
• Overview
• Systemtap
• Solaris
• dtrace - my fav
  
• BigAdmin: DTrace
• DTrace Review - Really Cool Video - A must see one!
  
• IBM - AIX
  
• truss/trace
  
• i haven't personally used these, still has to do some more research on these
• Probevue
  
• HPUX
• if lucky you will find some here
  
• Porting And Archive Centre for HP-UX

#!bin/sh

PROCCHECKDELAY=1
RUNCHECKDELAY=2
PAGEROWS=80
PROCNAME=$1
PROCID=

if [ $# -ne 1 ]
then
        echo give program name to watch;
        exit 1;
fi

stat_process()
{
    PROCID=
    _mypid=
    while [ true ]
    do
        _mypid=`ps -e 2>/dev/null|grep $1|head -n1|awk '{ print $1 }'`;
        if [ -z "$_mypid" ]
        then
            echo "$1 is not running"
        else
            break;
        fi
        echo "recheck after $RUNCHECKDELAY seconds";
        sleep $RUNCHECKDELAY;
    done
    PROCID=$_mypid;
}

i=$PAGEROWS;
while true;
do
stat_process $PROCNAME;
if [ $i -eq $PAGEROWS ] ;
then
        i=0;
        UNIX95= ps -p $PROCID -o vsz,pid,ruser,args 2>/dev/null
else
        UNIX95= ps -p $PROCID -o vsz,pid,ruser,args 2>/dev/null|tail +2
fi
if [ ! -z "$PROCID" ]
then
    i=`expr $i + 1`;
fi
sleep $PROCCHECKDELAY;
done;