GenIDLEST Performance Analysis with PerfSuite

The following graphics provide some examples of collaborative work in performance engineering and analysis on Linux clusters using PerfSuite.

The application driver is GenIDLEST (Danesh Tafti, Virginia Tech), a computational fluid dynamics application that is being developed and tuned for use on Pentium III, Pentium 4, Xeon, Itanium, and Itanium 2 Linux clusters located at the University of Illinois/NCSA and Virginia Tech. GenIDLEST is used for academic research in addition to use by community and corporate researchers.

GenIDLEST development lead Tafti has been working with colleagues to isolate, diagnose, and improve performance-related issues of this application in the Linux cluster environment. Several performance tools have been employed to identify problem areas and to help point the way towards improvement.

GenIDLEST, PerfSuite, and PAPI

Reliably measuring performance is key to the GenIDLEST performance analysis effort. The team has measured the application with processor hardware performance counters using the PerfSuite hardware performance support library and the PAPI library. PAPI, which in turn uses IA-64 perfmon performance monitoring support, allows the team to effectively "X-ray" what is occurring in the Itanium processor as GenIDLEST executes. This makes it possible to accurately assess and improve processor utilization by examining and interpreting reports like the following (postprocessed for display by PerfSuite's psprocess utility):

PerfSuite 1.0 summary for execution of gen.ver2.3.inte (PID=9867, domain=user) Based on 800 MHz -1 GenuineIntel 0 CPU CPU revision 6.000 Event Counter Name Counter Value =================================================================================== 0 Conditional branch instructions mispredicted............. 3006093956 1 Conditional branch instructions correctly predicted...... 32974709880 2 Conditional branch instructions taken.................... 26952022279 3 Floating point instructions.............................. 44525980237 4 Total cycles............................................. 353262206234 5 Instructions completed................................... 489764680025 6 Level 1 data cache accesses.............................. 56390921533 7 Level 1 data cache hits.................................. 41911206947 8 Level 1 data cache misses................................ 14615753570 9 Level 1 load misses...................................... 17611912424 10 Level 1 cache misses..................................... 17597248300 11 Level 2 data cache accesses.............................. 53158617899 12 Level 2 data cache misses................................ 8440205387 13 Level 2 data cache reads................................. 43528651785 14 Level 2 data cache writes................................ 10240563775 15 Level 2 load misses...................................... 3615923337 16 Level 2 store misses..................................... 667575973 17 Level 2 cache misses..................................... 8529931717 18 Level 3 data cache accesses.............................. 3826843278 19 Level 3 data cache hits.................................. 2799591986 20 Level 3 data cache misses................................ 999714206 21 Level 3 data cache reads................................. 3573882130 22 Level 3 data cache writes................................ 171800425 23 Level 3 load misses...................................... 944624814 24 Level 3 store misses..................................... 49427000 25 Level 3 cache misses..................................... 1024569375 26 Load instructions........................................ 84907675686 27 Load/store instructions completed........................ 95346092870 28 Cycles Stalled Waiting for memory accesses............... 140032176122 29 Store instructions....................................... 10267472354 30 Cycles with no instruction issue......................... 67247126931 31 Data translation lookaside buffer misses................. 8365029 Statistics =================================================================================== Graduated instructions/cycle................................... 1.386406 Graduated floating point instructions/cycle.................... 0.126042 Graduated loads & stores/cycle................................. 0.269902 Graduated loads & stores/floating point instruction............ 2.141359 L1 Cache Line Reuse............................................ 5.523515 L2 Cache Line Reuse............................................ 0.731682 L3 Cache Line Reuse............................................ 7.442618 L1 Data Cache Hit Rate......................................... 0.846708 L2 Data Cache Hit Rate......................................... 0.422527 L3 Data Cache Hit Rate......................................... 0.881553 % cycles w/no instruction issue................................ 19.036037 % cycles waiting for memory access............................. 39.639729 Correct branch predictions/branches taken...................... 1.000000 MFLOPS......................................................... 100.833839

Pentium (IA-32) and Itanium (IA-64) hardware performance counter summary information provides insight into the effective use of the processor resources. Raw event counts in addition to derived statistics (here, presented in a similar manner to the IRIX perfex utility) are very useful in assessing the effects of code, data, and algorithmic modifications as well as in determining the compiler's ability to emit well-optimized code.

GenIDLEST, OptView, and the Intel IA-64 compiler

GenIDLEST is used frequently as an application driver for the development of tools to assist not only with this effort, but that are of general use. One example of this work is the PerfSuite OptView tool, which is being developed to assist in understanding the effectiveness of compiler optimizations.

An example of OptView screens that show compiler optimization effectiveness. In this case, a loop from within GenIDLEST's solvers failed to software pipeline and needs attention/reworking in order to enable more efficient execution on the Itanium processor.

GenIDLEST and VProf

GenIDLEST performance analysis includes other tools developed externally that are appropriate to assist in improving the application's performance. Source-level profiling is an important mechanism for isolating problem areas in the application based on data collected during an actual run. For example, researchers working on GenIDLEST have used the profiling tool VProf to relate performance information to the source code.

The VProf visual profiler provides the important capability of relating events measured during profiling runs back to the original source code.

GenIDLEST and FPMPIview

The initial port of GenIDLEST to the Itanium cluster showed very poor performance relative to other platforms. Using the FPMPI profiling library and the graphical tool FPMPIview (a graphical tool previously developed as a prototype in early versions of PerfSuite software), problems were diagnosed by noting excessive synchronization time between nodes due to I/O issues. Appropriate code modifications to GenIDLEST significantly reduced the total runtime.

GenIDLEST before modifications. Over half the run-time was due to processors waiting for I/O to complete. MPI profiling identified this wait time as synchronization overhead.	GenIDLEST after modifications. Synchronization overhead was reduced to less than 20% of overall execution time, which in turn was reduced by more than 60%.

GenIDLEST Benchmarking and System Evaluation

The GenIDLEST team performs benchmarking of the application for scalability studies, not only of the code itself, but the underlying platform. These studies are useful in assessing how well the processor and communication facilities might be able to effectively support "Grid"-scale applications.

In the following case, the team noted substantial performance loss because of processor-to-memory bandwidth limits. When both processors on a node are employed, per-processor MFLOP rate is significantly reduced, due to high demands on the shared bus. The columns with a cross-hatched pattern show total MFLOP rate for a the two-processor per node configuration, while the columns with a solid pattern devote the node memory to a single CPU.

GenIDLEST performance analysis and benchmarking helps gain insight into strengths and limitations of the target processors and cluster environment.