IBM Power5 Processor

The Power5 processor has the same instruction pipeline as the Power4, but adds simultaneous multithreading (SMT) and has a different memory hierarchy. (There are also improvements to power management, RAS, etc.) This page highlights the Power5's memory hierarchy.

Memory hierarchy in general

• locality of reference - Hatfield and Gerald, 1971 IBM Systems Journal, memory access trace diagrams (25 pp. pdf)
  • see, e.g., Figure 10, p. 189
  • a horizontal reference pattern represents temporal locality
  • a diagonal reference pattern represents spatial locality (e.g., array traversal)
• typical hierarchy (e.g., Power2 single chip)
• examples of Alpha latencies (ldasim)
• memory hierarchy tutorial (G. Prabhu, Iowa State)
• simple set-associative cache example (see also other cache notes)
• matrix multiply performance depends on cache access patterns
• 16x16 matrix multiply access patterns

Cache comparisons (Power5 vs. other processors)

• cache organizations (comparison with Opteron, etc.)
• cache organizations (comparison with Itanium 2)

Details on Power5 memory hierarchy levels

	cycles (2004)	lmbench (2005)	sizes	line size	associativity	write policy	sharing	transfer rate
registers	1		32+32 (120+120)				per thread (per core)
L1 Icache	1		64 KB	128 bytes	2-way		per core
L1 Dcache	2	2	32 KB	128 bytes	4-way	write-through	per core	2 words/cycle
L2 cache	13	32	1.875 MB	128 bytes	10-way	write back	shared	4 words/cycle
L3 cache	87	92	36 MB	256 bytes	12-way	write back	shared	< 1 word/cycle
memory	220	403	4 GB per node

All caches on the Power5 use LRU replacement.

set/line/associativity/slice organization

  (note 1: the directory bits needed for valid/tag/writeback are not shown below)
  (note 2: some books use "bank" to mean what the IBM authors have called a slice;
     slices are used to avoid L2 access conflicts between the two CPUs)


L2     one of three slices (640 KB per slice)
         __________________________________     N-way set associativity
        /                                  \      => N banks operating
         bank 0     bank 1          bank 9           in parallel
        +-------+  +-------+       +-------+
     0  | line  |  |       |  ...  |       |    512 lines/bank
     1  |       |  |       |       |       |     * 128 bytes/line
   ...                                           * 10 banks/slice
   511  |       |  |       |       |       |     * 3 slices
        +-------+  +-------+       +-------+    = 1920 KB = 1.875 MB
        |<-128->|


L3                one of three slices (12 MB per slice)
         _______________________________________________________
        /                                                       \
             bank 0            bank 1                bank 11
        +--------------+  +--------------+       +--------------+
     0  |  sb0 / sb1   |  |              |  ...  |              |
     1  |              |  |              |       |              |  4096   l/b
     2  |              |  |              |       |              |   * 256 B/l
   ...                                                              * 12  b/s
  4094  |              |  |              |       |              |   * 3   s
  4095  |              |  |              |       |              |  = 36 MB
        +--------------+  +--------------+       +--------------+
        |<----256----->|

Power5 performance considerations

• dual core (chip multiprocessing - CMP)
• two threads per core (simultaneous multithreading - SMT)

• instruction level parallelism within each core
  • out-of-order execution within instruction window
  • up to 8 instructions issued and 5 instructions completed per cycle
  • over 200 instructions "in-flight"

• register renaming
  • the 32 GPR architected registers per thread are dynamically renamed to 120 physical registers
  • the 32 FPR architected registers per thread are dynamically renamed to 120 physical registers
  • other registers (condition fields, FP status, etc.) are also renamed
  • example generic diagram of hardware loop unrolling enabled by register renaming (not Power5)
  • the two threads per core share the pool of physical registers

• two floating-point units per core
  • fused multiply-add to support inner product -- sum = sum + x*y
  • fma - 6 cycle latency (pipelined - can start new one each cycle, 6 cycle forwarding to dependent instruction)
  • fdiv - 32 cycle latency dp (not pipelined)
  • fsqrt - 34 cycle latency dp (not pipelined)
  • avoid divides and square roots when possible
  • a loop needs 12 independent fma instructions to keep FPU at peak (but architected register use is limited to 32)

• two load/store units per core
  • L1 Dcache support two 8-byte reads per cycle or one 8-byte read and one 8-byte store per cycle
  • in a loop the number of load/stores should not exceed the number of fma instructions

• Up to 8 outstanding cache line misses
  • hit under miss
  • miss under miss
  • reload rate is 32 bytes per cycle

• Up to 8 data prefetch streams
  • two lines per stream
  • forward and backward strides

• Sectors in L3
  • each 256 B line consists of two 128 B subblocks that share the same tag
  • each subblock can be present or not (two valid bits per line, one per subblock)

• TLB/ERAT misses
  • 1024 TLB entries, 4 KB or 16 MB pages, 4-way set associative (also one 64-entry SLB per thread)
  • 128 D-ERAT entries, 4 KB pages, fully associative
  • TLB and SLB used on ERAT misses
  • 25 to 100s of cycles per TLB miss
  • Power3 TLB miss examples
  • avoid large strides, scatter/gather

Other general (non-Power5-specific) memory hierarchy issues

• Multi-level memory hierarchy profiling
  • see figure 4 (on SGI/MIPS system, Zagha, et al.)
  • notice TLB miss component for z sweep of NxNxN grid as N gets larger
  • notice cache thrashing for z sweep of NxNxN grid with N a power of 2

• SMT/CMP effects
  • OpenMP with SMT and CMP effects slides (Maury et al., 76 pp. pdf)
  • L2 miss rate can increase with second thread in SMT (slide 23)
  • TLB miss rate can increase with second thread in SMT (slide 32)

• False sharing
  • problem to avoid in shared memory programs
  • separate local read/write data items are allocated in the same cache line
  • write-invalidate mulitprocessor cache coherency bounces line from cache to cache
  • example (from S. Browne) -- Assume m=4 and the following code is run on four CPUs. The time for a parallel run will be longer than when just one CPU is used.

      integer m, n, i, j 
      real a(m,n), s(m) 
      
      c$doacross local(i,j), shared(s,a) 
      
      do i = 1,m 
        s(i) = 0.0 
        do j = 1, n 
          s(i) = s(i) + a(i,j) 
        enddo 
      enddo 
          

  • Intel page on avoiding false sharing
  • HP page on avoiding false sharing

Power5 links

• IBM JRD special issue on Power 5
• Elnozahy Power5 overview slides, 2004 (24 pp. pdf)
• Grassl Power5 tutorial slides, 2005 (76 pp. pdf)
• LLNL SP systems comparisons

Power4 optimization (Power5 has same instruction pipeline as Power4)

• Optimizing for Performance on IBM Power4 Systems, 2003 (158 pp. pdf)
• IBM Power4 Redbook
  • Chapter 1. Processor evolution
  • Chapter 2. The POWER4 system
  • Chapter 3. POWER4 system performance and tuning
  • Chapter 4. Optimizing with the compilers
  • Chapter 5. General tuning guidelines
  • Chapter 6. Performance libraries
  • Chapter 7. Parallel programming techniques and performance
  • Chapter 8. Application performance and throughput
• University of Texas guide
  • General Optimization Advice
  • Minor Source Code Modification
  • Use Best Compiler Optimization Options
  • Tune for the System Architecture
  • Use High Performance Libraries and Software
  • References
• Supercomputing in Plain English (uses Power4 as an example)

last updated: March 2006
Mark Smotherman, Dept. of Computer Science, Clemson University