Cache effects for C = A'*B

Cache effects for C <- A'*B

The first thing I did was make three changes to the code:

Move the initialization of array C to the location of the initialization of the other two arrays. Not only did this throw off the megaflop count, but it was responsible for unnecessary page faults under varying conditions.
Add an iteration count to the matmul() loop to average out transients, compile two codes for iteration count of 100 and 1. The AIX kernel has a statistical kernel analysis tool that's always on the run queue, it's called kproc. In addition, if any of the switch software is installed, there are two daemons that have heartbeeps.
Make a version of the code that uses gettimeofday() instead of readrtc(). I did this because I thought there might be a race condition in the overflow detection of the clock. This turned out not to be the case.

The anomalies in the graph are due to a number of different factors.

The spikes at the matrix dimensions that fit in cache are not due to loading of a partial cache line but rather due to the unrolling performed by the -O3 option of the compiler. After looking at the assembler of the generated code, the compiler unrolled the loops in such a way as the inner loop strides over n where n is the number of doubles that can fit into a cache line. So what we're seeing is not partial loading, but rather the loop overhead. The inner loop code looks like this after O3 gets done with it.
```
incr = line_size / sizeof(double) /* Number of doubles in a cache line */
round = n - (n % incr)            /* Number of lines in dim */

/* Strip mine */
do 20 i = 1,round,incr
  do a cache lines worth of s = s + a(k,i)*b(k,j)
  ...
  ...
  ...
20 continue

/* One at a time to take care of leftover */
do 30 i = round+1, n, 1
  s = s + a(k,i)*b(k,j)
30 continue
```
When you use -O3 on IBM's compiler, it gives it freedom to do massive code motion. To disable major code rewriting, IBM provides the -qstrict option.
The repeatable troughs that were thought to be TLB misses seem to have been page faults / memory thrashing caused by the loop that initializes C in matmul(). It should be noted that I simulated this code using a cache/tlb simulator from the SimpleScalar toolkit. I found no abnormal jumps in the TLB miss count. This simulator is highly accurate save for OS intervention.

Here are the graphs justifying my conclusions:

Here is the tar file containing the source. Look here for the prebuilt executables.