CPU is very spikey and erratic, severe latency. Server 2003

Something is making my server go crazy. First it was just the network had some bad latency, then I noticed it was on the server, not the network.
I thought it to be HDD related but I am not sure.
It seems to spike every few seconds for less than a second.
I shutdown Hmail and Apache, neither were the issue.

A link to 2 images, one of the CPU graph and one of the Process list sorted by CPU time. Also in there are a couple text files with Ping from anotehr machine and a netstat -a -b .
http://www.rscivil.com/dell/
Reply from 10.0.0.5: bytes=32 time<1ms TTL=128
Reply from 10.0.0.5: bytes=32 time=2898ms TTL=128
Reply from 10.0.0.5: bytes=32 time<1ms TTL=128
Reply from 10.0.0.5: bytes=32 time<1ms TTL=128
Reply from 10.0.0.5: bytes=32 time=2026ms TTL=128
Reply from 10.0.0.5: bytes=32 time<1ms TTL=128
Reply from 10.0.0.5: bytes=32 time<1ms TTL=128
Reply from 10.0.0.5: bytes=32 time=2022ms TTL=128
Reply from 10.0.0.5: bytes=32 time=2ms TTL=128
Reply from 10.0.0.5: bytes=32 time=1ms TTL=128
========================
Win Server 2003 SP2
Dell PowerEdge sc1425
Xeon 3.4GHZ (Dual Core)
2GB Ram
RAID 1 (mirror )with 2x 1TB Hard Drives

Used mainly as a file server and Active directory and DNS server.
Also runs Apache (very light load) and HMailServer for email.

It rarely has load issues.
The RAID controller shows the raid (and each drive) as Optimal.
===================

Any Suggestions where to go from here?
http://www.rscivil.com/dell/CPU.png
http://www.rscivil.com/dell/TOP.png

 

New info. it is being caused by Interrupts. Now what can I do about it?
new CPU?
http://www.rscivil.com/dell/process_explorer.PNG

 

Welcome to Majorgeeks

Well, I know next to nothing about web/mail servers. I do see a big problem with the CPU spikes from the Windows kernel, the red? spikes. When that happens, each running user process is effectively run at a much lower priority - if there's enough CPU headroom left to run them at all. That, I think, is where you're seeing the bad latency problems.

Process Explorer would be my first choice to check out what's happening there.

EDIT: LOL, I see you've already discovered Process Explorer

 

Now you'll need to run Latency Monitor, this should enable you to see which driver or process is causing the problem.

Maybe the built-in Performance monitoring tools will give you more info, from what I recall of it on XP, it's not exactly straightforward to setup and use though. Otherwise, I think it's a case of disabling drivers until you find the one responsible, good old trial and error.

With a little luck, it'll be something 'simple' like a corrupt or fragmented pagefile. Only careful monitoring will tell.

 

LatencyMon requires server 07 or vista+
I am running 2003. I tried to find an older version with no luck so far.
Any other suggestions?

 

Grr, no, no older version

The only other specific tool I know of is DPC Latency Checker but that's not good enough for this task. Perhaps Process Monitor will help? There are a lot of options with this, I've barely scratched the surface of what it can do.

Or it's time to use the built-in tools?

 

Well, I am trying to find the Interrupts with Process monitor.
No luck so far.

 

You can safely ignore all User processes, right-click and 'Exclude'. Most drivers will be loaded from System32 = Path column. Tools > Process activity summary might be a useful graph, too.

 

I can look at System, which is where the "Interrupts" lives, but I am not finding much useful there.

 

Ok, go back to Process Explorer and open Device Manager, disable non-essential devices whilst watching the Interrupts in ProcExp, if the driver has no effect on the latency, re-enable it and move on to the next one.

USB devices, network cards, display drivers, sound (or sound effects) drivers would be the most likely culprits I think.

Once you've found the offender, look for an updated driver for it.

You could always check through the System logs for clues, too.

 

Thanks for all your help Satrow.
I have 4 external drive bays hooked up to a pci-x raid controller.
One of these bays took a crap.
Turned it off and voila.

 

I'm pleased you found the culprit Easy replacement, not too costly to replace?

 

It will be no problem. I have extra drive bays, unless it is the disk. But I have extra disks too =)