Problem with latency spikes every 15 seconds latmon, dpclat. bad driver?

Hi all

I seem to be having a very frustrating problem with latency spikes, i am hoping someone can help me find and fix the issue

I am running a clean install of Win7 Ultimate, as i have just brought a new 2TB hard drive, i figured a nice clean audio workstation would do me good (i use my PC for music production)

The problem i am having is with latency spikes, i am running dpclat and Latmon to check my results.

dpclat has an average of 50us Latency, but every 15-30 seconds a spike will appear making the latency shoot up to anywhere between 90us and 1133us. I am pretty confident this is not normal.

Latmon (results attached) is doing the same thing, where most of the time i have a 50us latency, and every so often it will shoot up massively for a second, then go back to around 50-60us.

This is a problem i was having on an old machine that i spent months trying to fix, i think after trying everything i could i just decided to live with it. But being a new HD, and a clean install of Win7 64 and the problem coming back i am deciding to finally get this problem fixed.

Its worth mentioning that the first thing i installed after win7 was both the latency checkers, it was immediately apparent i was getting latency spikes BEFORE i even installed any graphics card drivers, windows updates, or any other programs for that matter!

i then proceeded to install all win updates, graphic drivers, etc in the hope that one of these might fix the issue... no luck there

Things i have already tried to solve the issue:

disabling USB and PCI devises one by one as DPC latency Checker suggested to try and isolate a resource conflict.

Updating Drivers using Device Manager

disabled speedstep eist in the BIOS.

putting the power plan on high.

I have no Wi-Fi device installed. I am connected via Ethernet cable

disabled the network adapter

disabling video card driver

disabling onboard audio driver

My Computer Specs:

Mother Board: P7H55-M PRO (LGA1156) 44 °C
Processor: Intel Core i3 530 @ 2.93GHz 45 °C
GPU: 512MB ATI Radeon HD 5450 (ATI AIB)
Memory: 6.00GB Dual-Channel DDR3 @ 668MHz (9-9-9-24)PC3-10666C9 1333MHz Triple Channel
HDD: 1863GB Seagate ST2000DM001-1CH164 ATA Device (SATA) 38 °C
932GB Seagate ST31000528AS ATA Device (SATA)
Sound: Firewire focusrite Pro14 audio interface
Power supply: OCZ StealthXStream 400W Silent
Operating system: Microsoft Windows 7 Ultimate 64-bit

This problem is really driving me mad, It would be worth me mentioning that this computer is a self-build with a clean install of Window 7 64 home edition so this may be of some importance, not sure :confused

anyway, i have attached my Latmon results, these results are from leaving latmon running for approx 5 hours with nothing else running apart from nightly (firefox 64 bit browser), and a background switcher that changes my wallpaper every 5 mins

from what i can tell the main culprits are the following:

most hard page faults caused by:

background switcher 19491
avg resident shield 16715
avg scanning core module 6151

drivers causing the highest execution (ms)

ohci1394.sys
rspLLL64.sys
tcpip.sys
ataport.sys
dxgkrnl.sys
hal.dll

Please help, if there is any more info i can supply then please ask. I really want to fix this issue so i can get back to actually making some music!

 

Hi Kilo.

Your Firewire card must be high on the list of suspects.

 

Thanks for the quick reply

it was but after disabling the IEEE and running the test again it made no difference. spikes still appear.

BTW the IEEE firewire connects to my soundcard

I have just now discovered that disabling my saffire (soundcard) driver (under sound,video and games controllers in device manager) that it definitely makes a big improvement... this makes sense because the last time i had CPU spikes on my comp it turned out i had to uninstall and use a earlier version of mixcontroll (the software that came with my soundcard). this fixed the issue last time.

currently i have saffire disabled (i will fix this later, once i find the other problems causing spikes), yet im still experiencing some latency spikes

I think i have whittled the problem down to ataport.sys, as this driver is the highest execution (ms) i am experiencing... its execution is well above any others, i just need to find out how to solve this

the other problem is with hard pagefaults. svchost.exe (which i remember is a bitch to find out whats actually responsible). svchost.exe has the highest number of hard pagefaults (1364 to be exact), the process causing the second highest amount of pagefaults only has 6!! :banghead

 

You have to get an understanding of what goes on under the hood normally on a Windows machine.

Ataport deals with data flow to/from the hard drives etc., there is a large amount of logging by NTFS etc. as well as by any User software - antivirus is usually one of the worst of those. Using tactics designed to minimise writes to SSD boot drives will help, as will taking more control over which files, how frequently and whether you need checks on reads/writes/creates in the a/v.

Svchost covers all the Windows Services, try reducing the traffic by minimising the services to those actually needed.

Changing the slot used by the Firewire card may reduce traffic, as might disabling anything not needed in the BIOS (Serial/Parallel ports etc.). Reducing any 'utility' software (thinking specifically of ASUS stuff here but things like ASC too) might help.

USBport will have a lot of traffic just from a mouse (mouse drivers can also have a big effect on DirectX), WMP media sharing will create network and drive traffic while increasing a/v checks.

There's a lot going on, it's difficult to turn an all-round PC into an audio or video workstation unless it has business class or workstation components with solid, low latency drivers. Keeping drivers as 'clean' as possible should help, the fewer 3rd party drivers involved the better.

There's a useful piece on DPC/latency issues here, it also looks at using Microsoft Windows Performance Analyzer to help with diagnosis.

(I just checked DPCLat here, it's been running since ~15 minutes before I made the earlier reply: 772us max. I normally only check over a 10-30 minute period)

 

If you are getting latency across the Internet to another site, I would not worry about it. If it is inside your LAN, the first places to look at, is the network equipment (Router, Switches, cables, Network adapters).

 

Thanks for the replys, if im honest i dont really have much knowledge on a lot you just said, i guess im going to be busy for a few months trying to get the knowledge i need to make my build a decent music production workhorse.

yeah i spent the last 8 hours disabling and turning off things one by one, then checking latmon for changes. AVG definatly was adding some latency issues.

Unfortunatly my mobo has only one slot for firewire

yeah i figured that out too, i changed it from automatic startup to disabled, not that it made things better.

your not wrong, seems i found this out the hard way. spent a lot of time the past few days trying to make things work. seems i made things worse so ive just done a system restore to pretty much back where i started, im now back to where i was just after installing windows, still with the problems mind! :cry

good stuff, thanks. looks like this might just help me out, i will be spending tomorrow digging into this..

Seems like a lot of other people are having similar problems, with not many people solving the issues.. hard to belive this is so difficult to fix

 

the hard pagefaults go up a lot while using the internet, but even with network adapters disabled im getting spikes. so i dunno... just getting annoyed now, feeling like ive tried everything.

 

Internet latency isn't really an issue, an audio workstation really doesn't need any internet connected while working on a project. (If this was XP, I'd suggest creating a new Hardware Profile and tweaking and using that purely for audio projects; there's a method to emulate that with W7 but it reads like you need to jump through several hoops of real fire at each stage over a period of several days to do that ).

Check that the JMicron IDE controller is disabled.

Enable the onboard graphics and remove the HD5450, replace it with the Firewire card (unless it's PCI, in which case, consider buying/borrowing a PCI-e card) and test again.

You might find the 'normal' Intel chipset/SATA drivers (if you can track them down) are better than the IaStor versions (designed for SSD caching), if that's your current driver. You could also try setting them via Device Manager to use the standard Microsoft drivers (they'll still use some of the Intel drivers but should be 'cleaner' and more stable).

You could try switching A/Vs - to Avast!, perhaps, it's reasonably configurable if you dig deep enough. With any security software removal or change of software, fully remove the old stuff using the specific removal tool from the makers, any remnants often cause issues later on.

Probably best to work from hardware up, disable anything that's not vital from the BIOS first then tweak Windows Services and settings, 'mess' with the 3rd party drivers and versions, uninstall 'utilities' that autoload any drivers/filters at boot, tweak any other 3rd party software that you must have running during a project, work out a routine whereby you reboot then shutdown anything not needed before firing up your project, etc.

Between each stage, reboot and then run the checks for 20-30 minutes and create logs to keep track of what works and what doesn't, once you've made checks and it's improved, make the next change before rebooting to check again. If the change makes it worse, reset it and reboot/check before making the next change.

If you could do this offline and without any A/V installed or Scheduled Tasks running, it should make differences easier to 'see'. Oh, and wait 10 or so minutes post-boot before testing, that way, any background Windows tests/checks should be finished.

 

Good morning!

I could not find this setting in the BIOS, but there was something similar. it was JMicron PATA. so i disabled that. still no diffrence.

in my last post i said i only had one port for firewire, i was wrong. i have 1 PCI express slot and 2 PCI slots, so swapping them round is impossible since my graphics card is on the PCIe. However, I enabled onboard graphics as you asked, and took out both the IEEE firewire card AND my graphics card.. still no change. i am getting big latency spikes every 15-30 seconds.

This is what im doing now.

BTW. I have Win 7 installed on BOTH my hard drives, and im experiencing the same latency spikes on both OS when testing, which makes me think the problem is either hardware related, or BIOS related. Which leads me on to my question...

while researching things that i can disable in the bios i found out that i am using IDE (which is why i assume i cant find RAID anywhere in BIOS). I think perhaps if i switch to AHCI this might solve my latency problem, as i read somewhere online, quote: "AHCI is generally a better option than IDE mode for several reasons such as NCQ and the ability to hot swap. Without getting in to too much detail, NCQ can improve disk performance depending on your setup"

edit: since one of the main problems is with ataport.sys, and ataport.sys is HD related, and AHCI improves disk performance, this is why im thinking it my stop the latency issues if i switch to AHCI

so perhaps if i re-format my new 2TB HD, enable AHCI, and re-install win7 this may solve my latency problem. But i think doing this may interfere with my old HD, and reformating that HD and installing win to that is not an option, as i have so much stuff on my old HD that it would be inconvient to say the least... Thoughts?

 

What are you using to benchmark all of this, and what are you comparing the results to, that you are getting these numbers, besides LatencyMon. All systems are going to have some form of latency when they are running.

Change out the OS, that Latency will also change. Windows has a lot of overhead, and I would not be sure, that you are just fretting about nothing. I really would not put much more into it, or you will end up being one of those that is writing their own OS.

I just went to http://www.resplendence.com/latencymon and looking over their info on the software, I would forget what it is reporting. The biggest part that set off a big red flag, was this statement "Windows is not a real-time operating system. All requests to the operating system are delivered on a best effort basis. There are no guarantees whatsoever that requests are delivered within a certain time frame, which are the characteristics of a real-time operating system." Then there is this one "Windows uses a concept of virtual memory which relies on the page translation system provided by the CPU. Whenever a memory address is requested which is not available in physical memory (not resident), an INT 14 will occur. The OS provided INT 14 handler will decide how to proceed next. If the page in which the address resides is known to Windows but not resident, Windows will read in the required page from the page file. That is known as a hard pagefault and can take a lot of time to complete. If the page can be read in from the hard disk cache, the price will be limited. However if it needs to physically read in the data from disk sectors this takes a lot of time. If an audio program hits a hard pagefault while it is playing it will almost certainly have audible consequences recognized as dropouts, clicks or pops."

Every OS is not a Real-Time OS, all of them use some sort of file to store items that are running as background or minimized processes. Personally stop worrying about some third party software, written to discredit Windows, by using false accusations in trying to "sell" their software. Just move on and do what you are doing, and please stop using Latencymon.

 

Pretty much every consumer OS is not a Real-Time OS, all of them use some sort of file to store items that are running as background or minimized processes. Until you get into those that are used for "Big Iron", then you will see a Real-Time OS.

http://en.wikipedia.org/wiki/Real-time_operating_system
http://en.wikipedia.org/wiki/List_of_real-time_operating_systems

 

Thanks for the input. But I am sure it is a problem as both DPClat and LatMon tell me this. I have an average latency of 40-60us but every 30 or so seconds there will be a massive spike anywhere between 200-1000us.

On Latmon the message reads "your system appears to be suitable for handling real-time audio and other tasks without dropouts" and will stay this way even after passing several of the "spikes". However after 5 minutes or so one of these "spikes" will shoot so high the message will change to "your system seems to have difficulty handling real-time audio and other tasks. you may experience, clicks or pops due to buffer under runs, one problem may be related to power management"

From what i understand a stable PC should be able to run for any period of time without this latter error message. unfortunately i have no way of comparing my results with other users so i am assuming there is a problem

Sure the spikes are infrequent and dont last long, but if i have the ability to get rid of them altogether i would like to

Please see the DPClat pic of my results (run for approx 2 mins) attached to this post for a visual reference. I am not sure this is normal behavior :confused

 

I had not seen this before and it does put a new perspective on things. perhaps im being a bit paranoid and my results are "normal". but as a full time producer I am kind of obsessed about getting my system to be fully optimized to maximize its efficiency. I will go over to gearslutz forum and see if anyone else there experiences similar issues. Thanks for the input

 

kilo85...

I don't know if this will help, but Process Lasso:

http://www.majorgeeks.com/files/details/process_lasso.html

has some restraining qualities that streamline the optimization process in Windows. There are a couple of tricks I have picked up about PL from the developer that make it indispensable for me, and I suspect it might help with your spikes if set up correctly.

If you decide to try it, try this:

Configuration and Log->Manually edit the configuration file

When it opens, search the text for the word "speed". Click twice and you should see "UpdateSpeedCore" highlighted. Change the value to 5 and save the file and see if that helps.

I use PL with the "UpdateSpeedCore" set at 8 for a core 2 duo PC. I think your i3 could handle 5, though. If not, set it to 8 or just try 8 first. It works like a charm for me. As brownizs mentioned, Windows is not a real-time OS, but PL improves dramatically the response times of a PC when it is set correctly. For me, information is being moved to RAM (prefetched) in a very much more stable and intuitive way, because PL's checks have served to push Windows into using its native run-time at a much faster rate. This comes at almost no cost in terms of processor usage, which is the best part. Boils down to much smoother transitions from task to task.

Just a word about this refresh setting. This is dramatically (to say the least) lower than the default of the program (500 ms). Actually, 5 ms cannot be achieved using the GUI, the fastest possible being 100 ms. I discussed this with the developer and after I mentioned to him that I would like to test PL at a faster governor refresh rate, he told me about the manual edit and that he had intentionally made it possible for testing purposes. As for the defaults, I can only speculate, but I think the developer is being purposely cautious with the program. I don't think PL can break a PC, but it appears to me that he is trying to avoid scenarios where users might say so by setting the default refresh at such a high number.

Sorry about the length of this, but I think it's necessary. This setting in PL really works. I've had it running this way for 4 months, and the responsiveness of the PC is just getting better and better with time...

One other setting to look into is:

Options->ProBalance settings...->Configure excluded processes...

You can add processes to exclude from the restraining process of PL. You could include your music programs and browser. I recommend including explorer.exe too...

I checked on DPC Latency checker, and the latency readings on this PC are between 20 and 80 ms at idle and are very stable...

PL (along with some other programs) has changed my outlook on computers. I recommend giving it a try...

 

giving it a go now! will post back with my findings

worth a shot... thanks!

 

Again, I would forget what LatencyMon and the other stuff is showing. Going back to the beginning, is all of this causing problems when you are editing videos, or music, or playing games?

Really you are looking much more deeper into an issue that I personally would not worry about.

As for the info I posted bolded and highlighted, again, it is the company that sells Latency mon slamming the OS, which also Mac OS, Linux are also under the category as not "Real-Time" os's, which as I also stated, it is when you get into "Big Iron", that you see Real-time OS's.

Again, if it is really causing issues, you are looking at the problem in the wrong direction, not the fact that it is the device itself, you and latency mon are blaming the OS.

 

A Valid point. Just seems a few other people have been having this "problem", and i assumed it was telling the truth. if it really is mis-information they are putting out there then they wasted a lot of my time in trying to fix a problem that dosent exist... well ill take your word for it. Not happy with the developer though

 

There's no misinformation; for anyone recording audio, dropouts are a major problem - a show stopper, for users, audio dropouts are a nuisance; music is completely spoiled by microseconds of silence.

These tools and methods are out there to enable those of us who have this issue to be able to get a handle on what's causing it.

Here's a useful Thread that's still updated and the author, Andre Ziegler, is still actively helping in it.

 

But in turn, most are using Mac OS, a variant of Linux, or a server built specifically on the high end, with also professional equipment installed in or on it, that will not cause these problems.

Majority of latency issues are the hard drive, next is the low end sound cards with poor quality DSP chips.

If you go cheap, you get cheap in return for what you spent. Another item, Normal Windows desktop OS's have always had latency on it, regardless which version. That is why most that do this kind of stuff, again either opt for Mac OS, a variant of Linux, or a Windows Server version and run on i7 or multi-core Xenon CPU systems.

Some more info about tuning for latency http://shabiryusuf.wordpress.com/20...f-latency-sensitive-workloads-in-vsphere-vms/ Some info from NAB2013 http://www.wmdt.com/story/21852000/...o-glass-latency-of-less-than-70ms-at-nab-2013

There is plenty of info out there on fine tuning to try to get rid of as much latency as you can, but again, until you move into higher end equipment and away for desktop Operating systems, low end equipment, there is nothing you can do about it.

 

A lot of good info here http://www.gearslutz.com/board/musi...erface-low-latency-performance-data-base.html Some more http://tweakheadz.com/latency/ Another DPC Latency program http://www.thesycon.de/deu/latency_check.shtml

 

We're not dealing with a server OS, Mac or any form of Linux here and Kilo has been using DPCLat since before he started the topic (look at the OP). I brought up the topic of quality hardware back in #4.

Let's try to work with what we have, shall we?

 

The problem is that they are using what is considered low end economy grade hardware for the motherboard & CPU, by today's standards. Unless they move up to multi-core i7 or multi-core Xenon, or even into a Mac Server, they are not going to get around the problem they are having with that budget grade equipment.

 

Spent today going through this, getting a little closer each day i think. since i hardly ever use cmd function i think im learning quick enough.

Having trouble finding the "DPC CPU Usage" graph in windows performance analyzer though. i followed all the previous advice to the letter but i just cant find that option. So im going off the DPC/ISR graph instead.

one thing i did notice from reading the thread is i might have a possible IRQ conflict, as that seems to be a possible cause (might be wrong)

If you know much about IRQ's could you take a look at the attached print screen and see if anything strikes you as problematic?

Thanks

 

With the newer operating systems, it will turn on and off devices for those IRQ's, if they share the same one. If there is one not being used, and that device has the same as one that is running, the OS will look for a different IRQ.

Bios's and OS's have gotten smarter in how they utilize resources, but still even Windows has a long ways to go before it gets better in how some people place large demands on it, which it was not intended for, so you get problems like this that is being discussed.

 

so if it was an IRQ conflict it would have been detected. i thought perhaps because my high definition audio controller was sharing the same IRQ as my firewire it might of been problematic.

never mind, on the the next possible cause

 

OK, so after 4 days messing about I have managed to solve the problems with latency spikes on DPClat and Latency mon

I will go over the steps I took in the hope they may help someone else

WARNING: These steps include turning off startup services, doing this can cause problems if you dont know what you are doing. I recommend backing everything up and making notes of any changes you make. I will not be held responsible if you muck things up!

I was lucky, as because I was having these problems on a new hard drive with a new copy of win7 installed if I messed things up too bad i could re-format the HD and re-install Win7 (which I ended up doing 3-4 times lol)

To fix the spikes I was getting in DPClat was quite easy. I had to reset my BIOS settings, for some reason even though I had disabled everything that needed disabling (e.g throttling progs like speedstep) the BIOS settings were still causing me problems. All I did was factory reset the BIOS settings and go back and turn off anything that might be throttling my system, for my P7H55-M Pro MB these were Speedstep, Intel(R) C-STATE, and C1E Support.

Once I did this I had no more spikes in DPC lat, and an average latency of 50us.

Even after doing this I was still experiencing some problems with LatMon, basically after 2-5 mins of running latmon I was getting the following error:

Your system seems to have difficulty handling real-time audio and other tasks. You may experience drop outs, clicks or pops due to buffer underruns. One problem may be related to power management, disable CPU throttling settings in Control Panel and BIOS setup. Check for BIOS updates.

This was a tricky one to solve. But through the process of elimination I found I was having problems with one of the startup programs

I realized this when I went into msconfig and unchecked the box that said "load system services", once all system services were disabled I restarted my PC and ran Latmon again, and thankfully i was no longer seeing the error mentioned above.

So from here on out it was a case of isolating the problematic service/s and disabling it.

I re-checked the Load system services box and sorting the services alphabetically i would start at the top of the list turning on the services in batches of 5-10, restarting my PC and seeing if latmon error message came back.

For me there was only one problematic service, and that was Power, once this service was turned on latmon lit up like a Christmas tree and the error came back. I still carried on through the list turning on services in batches of 5 (with the "power" service still disabled), restarting my PC, and running latmon.

so for me the "Power" service was the only service causing problems. From what I understand this service is responsible for when the computer sleeps or hibernates, so its no biggie to turn it off.

Again, if you are experiencing problems and want to try this method, I recommend only doing it with everything backed up and recent restore points set. These fixes may sound simple, but they took me days to find out. I know there are many people having problems with similar issues, and very little in the way of fixes offered. If I can save anyone the headaches I suffered then im happy!

Chronos

oh, and a big thanks to all who tried helping me on this issue

 

Hmm, 'Power', could be Windows scanning for inactive USB or networking devices; did you turn off any power saving on those earlier in your troubleshooting process?

Anyway, I'm really pleased you got it sorted out, saved you from having to spend $$$$s on new hardware, eh?

 

yeah i checked everything... twice!!

and yeah, i dont think ill fork out, unless i absolutely have to!

well my joy was short lived... turns out disabling the power service also disabled the speakers for some stupid reason! hardly acceptable since i needed my speakers to work! and turning the power service back on ment i had sound, but massive issues within latencymon again!! arghhh!

so i spent the day (once again) trying to get to the bottom of it... i tried everything from changing power options to disabling "core parking".. these last few days have been a nightmare!

but... i have now found what i think is a stable cure.. i have the power service enabled, no pops and clicks, and ran latmon for the first time for 5 mins without a hitch!

the fixes can be found here for anyone who might be interested...

http://presonus.zendesk.com/entries/203607-optimizing-windows-vista-7-power-settings-for-studio-one

http://duc.avid.com/archive/index.php/t-292466.html

thanks once again for the help, hopefully i wont be back with more problems!

 

I hope that's the end of that, now you can begin work!

Read and bookmarked the links, thanks :major