Network lags when one person downloads

Alright, I've been working with computers for quite a long time, but I can't figure out why this problem keep plaguing me.

I have a "RNX-EasyN4 WLAN 802.11n/b/g Router" with anywhere from 1 to 5 (rarely) people using it simultaneously. The majority of the time it is limited to 3 people with wireless connections.

I have enabled QoS to allow 2Mbit download rate for each IP in the range x.x.x.100 to x.x.x.107 and 1 Mbit download rate for x.x.x.109 and on. (My IP is x.x.x.108 and enjoys full access )

Anyhow we have a 25Mbit connection, which should be more than enough to sustain 3*2Mbit users, myself (when I'm not downloading!), and 1-2 1Mbit connections, all at once. I've also taken into account our upload rates which are minimal.

The problem is that whenever anyone is downloading while we are playing games online (netflix, FTP downloads, streaming youtube, even music) the entire network will lag. I don't understand it. I watch the network usage monitor and the numbers are all reasonable. I've tested this problem with netflix: I can even reduce the 'netflixing' computer to 128Kbits (16KB/s) and the network will STILL lag. Most of the time the lag stutters with periods of smooth gameplay then large lag spikes - other times the lag won't stop until the download stops.

Someone please help me out here.

 

Also, a quick unrelated question:

Can USB 2.0 speeds affect USB Wireless Adapter speeds/would the USB 2.0 be able to handle full wireless n capability?

My guess is no. Which leads to another networking problem. When I plug my USB Wireless Adapter into a USB 3.0 slot it will stop working after a while, requiring a reboot. Anyone know why?