Counter-Strike Source, 2 pc's, and a router...need help

Ok, the problem isnt a huge one but it is one i would like solved. Here goes.... My brother and I each have our own steam account, we each have our own copy of counter-strike source (seperate cd keys). We can play both connect to steam fine, we can both play counter-strike at the same time fine, but what we CANNOT do is play counter-strike together in the same server at the same time. For some reason if we try to join a server together one of us will get in and the other will not (he will get blocked or something i guess) it will usually say something like could not connect after 4 tries.