A little help with overclocking memory

I am trying this for the first time so be gentlerolleyes

I have a DFI Lanparty ut x48 T2R mobo with ddr2 800 (pc6400) memory. When trying to increase the speed it gives me presets. Like this 266/800. Which runs fine but when I try to run 333/1066 it does some weird things. It doesnt crash but it wont boot to disc. I am trying to get it to run memtest but no matter what the boot priority is it always goes to the OS. As for the timings I have left them on auto. There are so many different settings on this mobo I am lost. And yes I did read the sticky on oc'ing. As for the rest of the system I have a kentsfield Q6600 C2Q with 4 gb g.skill ddr2 800 gtx 460 antec TP650 tuniq 120 extreme cooler and of course dvd drive and hd dvd rom drive. 2x360 seagate in raid0 and 1tb seagate hdd.

Any help would be appreciated.