In this study they examined the effects of cows milk on mouse testicles. They found that Cows milk lowered serum testosterone levels and reduced the weight of the seminal vessicles. In general, dairy is not a great idea for male fertility in my opinion. For female fertility there is some support for using dairy, however, there is a lot of contamination in dairy products so unless you are getting pristine dairy, its best to avoid.
 
Biomed Environ Sci. 2009 Apr;22(2):161-3.