You should not take advises from random strangers over the internet, maybe take a sperm analysis test or maybe see a f*****g doctor? It's well know that it decreases sperm count and that it can even make one infertile with time, but they say it's a small chance or that maybe you have to be already predisposed to get it and many other theories.
Nobody knows exactly and you don't know either what the future holds, you either trust the science behind it, you either trust your doctors or your instincts and gut or whatever you do, it's your decision only, nobody can assure your of nothing because as i said, nobody knows...
My personal opinion is that you should do what i said above, a sperm analysis or at least see a doctor who actually knows sh*t about this things, not just a random derm that looks at your head with a magnifying glass and after prescribes you hormonal pills like candy without even taking a f*****g blood test...