i've been on finasteride in the past and currently use dutasteride EOD and have (knock on wood!) never had this problem. yes, there is a potential risk but plenty use these drugs without such sides.
that being said, it could have something to do with the fact preventing conversion of T to DHT can increase estrogens which possibly lowers T....but i really don't know for sure
*I am not a doctor, don't take my word for anything