Do you drink? How's your liver? I had problems but it was because my liver couldn't handle the medication. Basically, damage to the liver can cause a rise in estrogen which can impair sexual function somewhat. You could get your liver tested to see if your enzyme count is high, then go off the drug for a month and then see if the enzyme count goes down. It will be hard to judge because you didn't get your liver tested prior to the drug.
If the enzyme is high, maybe you have a pre-existing problem with your liver and taking finasteride is exacerbating it. Perhaps removing the cause of the problem will allow you to take the drug unhindered. OR finasteride is the reason for the high enzyme count, in which case you should quit the drug because it's damaging your liver over time.
I don't have any studies to back this, just my own experience. I don't see many who complain about sides getting bloodwork for their liver but it's something I would definitely encourage. I'm not sure lowering dht can directly cause such things, and wish more people would look into liver problems/other possible health issues before claiming finasteride for their issues. If it's not an option for whatever reason then you need to weigh up what means more to you. I would quit the drug though personally. I took the drug for about 4-5 months and I'm not sure exactly how many months it took for me to go back to normal after I quit but I would say three at most.
Good luck.