There's no point taking finasteride if you are also using dutasteride. Dutasteride does everything finasteride does and more, and taking finasteride alongside it will not result in any extra hair gains.
In your case, you probably just have extremely aggressive hair loss anyway. The dutasteride you are taking is sure to be slowing down your hair loss, but unfortunately it is not strong enough to put a complete halt to your own hair loss. It would seem to me to be highly improbable that finasteride and dutasteride can actually make a person's hair loss worse than it would have been if they had never treated their hair loss to begin with. That belief just doesn't add up, and it has not been shown to be true in any study.
The reason why your hair loss is continuing despite using dutasteride is likely due largely to the small amount of DHT leftover that the dutasteride is not inhibiting, and as you say, also due to the presence of the much weaker androgen testosterone, which dutasteride has the effect of raising slightly.
You could consider adding a low dose of oral minoxidil to your regimen, and also the topical antiandrogen CB-03-01, which helps by locally reducing the effect of both DHT and testosterone on your hair follicles. Use these in addition to dutasteride though, and not instead of dutasteride.