Do what you want. Even men with low levels of testosterone go bald, so decreasing your dietary problems isn't going to do much in the long run.
High testosterone levels, low SHBG, or high insulin levels may increase free testosterone, which will increase DHT. However, lowering testosterone, raising SHBG, and lowing insulin levels through diet will do nothing.
You just can't lower testosterone enough to reduce the concentration of DHT to what is needed to make a difference. The body will compensate and you will still have testosterone values that are within normal limits just like balding and non-balding men. Raising the level of SHBG will do the same thing. You'll bind more testosterone, but you'll also produce more testosterone to compensate.
This idea will not prevent baldness. It may slow it, but it will not reverse it and it will not prevent it from happening. Hormone levels in men are just too high for diet to change them. Only drugs that target 5AR, the androgen receptor, or the downstream mediators that are increased by androgen binding will prevent hair loss.
Everyone needs to think this through. It's good to have a healthy diet, but in this case it wont' matter.