Trust me when I say this, and I'm pretty sure I speak for most members on this forum: stress, poor nutrition, wearing a hat all the time, telogen effluvium and so forth is EXTREMELY RARE and I have yet not seen any case here on hairlosstalk that is not caused by normal male pattern baldness.
Male pattern baldness almost always starts with a receding hairline, such as yours (telogen effluvium does not, so drop those thoughts). Mine started out in the same way. And yes, it is highly likely that you've inherited the baldness gene from your mothers side: lots of people do. And yes, it sucks, but it's not the end of the world!