What you describe is VERY normal.
All humans have a 'stronger' side to their hair than the other. Men will always have one temple that is receeded further than the other, and it is usually their right temple that is the worst side.
This is because hair does NOT grow from rear to front, it grows in a spiral pattern. And, follicles in male pattern baldness sufferers are programmed to miniaturize in a certain sequential pattern. So, for those whose hair grows in a clockwise spiral the right hand side thins the worst because the hairs there are affected for being temple hairs, and also hairs on the outermost ring of your spiral growth pattern. You know what, I can't describe it in words, give me a whiteboard and I could draw a little illustration that would peg it but believe me, it is normal for a man to have one temple thin faster than the other.