If you only wash your hair every 4-5 days, losing 40-50 hairs is not a lot. Losing hair in the shower is normal.
Some people just lose more hair than others. It's not important how many hairs you lose, it is important how many hairs actually grow back.
Just check if you are thinning or balding...