Unless you are extremely good looking, or distinct enough e.g high-cheekbones, stand out eyes, then I'd agree it really is.
Look at women. Not many suit short hair, and the ones that do usually have very attractive natural features.
A friend of mine has a youthful face even though he's 29 and...