Cloth wearing seems so natural that everyone is self-deemed knowledgeable and has some expert opinions about it. However to clearly explain the physics involved, and hence to make predictions for clothing design or selection, turns out as demonstrated below to be quite challenging even for experts. Cloth is a multiphased, porous and anisotropic material system and usually in multilayers. Unlike ordinary engineering heat transfer problems, the human body acts as an internal heat source in a clothing situation, thus forming a temperature gradient between body and ambient, and the sign of this gradient often changes as the ambient temperature varies. Our body also perspires and the sweat evaporates, an effective body cooling process via phase change. To bring all the variables into analysis quickly escalates into a formidable task. This work attempts to unravel the problem from a physics perspective, focusing on several rarely noticed yet critically important mechanisms involved so as to offer a clear and accurate depiction of the principles in clothing thermal comfort.