Abstract—Human action classification has been and still a highly interesting and important research topic. The research results are capable of analyzing human action that we visually perceive in many aspects. Therefore the research requires an effective and competent approach to accurately interpret human action. In this paper, we present a novel model called the body energy action model for finding actual semantic action. The model is based on the fundamental concepts of biomechanics that human movement in different classes is likely to spend different amounts of energy. The model is classified by using kernel-base data fusion to obtain from the 5-fold cross validation. Experimental results show that the proposed provides much more authentic meaning of human actions.