The emerging of need of domestic robots in real world applications has raised enormous need for instinctive and interaction among human and computer interaction (HCI). In a few conditions where humans can’t contact hardware, the hand motion recognition framework more suitable. Hand gesture recognition is used on controlling robots, portable controllers, or application in smart home. Gesture recognition is mainly applicable for video conferencing, sign language recognition, distance learning and in some forensic identification. Based on fingers’ angles relative to the wrist, a finger angle prediction algorithm and a template matching metric are proposed. All possible gesture types of the captured image are first predicted, and then evaluated and compared to the template image to achieve the classification. Unlike other template matching methods relying highly on large training set, this scheme possesses high flexibility since it requires only one image as the template, and can classify gestures formed by different combinations of fingers.