In Human-Computer Interaction, gesture recognition is a prominent topic. Human-computer interaction (HCI) allows computers to recognize and interpret human gestures as commands. Gesture recognition is important for ease of use to operate computer machines. It has wider range of applications in the area like talking with machine, medical operation, computer game control, control of home appliances, car control driving and communication. In this research work A real-time Hand Gesture Recognition System is proposed with hybrid approach of Convolution Neural Network (CNN) and the Recurrent Neural Network (RNN). Moreover experimentation with pre-trained VGG16 is carried out with LSTM. Here LSTM is used to replace the final three layers of the VGG16 architecture, and a soft-max layer is used to produce the output. The integrated model is recognizing both static and dynamic hand motions. Proposed model has obtained training accuracy as 92.71% and validation accuracy is 87.50%.

Convolution Neural Network (CNN), Human-Computer Interaction (HCI), Recurrent Neural Network (RNN), LSTM (Long Short Term Memory)

[1] Sanmukh Kaur, Anuranjana, “Electronic Device Control Using Hand Gesture Recognition System for Differently Abled”, 8th International Conference on Cloud Computing, Data Science & Engineering, 2018.
[2] Youngwook Kim And Brian Toomajian, “Hand Gesture Recognition Using Micro-Doppler Signatures With Convolutional Neural Network”, IEEE Access Vol.4, 2016.
[3] Soeb Hussain, Rupal Saxena, Xie Han, Jameel Ahmed Khan, Hyunchul Shin, “Hand Gesture Recognition Using Deep Learning”, International SoC Design Conference, 2017.
[4] Md Rashedul Islam, Rasel Ahmed Bhuiyan, Ummey Kulsum Mitu, Jungpil Shin, “Hand Gesture Feature Extraction Using Deep Convolutional Neural Network for Recognizing American Sign Language”, 4th International Conference on Frontiers of Signal Processing, 2018.
[5] Panwar, M., February. Hand gesture recognition based on shape parameters. In 2012 international conference on computing, communication and applications, pp.1-6, 2012. IEEE.
[6] Zhang, L.G., Chen, Y., Fang, G., Chen, X. and Gao, W., October. A vision-based sign language recognition system using tied-mixture density HMM. In Proceedings of the 6th international conference on Multimodal interfaces, pp.198-204, 2004.
[7] Molchanov, P., Gupta, S., Kim, K. and Pulli, K., May. Multi-sensor system for driver`s hand-gesture recognition. In 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), Vol.1, pp.1-8, 2015. IEEE.
[8] Jing-Hao Sun, Shu-Bin Zhang, “Research on the Hand Gesture Recognition Based on Deep Learning”, 12th International Symposium on Antennas, Propagation and EM Theory, 2018.
[9] Anush Ananthakumar, “Efficient Face and Gesture Recognition for Time Sensitive Application”, IEEE Southwest Symposium on Image Analysis and Interpretation, 2018.
[10] Mohd. Baqir Khan, Kavya Mishra, Mohammed Abdul Qadeer, “Gesture Recognition using Open-CV”, 7th International Conference on Communication Systems and Network Technologies, 2017.
[11] Arathi P.N, S.Arthika, S.Ponmithra, K.Srinivasan, V.Rukkumani, "Gesture-Based Home Automation System", 2017 International Conference on Nextgen Electronic Technologies: Silicon to Software, 2017.
[12] Kenneth Lai and Svetlana N. Yanushkevich, “CNN+RNN Depth and Skeleton based Dynamic Hand Gesture Recognition”, 24th International Conference on Pattern Recognition, 2018.
[13]. P Raghu Veera Chowdary, M Nagendra Babu, Thadigotla Venkata Subbareddy, Bommepalli Madhava Reddy, V Elamaran, “Image Processing Algorithms for Gesture Recognition using Communication Control and Computing Technologies, 2014.
[14]. Chih-Hung Wu and Chang Hong Lin, “Depth-based hand gesture recognition for home appliance control”, IEEE 17th International Symposium on Consumer Electronics, 2013.
[15] Ransalu Senanayake, Sisil Kumarawadu, “A Robust Vision-based Hand Gesture Recognition System for Appliance Control in Smart Homes”, IEEE International Conference on Signal Processing, Communication and Computing, 2012.
[16] Archana S. Ghotkar, Rucha Khatal, Sanjana Khupase, Surbhi Asati &Mithila Hadap, “Hand Gesture Recognition for Indian Sign Language”, International Conference on Computer Communication and Informatics, 2012.
[17] Lan Tiantian, Shen Jinyuan, Liu Runjie, Guo Yingying, “Hand Gesture Recognition Based on Improved Histograms of Oriented Gradients”, The 27th Chinese Control and Decision Conference, 2015.
[18] Guillaume Plouffe and Ana-Maria Cretu, “Static and Dynamic Hand Gesture Recognition in Depth Data Using Dynamic Time Warping”, IEEE Transactions on Instrumentation and Measurement, Vol.65, Issue.2, 2016.
[19] Dong-Woo Lee, Jeong-Mook Lim, John Sunwoo, Il-Yeon Cho and Cheol-Hoon Lee, “Actual Remote Control: A Universal Remote Control using Hand Motions on a Virtual Menu”, IEEE Transactions on Consumer Electronics, Vol.55, Issue.3, 2009.
[20] Bin Xie1, Xiaoyu He1, Yi Li1, “RGB-D static gesture recognition based on convolutional neural network”, The Journal of Engineering, Vol. 2018, Issue.16, 2018.
[21] Nikam, A.S. and Ambekar, A.G., November. Sign language recognition using image based hand gesture recognition techniques. In 2016 online international conference on green engineering and technologies (IC-GET), pp.1-5, 2016. IEEE.
[22] Haria, A., Subramanian, A., Asokkumar, N., Poddar, S. and Nayak, J.S., Hand gesture recognition for human computer interaction. Procedia computer science, 115, pp.367-374, 2017.
[23] Oyedotun, O.K. and Khashman, A., 2017. Deep learning in vision-based static hand gesture recognition. Neural Computing and Applications, Vol.28, Issue.12, pp.3941-3951, 2017.
[24] Hussain, S., Saxena, R., Han, X., Khan, J.A. and Shin, H., 2017, November. Hand gesture recognition using deep learning. In 2017 International SoC design conference (ISOCC), pp.48-49, 2017. IEEE.
[25] Lu, W., Tong, Z. and Chu, J., 2016. Dynamic hand gesture recognition with leap motion controller. IEEE Signal Processing Letters, Vol.23, Issue.9, pp.1188-1192, 2016.
[26] De Smedt, Q., Wannous, H. and Vandeborre, J.P., 2016. Skeleton-based dynamic hand gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp.1-9, 2016.
[27] Xu, P., 2017. A real-time hand gesture recognition and human-computer interaction system. arXiv preprint arXiv:1704.07296.
[28] Sun, J.H., Ji, T.T., Zhang, S.B., Yang, J.K. and Ji, G.R., 2018, December. Research on the hand gesture recognition based on deep learning. In 2018 12th International symposium on antennas, propagation and EM theory (ISAPE), pp.1-4, 2018. IEEE.
[29] Devineau, G., Moutarde, F., Xi, W. and Yang, J., 2018, May. Deep learning for hand gesture recognition on skeletal data. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp.106-113, 2018. IEEE.
[30] Zengeler, N., Kopinski, T. and Handmann, U., 2018. Hand gesture recognition in automotive human–machine interaction using depth cameras. Sensors, Vol.19, Issue.1, p.59, 2018.
[31] Kellogg, B., Talla, V. and Gollakota, S., 2014. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pp.303-316, 2014.
[32] Abdelnasser, H., Youssef, M. and Harras, K.A., 2015, April. Wigest: A ubiquitous wifi-based gesture recognition system. In 2015 IEEE conference on computer communications (INFOCOM), pp.1472-1480, 2015. IEEE.
[33] Fang, Y., Wang, K., Cheng, J. and Lu, H., 2007, July. A real-time hand gesture recognition method. In 2007 IEEE International Conference on Multimedia and Expo, pp.995-998, 2007. IEEE.
[34] Ren, Z., Yuan, J., Meng, J. and Zhang, Z., 2013. Robust part-based hand gesture recognition using kinect sensor. IEEE transactions on multimedia, Vol.15, Issue.5, pp.1110-1120, 2013.
[35] Lin, H.I., Hsu, M.H. and Chen, W.K., 2014, August. Human hand gesture recognition using a convolution neural network. In 2014 IEEE International Conference on Automation Science and Engineering (CASE), pp.1038-1043, 2014. IEEE.