Open Access   Article Go Back

A Survey: Analytics of Web Log File Using Map Reduce and Hadoop

Rahul Pateriya1 , Nishchol Mishra2 , Sanjeev Sharma3

Section:Survey Paper, Product Type: Journal Paper
Volume-4 , Issue-5 , Page no. 25-30, May-2016

Online published on May 31, 2016

Copyright © Rahul Pateriya, Nishchol Mishra, Sanjeev Sharma . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

View this paper at   Google Scholar | DPI Digital Library

How to Cite this Paper

  • IEEE Citation
  • MLA Citation
  • APA Citation
  • BibTex Citation
  • RIS Citation

IEEE Style Citation: Rahul Pateriya, Nishchol Mishra, Sanjeev Sharma, “A Survey: Analytics of Web Log File Using Map Reduce and Hadoop,” International Journal of Computer Sciences and Engineering, Vol.4, Issue.5, pp.25-30, 2016.

MLA Style Citation: Rahul Pateriya, Nishchol Mishra, Sanjeev Sharma "A Survey: Analytics of Web Log File Using Map Reduce and Hadoop." International Journal of Computer Sciences and Engineering 4.5 (2016): 25-30.

APA Style Citation: Rahul Pateriya, Nishchol Mishra, Sanjeev Sharma, (2016). A Survey: Analytics of Web Log File Using Map Reduce and Hadoop. International Journal of Computer Sciences and Engineering, 4(5), 25-30.

BibTex Style Citation:
@article{Pateriya_2016,
author = {Rahul Pateriya, Nishchol Mishra, Sanjeev Sharma},
title = {A Survey: Analytics of Web Log File Using Map Reduce and Hadoop},
journal = {International Journal of Computer Sciences and Engineering},
issue_date = {5 2016},
volume = {4},
Issue = {5},
month = {5},
year = {2016},
issn = {2347-2693},
pages = {25-30},
url = {https://www.ijcseonline.org/full_paper_view.php?paper_id=897},
publisher = {IJCSE, Indore, INDIA},
}

RIS Style Citation:
TY - JOUR
UR - https://www.ijcseonline.org/full_paper_view.php?paper_id=897
TI - A Survey: Analytics of Web Log File Using Map Reduce and Hadoop
T2 - International Journal of Computer Sciences and Engineering
AU - Rahul Pateriya, Nishchol Mishra, Sanjeev Sharma
PY - 2016
DA - 2016/05/31
PB - IJCSE, Indore, INDIA
SP - 25-30
IS - 5
VL - 4
SN - 2347-2693
ER -

VIEWS PDF XML
1813 1727 downloads 1485 downloads
  
  
           

Abstract

The web is vast, diverse and dynamic and increases scalability, temporal data and multimedia issues respectively. The expansion of the Internet has given rise to a wealth of data as big data that is now available for user access. Different types of data must be managed and organized so that can be accessed by different users effectively and efficiently. The log analysis is an important issue for the web application. Log file is not to be over emphasized as a source of information in systems and network management. Whereas conduct efficient investigation and gathering of use full information need to correlate different log file. Task of analyzing event log files with the ever-increasing size and complexity of today’s event logs has become cumbrous to carry out manually. Nowadays latest spotlighted is automatic analysis of these logs files. This paper is a review of the basics of log analysis as big data in the web environment.

Key-Words / Index Term

Web Application, Log File, Data Mining, Big Data

References

[1]. Xiuqin Lin; Peng Wang; Bin Wu, "Log analysis in cloud computing environment with Hadoop and Spark," in Broadband Network & Multimedia Technology (IC-BNMT), 2013 5th IEEE International Conference on , vol., no., pp.273-276, 17-19 Nov. 2013
[2]. Chaofei Wang; Jing Chen; Xiaopeng Liu; Jinwei Zhao, "An improved deep log analysis method based on data reconstruction," in Cloud Computing and Intelligence Systems (CCIS), 2014 IEEE 3rd International Conference on , vol., no., pp.86-90, 27-29 Nov. 2014
[3]. da Silva Machado, Roger; Borges Almeida, Ricardo; Correa Yamin, Adenauer; Marilza Pernas, Ana, "LogA-DM: An Approach of Dynamic Log Analysis," in Latin America Transactions, IEEE (Revista IEEE America Latina) , vol.13, no.9, pp.3096-3102, Sept. 2015
[4]. Xiaokui Shu; Smiy, J.; Danfeng Yao; Heshan Lin, "Massive distributed and parallel log analysis for organizational security," in Globecom Workshops (GC Wkshps), 2013 IEEE , vol., no., pp.194-199, 9-13 Dec. 2013
[5]. Hingave, H.; Ingle, R., "An approach for MapReduce based log analysis using Hadoop," in Electronics and Communication Systems (ICECS), 2015 2nd International Conference on , vol., no., pp.1264-1268, 26-27 Feb. 2015
[6]. K Savitha and MS Vijaya , "Mining of Web Server Logs in a Distributed Cluster using Big Data Technologies" , International Journal of Advanced Computer Science and Applications (IJACSA), vol. 5 , 2014
[7]. T. K. Das , "BIG Data Analytics: A Framework for Unstructured Data Analysis" , International Journal of Engineering and Technology (IJET) , vol. 5 , no. 1 , 2013
[8]. Jiang Dawei, K.H. Antony and Gang Chen , "MAP-JOINREDUCE: Toward Scalable and Efficient Data Analysis on Large Clusters" , IEEE Transactions on Knowledge and Data Engineering, pp.1299 -1311 , 2011
[9]. Pavlo Andrew, Paulson Erik, Rasin Alexander, J. Daniel, J. David, De Witt, Samuel Madden and Michael Stonebraker , "A Comparison of Approaches to Large-Scale Data Analysis" , ACM SIGMOD International Conference on Management of data , pp.165 -178 , 2009
[10]. W. Xu, L. Huang, A. Fox, D. Patterson, M. Jordan. "Online System Problem Detection by Mining Patterns of Console Logs". In the Proceeding of ICDM '09 Proceedings of the 2009 Ninth IEEE International Conference on Data Mining.
[11]. AiLing Duan et. al. “Research and Practice of Distributed Parallel Search Algorithm on Hadoop_MapReduce”, 2012 International Conference on Control Engineering and Communication Technology, 2012 IEEE DOI 10.1109/ICCECT.2012.131, pp 105108
[12]. D. Agrawal, A. El Abbadi, S. Antony, and S. Das. Data Management Challenges in Cloud Computing Infrastructures. In DNIS, pages 1–10, 2010.
[13]. White paper on “Solution Brief Big Data in the Cloud: Converging Technologies , How to Create Competitive Advantage Using Cloud- Based Big Data Analytics.
[14]. Tharam Dillon et. al. “Cloud Computing: Issues and Challenges”, 2010 24th IEEE International Conference on Advanced Information Networking and Applications,
[15]. D. Agrawal, S. Das, and A. E. Abbadi. Big data and cloud computing: New wine or just new bottles? PVLDB, 3(2):1647– 1648.
[16]. Nacim Fateh Chikhi, Bernard Rothenburger, Nathalie Aussenac-Gilles “A Comparison of Dimensionality Reduction Techniques for Web Structure Mining”, Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence 2007, pp 116-119.
[17]. Lefteris Moussiades, Athena Vakali, "Mining the Community Structure of a Web Site," bci Fourth Balkan Conference in Informatics 2009, pp.239-244.
[18]. Toufiq Hossain Kazi, Wenying Feng and Gongzhu Hu, “Web Object Prefetching: Approaches and a New Algorithm”, IEEE 2010, pp 115-120.
[19]. Brijendra Singh and Hemant Kumar Singh, “Web Data Mining Research: A Survey”, IEEE 2010.
[20]. Kavita Sharma, Gulshan Shrivastava and Vikas Kumar, “Web Mining: Today and Tomorrow”, IEEE 2011, pp 399-403.
[21]. WANG Yong-gui and JIA Zhen, “Research on Semantic Web Mining” IEEE 2010, pp 67-70.
[22]. P. Sampath, C. Ramesh, T. Kalaiyarasi, S. Sumaiya Banu and G. Arul Selvan, “An Efficient Weighted Rule Mining for Web Logs Using Systolic Tree”, IEEE 2012, pp 432-436.
[23]. Nizar R. Mabroukeh and C. I. Ezeife, “Semantic-rich Markov Models for Web Prefetching”, IEEE 2009, pp 465-470.
[24]. A.B.M.Rezbaul Islam and Tae-Sun Chung, “An Improved Frequent Pattern Tree Based Association Rule Mining Technique”, IEEE 2011.
[25]. R.Agrawal, and R.Srikant, “Fast algorithms for mining association rules”, In VLDB’94, pp. 487-499, 1994 Borges and M. Levene,”A dynamic clustering-based markov model for web usage Mining”, cs.IR/0406032, 2004.
[26]. Zhu, J., Hong, J. and Hughes, J. G. (2002a) Using Markov Chains for Link Prediction in Adaptive Web Sites. In Proc. of Soft-Ware 2002: the First International Conference on Computing in an Imperfect World, pp. 60-73, Lecture Notes in Computer Science, Springer, Belfast, April.
[27]. K.Ramu Dr.R.Sugumar and B.Shanmugasundaram “A Study on Web Prefetching Techniques” Journal of Advances in Computational Research: An International Journal Vol. 1 No. 1-2 (January-December, 2012)