Abstract
Classification is a data mining (machine learning) technique used to predict group membership for data instances. There are several classification techniques that can be used for classification purpose. In this paper, we present the basic classification techniques. Later we discuss some major types of classification method including Bayesian networks, decision tree induction, k-nearest neighbor classifier and Support Vector Machines (SVM) with their strengths, weaknesses, potential applications and issues with their available solution. The goal of this study is to provide a comprehensive review of different classification techniques in machine learning. This work will be helpful for both academia and new comers in the field of machine learning to further strengthen the basis of classification methods.
References
Ghahramani Z. "Unsupervised learning," in Advanced lectures on machine learning, ed: Springer, 2004; pp. 72-112. https://doi.org/10.1007/978-3-540-28650-9_5
Kotsiantis SB, Zaharakis I, Pintelas P. Supervised machine learning: A review of classification techniques. ed, 2007.
Zhang D, Nunamaker JF. Powering e-learning in the new millennium: an overview of e-learning and enabling technology. Information Systems Frontiers 2003; 5: 207-218. https://doi.org/10.1023/A:1022609809036
Maimon O, Rokach L. Introduction to supervised methods, in Data Mining and Knowledge Discovery Handbook, ed: Springer, 2005 pp. 149-164.
Ng A. "CS229 Lecture notes."
Kesavaraj G, Sukumaran S. A study on classification techniques in data mining. in Computing, Communications and Networking Technologies (ICCCNT), 2013 Fourth International Conference on, 2013; pp. 1-7.
Singh M, Sharma S, Kaur A. Performance Analysis of Decision Trees. International Journal of Computer Applications 2013; 71.
Baradwaj BK, Pal S. Mining educational data to analyze students' performance. arXiv preprint arXiv:1201.3417, 2012.
Dunham MH. Data mining: Introductory and advanced topics: Pearson Education India, 2006.
Kantardzic M. Data mining: concepts, models, methods, and algorithms: John Wiley & Sons, 2011.
Twa MD, Parthasarathy S, Roberts C, Mahmoud AM, Raasch TW, Bullimore MA. Automated decision tree classification of corneal shape. Optometry and vision science: official publication of the American Academy of Optometry 2005; 82: 1038. https://doi.org/10.1097/01.opx.0000192350.01045.6f
Brodley CE, Utgoff PE. Multivariate versus univariate decision trees: Citeseer, 1992.
Jang J-SR. ANFIS: adaptive-network-based fuzzy inference system. Systems, Man and Cybernetics, IEEE Transactions on, 1993; 23: 665-685. https://doi.org/10.1109/21.256541
Rutkowski L, Pietruczuk L, Duda P, Jaworski M. Decision trees for mining data streams based on the McDiarmid's bound. Knowledge and Data Engineering, IEEE Transactions on, 2013; 25: 1272-1279. https://doi.org/10.1109/TKDE.2012.66
Patil DD, Wadhai V, Gokhale J. Evaluation of decision tree pruning algorithms for complexity and classification accuracy, 2010.
Quinlan JR. Induction of decision trees. Machine learning 1986; 1: 81-106. https://doi.org/10.1007/BF00116251
Quinlan JR. Simplifying decision trees. International Journal of man-Machine Studies 1987; 27: 221-234. https://doi.org/10.1016/S0020-7373(87)80053-6
Sharma S, Agrawal J, Agarwal S. Machine learning techniques for data mining: A survey, in Computational Intelligence and Computing Research (ICCIC), 2013 IEEE International Conference on, 2013; pp. 1-6.
Bhukya DP, Ramachandram S. Decision tree induction: an approach for data classification using AVL-tree. International Journal of Computer and Electrical Engineering 2010; 2: 660. https://doi.org/10.7763/IJCEE.2010.V2.208
Adhatrao K, Gaykar A, Dhawan A, Jha R, Honrao V. Predicting Students' Performance using ID3 and C4. 5 Classification Algorithms, arXiv preprint arXiv:1310.2071, 2013.
Phyu TN. Survey of classification techniques in data mining, in Proceedings of the International MultiConference of Engineers and Computer Scientists 2009; pp. 18-20.
Yang Y, Webb GI. Discretization for naive-Bayes learning: managing discretization bias and variance. Machine learning 2009; 74: 39-74. https://doi.org/10.1007/s10994-008-5083-5
Friedman N, Goldszmidt M. Discretizing continuous attributes while learning Bayesian networks, in Icml 1996; pp. 157-165.
Wang S-C, Gao R, Wang L-M. Bayesian network classifiers based on Gaussian kernel density. Expert Systems with Applications, 2016.
Myllymäki P. Advantages of Bayesian Networks in Data Mining and Knowledge Discovery Available: http://www.bayesit.com/docs/advantages.html
Cover T, Hart P. Nearest neighbor pattern classification. IEEE Transactions on Information Theory 1967; 13: 21-27. https://doi.org/10.1109/TIT.1967.1053964
Wu X, Kumar V, Quinlan JR, Ghosh J, Yang Q, Motoda H, et al. Top 10 algorithms in data mining. Knowledge and Information Systems 2008; 14: 1-37. https://doi.org/10.1007/s10115-007-0114-2
Bhatia N. Survey of nearest neighbor techniques. arXiv preprint arXiv:1007.0085, 2010.
Teknomo K. Strengths and weaknesse of K Nearest Neighbor. Available: http://people.revoledu.com/kardi/tutorial/ KNN/Strength%20and%20Weakness.htm
Li H, Liu L, Zhang X, Wang S. Hike: A High Performance kNN Query Processing System for Multimedia Data, in 2015 IEEE Conference on Collaboration and Internet Computing (CIC), 2015; pp. 296-303. https://doi.org/10.1109/CIC.2015.44
Kumar N, Obi Reddy G, Chatterjee S, Sarkar D. An application of ID3 decision tree algorithm for land capability classification. Agropedology 2013; 22: 35-42.
Shao X, Zhang G, Li P, Chen Y. Application of ID3 algorithm in knowledge acquisition for tolerance design. Journal of Materials Processing Technology 2001; 117: 66-74. https://doi.org/10.1016/S0924-0136(01)01016-0
Tan Y, Qi Z, Wang J. Applications of ID3 algorithms in computer crime forensics, in Multimedia Technology (ICMT), 2011 International Conference on, 2011; pp. 4854-4857.
Zou K, Sun W, Yu H, Liu F. ID3 Decision Tree in Fraud Detection Application, in Computer Science and Electronics Engineering (ICCSEE), 2012 International Conference on, 2012; pp. 399-402.
Amin RK, Indwiarti, Sibaroni Y. Implementation of decision tree using C4.5 algorithm in decision making of loan application by debtor (Case study: Bank pasar of Yogyakarta Special Region), in Information and Communication Technology (ICoICT ), 2015 3rd International Conference on, 2015; pp. 75-80.
Li B, Shen B, Wang J, Chen Y, Zhang T. A Scenario-Based Approach to Predicting Software Defects Using Compressed C4.5 Model, in Computer Software and Applications Conference (COMPSAC), 2014 IEEE 38th Annual, 2014; pp. 406-415.
Soliman SA, Abbas S, Salem ABM. Classification of thrombosis collagen diseases based on C4.5 algorithm, in 2015 IEEE Seventh International Conference on Intelligent Computing and Information Systems (ICICIS), 2015; pp. 131-136. https://doi.org/10.1109/IntelCIS.2015.7397209
Hehui Q, Zhiwei Q. Feature selection using C4.5 algorithm for electricity price prediction, in 2014 International Conference on Machine Learning and Cybernetics, 2014; pp. 175-180. https://doi.org/10.1109/ICMLC.2014.7009113
Duan F, Zhao Z, Zeng X. Application of Decision Tree Based on C4.5 in Analysis of Coal Logistics Customer, in Intelligent Information Technology Application, 2009. IITA 2009. Third International Symposium on, 2009; pp. 380-383.
Seet AM, Zualkernan IA. An Adaptive Method for Selecting Question Pools Using C4.5, in 2010 10th IEEE International Conference on Advanced Learning Technologies, 2010; pp. 86-88.
Zhang L, Ji Q. A Bayesian network model for automatic and interactive image segmentation, Image Processing, IEEE Transactions on, 2011; 20: 2582-2593. https://doi.org/10.1016/j.trc.2006.11.001
Zhang K, Taylor MA. Effective arterial road incident detection: a Bayesian network based algorithm. Transportation Research Part C: Emerging Technologies 2006; 14: 403-417.
Xiao X, Leedham G. Signature verification using a modified Bayesian network. Pattern Recognition 2002; 35: 983-995. https://doi.org/10.1016/S0031-3203(01)00088-7
Aoki S, Shiba M, Majima Y, Maekawa Y. Nurse call data analysis using Bayesian network modeling, in Aware Computing (ISAC), 2010 2nd International Symposium on, 2010; pp. 272-277.
Chattopadhyay S, Davis RM, Menezes DD, Singh G, Acharya RU, Tamura T. Application of Bayesian classifier for the diagnosis of dental pain, Journal of Medical Systems 36: 2012; 1425-1439. https://doi.org/10.1007/s10916-010-9604-y
Bashar A, Parr G, McClean S, Scotney B, Nauck D. Knowledge discovery using Bayesian network framework for intelligent telecommunication network management, in Knowledge Science, Engineering and Management, ed: Springer, 2010; pp. 518-529.
Kumar M, Rath SK. Microarray data classification using Fuzzy K-Nearest Neighbor, in Contemporary Computing and Informatics (IC3I), 2014 International Conference on, 2014; pp. 1032-1038.
Rizwan M, Anderson DV. Using k-Nearest Neighbor and Speaker Ranking for Phoneme Prediction, in Machine Learning and Applications (ICMLA), 2014 13th International Conference on, 2014; pp. 383-387.
Kasemsumran P, Auephanwiriyakul S, Theera-Umpon N. Face recognition using string grammar fuzzy K-nearest neighbor, in 2016 8th International Conference on Knowledge and Smart Technology (KST), 2016; pp. 55-59.
Ismail N, Rahiman MHF, Taib MN, Ali NAM, Jamil M, Tajuddin SN. The grading of agarwood oil quality using k-Nearest Neighbor (k-NN), in Systems, Process & Control (ICSPC), 2013 IEEE Conference on, 2013; pp. 1-5.
Tiwari AK, Srivastava R. Feature based classification of nuclear receptors and their subfamilies using fuzzy K-nearest neighbor, in Computer Engineering and Applications (ICACEA), 2015 International Conference on Advances in, 2015; pp. 24-28.
Li S, Shen Z, Xiong G. A k-nearest neighbor locally weighted regression method for short-term traffic flow forecasting, in 2012 15th International IEEE Conference on Intelligent Transportation Systems, 2012; pp. 1596-1601.
Munisami T, Ramsurn M, Kishnah S, Pudaruth S. Plant Leaf Recognition Using Shape Features and Colour Histogram with K-nearest Neighbour Classifiers. Procedia Computer Science 2015; 58: 740-747. https://doi.org/10.1016/j.procs.2015.08.095
Mandhala VN, Sujatha V, Devi BR. Scene classification using support vector machines, in Advanced Communication Control and Computing Technologies (ICACCCT), 2014 International Conference on, 2014; pp. 1807-1810.
Zhao Y, Zhu S, Yu J, Wang L. Predicting corporate financial distress by PCA-based support vector machines, in 2010 International Conference on Networking and Information Technology, 2010; pp. 373-376. https://doi.org/10.1109/ICNIT.2010.5508491
Aydin I, Karakose M, Akin E. Artificial immune based support vector machine algorithm for fault diagnosis of induction motors, in Electrical Machines and Power Electronics, 2007. ACEMP '07. International Aegean Conference on, 2007; pp. 217-221.
Yehui L, Yuye Y, Liang H. Fault diagnosis of analog circuit based on support vector machines, in Communications Technology and Applications, 2009. ICCTA '09. IEEE International Conference on, 2009; pp. 40-43.
Jialong H, Yanbin W. Classification of the enterprise market competition based on support vector machines, in 2010 Chinese Control and Decision Conference, 2010; pp. 1644-1647. https://doi.org/10.1109/CCDC.2010.5498321
Viswanath P, Sarma TH. An improvement to k-nearest neighbor classifier, in Recent Advances in Intelligent Computational Systems (RAICS), 2011 IEEE, 2011; pp. 227-231. https://doi.org/10.1109/RAICS.2011.6069307
Dudani SA. The Distance-Weighted k-Nearest-Neighbor Rule. IEEE Transactions on Systems, Man, and Cybernetics, 1976; SMC-6: 325-327. https://doi.org/10.1109/TSMC.1976.5408784
Cunningham P, Delany SJ. k-Nearest neighbour classifiers, 2007.
Chen J, Luo D-l, Mu F-X. An improved ID3 decision tree algorithm, in Computer Science & Education, 2009. ICCSE '09. 4th International Conference on, 2009; pp. 127-130.
Thakur D, Markandaiah N, Raj DS. Re optimization of ID3 and C4.5 decision tree, in Computer and Communication Technology (ICCCT), 2010 International Conference on, 2010; pp. 448-450.
Huang M, Niu W, Liang X. An improved Decision Tree classification algorithm based on ID3 and the application in score analysis, in 2009 Chinese Control and Decision Conference, 2009; pp. 1876-1879. https://doi.org/10.1109/CCDC.2009.5192865
Mantas CJ, Abellán J. Credal-C4.5: Decision tree based on imprecise probabilities to classify noisy data, Expert Systems with Applications 2014; 41: 4625-4637. https://doi.org/10.1016/j.eswa.2014.01.017
Mori J, Mahalec V. Inference in hybrid Bayesian networks with large discrete and continuous domains. Expert Systems with Applications 2016; 49: 1-19. https://doi.org/10.1016/j.eswa.2015.11.019
Hobæk Haff I, Aas K, Frigessi A, Lacal V. Structure learning in Bayesian Networks using regular vines. Computational Statistics & Data Analysis 2016; 101: 186-208. https://doi.org/10.1016/j.csda.2016.03.003
Babu VS, Viswanath P. Rough-fuzzy weighted k-nearest leader classifier for large data sets. Pattern Recognition 2009; 42: 1719-1731. https://doi.org/10.1016/j.patcog.2008.11.021
Duda RO, Hart PE, Stork DG. Pattern classification: John Wiley & Sons, 2012.
Guttman A. R-trees: a dynamic index structure for spatial searching 1984; 14: ACM.
Moraes D, Wainer J, Rocha A. Low false positive learning with support vector machines. Journal of Visual Communication and Image Representation 2016; 38: 340-350. https://doi.org/10.1016/j.jvcir.2016.03.007
Carrizosa E, Nogales-Gómez A, Romero Morales D. Clustering categories in support vector machines, Omega.
Abe S. Fuzzy support vector machines for multilabel classification. Pattern Recognition 2015; 48: 2110-2117. https://doi.org/10.1016/j.patcog.2015.01.009
Vapnik VN. The Nature of Statistical Learning Theory, 1995.
Nizar A, Dong Z, Wang Y. Power utility nontechnical loss analysis with extreme learning machine method. Power Systems, IEEE Transactions on, 2008; 23: 946-955. https://doi.org/10.1109/TPWRS.2008.926431
Xiao H, Peng F, Wang L, Li H. Ad hoc-based feature selection and support vector machine classifier for intrusion detection, in 2007 IEEE International Conference on Grey Systems and Intelligent Services, 2007; pp. 1117-1121. https://doi.org/10.1109/GSIS.2007.4443446
Berwick R. An Idiot’s guide to Support vector machines (SVMs).
Ahmad I, Abdulah AB, Alghamdi AS. Towards the designing of a robust intrusion detection system through an optimized advancement of neural networks, in Advances in Computer Science and Information Technology, ed: Springer, 2010; pp. 597-602.
Han J, Kamber M, Pei J. Data mining: concepts and techniques: Elsevier 2011.
SVM. Available: http://www.nickgillian.com/wiki/pmwiki. php/GRT/SVM
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright (c) 2017 Aized Amin Soofi , Arshad Awan