MapReduce Parallel Programming Model: A State-of-the-Art Survey
Tóm tắt
Từ khóa
Tài liệu tham khảo
Shamsi, J., Khojaye, M.A., Qasmi, M.A.: Data-intensive cloud computing: requirement, expectations, challenges and solutions. J. Grid Comput. 11(2), 281–310 (2013)
Meng, X., Ci, X.: Big data management: concepts, techniques and challenges. J. Comput. Res. Dev. 50(1), 146–169 (2013)
Wang, S., Wang, H., Qin, X., Zhou, X.: Architecting big data: challenges, studies and forecasts. Chin. J. Comput. 34(10), 1741–1752 (2013)
Mell, P., Grance, T.: Definition of Cloud Computing. Technical report, National Institute of Standards and Technology (NIST) (2009)
Sakr, S., Liu, A., Batista, D.M., Alomari, M.: A survey of large scale data management approaches in cloud environments. IEEE Commun. Surv. Tutor. 13(3), 311–336 (2011)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–133 (2008)
Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google file system. In: Proceedings of 19th ACM Symposium on Operating Systems Principles, pp. 29–43. ACM (2003)
Apache Hadoop. http://hadoop.apache.org/ . Accessed Oct. 2014
Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: PigLatin: a not-so-foreign language for data processing. In: Proceedings of ACM SIGMOD International Conference of Management of Data (2008)
Mahou. http://mahout.apache.org/ . Accessed Oct 2014
Urbani, J., Kotoulas, S., Maassen, J., Harmelen, F.V., Bal, H.: WebPIE: a web-scale parallel inference engine using mapreduce. J. Web Semant. 10, 59–75 (2012)
Ding, L., Xin, J., Wang, G., Huang, S.: Efficient skyline query processing of massive data based on MapReduce. Chin. J. Comput. 34(10), 1785–1796 (2011)
Doulkeridis, C., Nørvåg, K.: A survey of large-scale analytical query processing in MapReduce. VLDB J. 23(3), 355–380 (2014)
Li, F., Ooi, B.C., Özsu, M.T., Wu, S.: Distributed data management using MapReduce. ACM Comput. Surv. 46(3), 31 (2014)
Lee, K.H., Lee, Y.J., Choi, H., Chung, Y.D., Moon, B.: Parallel data processing with MapReduce: a survey. ACM SIGMOD Rec. 40(4), 11–20 (2011)
Van Biema, M.: Parallelism in Lisp. In: IJCAI87, pp. 56–61 (1987)
QT Concurrent. http://doc.qt.io/qt-5/qtconcurrent-index.html . Accessed Oct 2015
Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating MapReduce for multi-core and multiprocessor systems. In: Proceedings of IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13–24 (2007)
Talbot, J., Yoo, R.M., Kozyrakis, C.: Phoenix++: modular mapreduce for shared-memory systems. In: Proceedings of 2nd International Workshop on MapReduce and Its Applications, pp. 9–16 (2011)
Disco massive data—minimal code. http://discoproject.org/ . Accessed Oct 2014
Geni, Skynet A Ruby MapReduce Framework. http://skynet.rubyforge.org/ . Accessed Oct 2014
GridGain. http://www.gridgain.com/ . Accessed Oct 2014
Ekanayake, J., Li, H., Zhang, B., Gunarathne, T., Bae, S.-H., Qiu, J., Fox, G.: Twister: a runtime for iterative MapReduce. In: Proceedings of First International Workshop on MapReduce and Its Applications of ACM, pp. 810–818. ACM (2010)
Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Misco: a MapReduce Framework for mobile systems. In: Proceedings of 3rd International Conference on Pervasive Technologies Related to Assistive Environment (2010)
DeWitt, D., Stonebraker, M.: MapReduce: a major step backwards. The Database Column 1 (2008)
Sun, X., He, C., Lu, Y.: ESAMR: an enhanced self-adaptive MapReduce scheduling algorithm. In: Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems, pp. 148–155 (2012)
Husain, M.F., McGlothlin, J., Masud, M.M., Khan, L.R., Thuraisingham, B.: Heuristics-based query processing for large RDF graphs using cloud computing. IEEE Trans. Knowl. Data Eng. 23(9), 1312–1327 (2011)
Sandholm, T., Lai, K.: MapReduce optimization using regulated dynamic prioritization. In: Proceedings of the 11th International Joint Conference on Measurement and Modeling of Computer Systems, pp. 299–310 (2009)
Seo, S., Jang, I., Woo, K., Kim, I., Kim, J.-S., Maeng, S.: HPMR: prefetching and pre-shuffling in shared MapReduce computation environment. In: Proceedings of 2009 IEEE International Conference on Cluster Computing and Workshops (2009)
Polo, J., Carrera, D., Becerra, Y., Torres, J., Ayguade, E., Steinder, M., Whalley, I.: Performance-driven task co-scheduling for MapReduce environments. In: Proceedings of 12th IEEE/IFIP Network Operations and Management Symposium, pp. 373–380 (2010)
Polo, J., Carrera, D., Becerra, Y., Beltran, V., Torres, J., Ayguade, E.: Performance management of accelerated MapReduce workloads in heterogeneous clusters. In: Proceedings of 39th International Conference on Parallel Processing, pp. 653–662 (2010)
Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: EuroSyst, pp. 265–278 (2010)
Yao, Y., Tai, J., Sheng, B., Mi, N.: Scheduling heterogeneous MapReduce jobs for efficiency improvement in enterprise clusters. In: Proceedings of 2013 IFIP/IEEE International Symposium on Integrated Network Management, pp. 872–875 (2013)
Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R., Stoica, I.: Improving MapReduce performance in heterogeneous environments. In: Proceedings of 8th USENIX Symposium on Operating System Design and Implementation, pp. 29–42 (2008)
Chen, Q., Zhang, D., Guo, M., Deng, Q., Guo, S.: SAMR: A self-adaptive MapReduce scheduling algorithm in heterogeneous environment. In: Proceedings of 10th IEEE International Conference on Computer and Information Technology, CIT-2010, 7th IEEE International Conference on Embedded Software and Systems, ICESS-2010, 10th IEEE International Conference on Scalable Computing and Communications, pp. 2736–2743 (2010)
Chen, Q., Guo, M.Y., Deng, Q.N., Zheng, L., Guo, S., Shen, Y.: HAT: history-based auto-tuning MapReduce in heterogeneous environments. J. Supercomput. 64(3), 1038–1054 (2013)
Zhang, X., Feng, Y., Feng, S., Fan, J., Ming, Z.: An effective data locality aware task scheduling method for MapReduce framework in heterogeneous environments. In: Proceedings of 2011 International Conference on Cloud and Service Computing, pp. 235–242 (2011)
Ahmad, F., Chakradhar, S.T., Raghunathan, A., Vijaykumar, T.N.: Tarazu: optimizing MapReduce on heterogeneous clusters. In: Proceedings of 17th International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 61–74 (2011)
Yang, Y., Shi, B., Jiang, B., Xiang, L.: Data-replicas scheduler for heterogeneous MapReduce cluster. J. Netw. 8(5), 1096–1103 (2013)
Sehrish, S., Mackey, G., Shang, P.J., Wang, J., Bent, J.: Supporting HPC analytics applications with access patterns using data restructuring and data-centric scheduling techniques in MapReduce. IEEE Trans. Parallel Distrib. Syst. 24(1), 158–169 (2013)
Peng, D., Dabek, F., Inc, G.: Large-scale incremental processing using distributed transactions and notifications. In: Usenix Symposium on Operating Systems Design & Implementation, pp. 4–6 (2010)
Bu, X., Rao, J., Xu, C.: Interference and locality-aware task scheduling for MapReduce applications in virtual clusters. In: Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, pp. 227–238 (2013)
Yang, H.-C., Dasdan, A., Hsiao, R.-L., Parker, D.S.: Map-Reduce-Merge: simplified relational data processing on large clusters. In: Proceedings of ACM SIGMOD International Conference on Management of Data, pp. 1029–1040 (2007)
Jiang, D., Tung, A.K.H., Chen, G.: Map–Join–Reduce: toward scalable and efficient data analysis on large clusters. IEEE Trans. Knowl. Data Eng. 23(9), 1299–1311 (2011)
Ferrera, P., de Prado, I., Palacios, E., Fernandez-Marquez, J.L., Serugendo, G.D.: Tuple MapReduce: beyond classic MapReduce. In: Proceedings of 12th IEEE International Conference on Data Mining, pp. 260–269 (2012)
Vu, T.-T., Huet, F.: A lightweight continuous jobs mechanism for MapReduce frameworks. In: Proceedings of 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, pp. 269–276 (2013)
Premchaiswadi, W., Pomsaiyud, W.: Optimizing and tuning MapReduce jobs to improve the large-scale data analysis process. Int. J. Intell. Syst. 28(2), 185–200 (2013)
Bu, Y., Howe, B., Balazinska, M., Ernst, M.D.: Haloop: efficient iterative data processing on large clusters. Proc. VLDB Endow. 3(1), 285–296 (2010)
Kambatla, K., Rapolu, N., Jagannathan, S., Grama, A.: Asynchronous algorithms in MapReduce. In: Proceedings of 2010 IEEE International Conference on Cluster Computing, Cluster, pp. 245–254 (2010)
Elnikety, E., Elsayed, T., Ramadan, H.E.: IHadoop: asynchronous iterations for MapReduce. In: Proceedings of 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, pp. 81–90 (2011)
Wang, L., Ni, Z., Zhang, Y., Wu, Z., Tang, L.: Pipelined-MapReduce: an improved MapReduce parallel programming model. In: Proceedings of 4th International Conference on Intelligent Computation Technology and Automation, pp. 871–874 (2011)
Xu, W., Gong, X.J., Li, X.Y.: MapCombine: a lightweight solution to improve the efficiency of iterative MapReduce. In: Proceedings of iCETS 2012, pp. 444–456
Zhang, Y.F., Gao, Q.X., Gao, L.X., Wang, C.R.: iMapReduce: a distributed computing framework for iterative computation. J. Grid Comput. 10(1), 47–68 (2012)
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., Mccauley, M., Franklin, M., Shenker, S., Stoica, I.: Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In: USENIX Symposium on Networked Systems Design and Implementation, vol. 70, pp. 141–146 (2012)
Gunarathne, T., Zhang, B.J., Wu, T.L., Qiu, J.: Scalable parallel computing on clouds using Twister4Azure iterative MapReduce. Future Gener. Comput. Syst. 29(4), 1035–1048 (2013)
Condie, T., Conway, N., Alvaro, P., Hellerstein, M., Elmeleegy, K., Sears, R.: MapReduce Online. EECS Department, University of California, Berkeley. Tech. Rep. http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-136.html
Böse, J. H., Andrzejak, A., Hogqvist, M.: Beyond online aggregation: Parallel and incremental data mining with online Map-Reduce. In: Proceedings of 2010 Workshop on Massive Data Analytics on the Cloud, MDAC 2010, in Association with the 19th Annual World Wide Web Conference (2010)
Phan, L.T.X., Zhang, Z., Loo, B.T., Lee, I.: Real-Time MapReduce Scheduling. Technical Report, University of Pennsylvania (2010)
Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Scheduling for real-time mobile MapReduce systems. In: Proceedings of the 5th ACM International Conference on Distributed Event-Based Systems, pp. 247–258 (2011)
Peng, C.-Z., Jiang, Z.-J., Cai, X.-B., Zhang, Z.-K.: Real-time analytics processing with MapReduce. In: Proceedings of 2012 International Conference on Machine Learning and Cybernetics, vol. 4, pp. 1308–1311 (2012)
Qi, K., Zhao, Z., Fang, J., Ma, Q.: Real-time processing for high speed data stream over large scale data. Chin. J. Comput. 35(3), 477–490 (2012)
Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (2010)
Hong, C., Chen, D., Chen, W., Zheng, W., Lin, H.: MapCG: Writing parallel program portable between CPU and GPU. In: Proceedings of 19th International Conference on Parallel Architecture and Compilation Techniques, pp. 217–226 (2010)
Jiang, W., Ravi, V.T., Agrawal, G.: A Map-reduce system with an Alternate API for multi-core environments. In: Proceedings of 10th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, pp. 84–93 (2010)
Stuart, J.A., Chen, C.-K., Ma, K.-L., Owens, J.D.: Multi-GPU volume rendering using MapReduce. In: Proceedings of 19th ACM International Symposium on High Performance Distributed Computing, pp. 841–848 (2010)
Fang, W., He, B., Luo, Q., Govindaraju, N.K.: Mars: accelerating MapReduce with graphics processors. IEEE Trans. Parallel Distrib. Syst. 22(4), 608–620 (2011)
Elteir, M., Lin, H., Feng, W.-C., Scogland, T.: StreamMR: an optimized MapReduce framework for AMD GPUs. In: Proceedings of 17th IEEE International Conference on Parallel and Distributed Systems, pp. 364–371 (2011)
Chen, L., Agrawal, G.: Optimizing MapReduce for GPUs with effective shared memory usage. In: Proceedings of 21st ACM Symposium on High-Performance Parallel and Distributed Computing, pp. 199–210 (2012)
Basaran, C., Kang, K.D.: Grex: an efficient MapReduce framework for graphics processing units. J. Parallel Distrib. Comput. 73(4), 522–533 (2013)
Chen, R., Chen, H.: Tiled-mapreduce: efficient and flexible mapreduce processing on multicore with tiling. Trans. Archit. Code Optim. 10, 1 (2013)
de Krujif, M., Sankaralingam, K.: MapReduce for the Cell B.E. architecture. IBM J. Res. Dev. 53(5), 10:1–10:12 (2009)
Rafique, M.M., Rose, B., Butt, A.R., Nikolopoulos, D.S.: Supporting MapReduce on large-scale asymmetric multi-core clusters. Oper. Syst. Rev. 43, 25–34 (2009)
Papagiannis, A., Nikolopoulos, D.S.: Rearchitecting MapReduce for heterogeneous multicore processors with explicitly managed memories. In: Proceedings of 39th International Conference on Parallel Processing, pp. 121–130 (2010)
He, C., Weitzel, D., Swanson, D., Lu, Y.: HOG: Distributed Hadoop MapReduce on the grid. In: Proceedings of 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, pp. 1276–1283 (2012)
Heintz, B., Wang, C., Chandra, A., Weissman, J.: Cross-phase optimization in mapreduce. In: Proceedings of 1st IEEE International Conference on Cloud Engineering, pp. 338–347 (2013)
Wang, L., Tao, J., Ranjan, R., Marten, H., Streit, A., Chen, J., Chen, D.: G-Hadoop: MapReduce across distributed data centers for data-intensive computing. Future Gener. Comput. Syst. 29(3), 739–750 (2013)
Mantha, P.K., Luckow, A., Jha, S.: Pilot-MapReduce: an extensible and flexible MapReduce implementation for distributed data. In: Proceedings of 2012 3rd International Workshop on MapReduce and Its Applications, pp. 17–24 (2012)
Marozzo, F., Talia, D., Trunfio, P.: P2P-MapReduce: parallel data processing in dynamic cloud environments. J. Comput. Syst. Sci. 78(5), 1382–1402 (2012)
Wang, G., Butt, A.R., Pandey, P., Gupta, K.: Using realistic simulation for performance analysis of MapReduce setups. In: Proceedings of 1st ACM Workshop on Large-Scale System and Application Performance, pp. 16–29 (2009)
Hammoud, S., Li, M., Liu, Y., Alham, N.K., Liu, Z.: MRSim: a discrete event based MapReduce simulator. In: Proceedings of 2010 Seventh International Conference on Fuzzy Systems and Knowledge Discovery, pp. 2993–2997 (2010)
Huang, D., Shi, X., Ibrahim, S., Lu, L., Liu, H., Wu, S., Jin, H.: MR-Scope: a real-time tracing tool for MapReduce. In: Proceedings of 19th ACM International Symposium on High Performance Distributed Computing, pp. 849–855 (2010)
Teng, F., Yu, L., Magoules, F.: SimMapReduce: a simulator for modeling MapReduce framework. In: Proceedings of the 2011 5th FTRA International Conference on Multimedia and Ubiquitous Engineering, pp. 277–282 (2011)
Wang, K., Lin, X., Tang, W.: Predator—an experience guided configuration optimizer for Hadoop MapReduce. In: Proceedings of 4th IEEE International Conference on Cloud Computing Technology and Science, pp. 419–426 (2012)
Yang, H.L., Luan, Z.Z., Li, W.J., Qian, D.P.: MapReduce workload modeling with statistical approach. J. Grid Comput. 10(2), 279–310 (2012)
Kolberg, W., Marcos, P.D., Anjos, J.C.S., Miyazaki, A.K.S., Geyer, C.R., Arantes, L.B.: MRSG—a MapReduce simulator over SimGrid. Parallel Comput. 39(4–5), 233–244 (2013)
Liu, Y., Li, M.Z., Alham, N.K., Hammoud, S.: HSim: a MapReduce simulator in enabling cloud computing. Future Gener. Comput. Syst. 29(1), 300–308 (2013)
Vianna, E., Comarela, G., Pontes, T., Almeida, J., Almeida, V., Wilkinson, K., Kuno, H., Dayal, U.: Analytical performance models for MapReduce workloads. Int. J. Parallel Prog. 41(4), 495–525 (2013)
Leverich, J., Kozyrakis, C.: On the energy (In) efficiency of Hadoop Clusters. In: Proceedings of HotPower (2009)
Kaushik, R.T., et al.: Evaluation and analysis if GreenHDFS: a selft-adaptive. Energy-conserving variant of the Hadoop distributed file system. In: Proceedings of IEEE CloudCom (2010)
Chen, Y., Alspaugh, S., Borthakur, D., Katz, R.: Energy efficiency for large-scale MapReduce workloads with significant interactive analysis. In: Proceedings of EuroSys (2012)
Yoon, E., Squicciarini, A.: Toward detecting compromised MapReduce workers through log analysis. In: Proceedings of 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (2014)
Roy, I., Setty, S.T.V., Kilzer, A., Shmatikov, V., Witchel, E.: Airavat: security and privacy for MapReduce. In: NSDI (2010)
Wei, W., Du, J., Yu, T., Gu, X.: SecureMR: a service integrity assurance framework for MapReduce. In: Proceedings of 2009 Annual Computer Security Applications Conference (2009)
Chen, X., Huang, Q.: The data protection of MapReduce using homomorphic encryption. In: Proceedings of 4th IEEE International Conference on Software Engineering and Service Science, pp. 419–421 (2013)
Guo, Z., Zhu, X., Guo, L., Kang, S.: Design of a security framework On MapReduce. In: Proceedings of 5th International Conference on Intelligent Networking and Collaborative Systems, pp. 139–145 (2013)
Han, H., Zheng, W.: A privacy data-oriented hierarchical MapReduce programming model. Telkomnika Indones. J. Electr. Eng. 11(8), 4587–4593 (2013)
Zhang, C., Chang, E.-C., Yap, R.H.C.: Tagged-MapReduce: a general framework for secure computing with mixed-sensitivity data on hybrid clouds. In: Proceedings of 14th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (2014)
Apache Pig. http://pig.apache.org/ . Accessed May 2015
Cascading. http://www.cascading.org/ . Accessed May 2015
Scalding. http://www.cascading.org/projects/scalding/ . Accessed May 2015
Pike, R., Dorward, S., Griesemer, R., Quinlan, S.: Interpreting the data: parallel analysis with Sawzall. Sci. Program. 13(4), 277–298 (2005)
Summingbird. https://github.com/twitter/summingbird . Accessed May 2015
Akidau, T., Balikov, A., Chernyak, S., Haberman, J., Lax, R., Mcveety, S., Mills, D., Nordstrom, P., Whittle, S.: MillWheel: faulttolerant stream processing at Internet scale. In: Proceedings of the 39th International Conference on Very Large Data Bases, VLDB, vol. 6 (2013)
Lu, M., Zhang, L., Huynh, H.P., Ong, Z., Liang, Y., He, B., Goh, R.S.M., Huynh, R.: Optimizing the MapReduce framework on Intel Xeon Phi coprocessor. In: Proceedings of IEEE International Conference on Big Data (2013)
QT. http://www.qt.io/download/ . Accessed Oct 2015