ArgiBigData Platform

Introduction

Big data have been presenting us with exciting opportunities and challenges in economic development. For instance, in the agriculture sector, mixing up of various agricultural data (e.g., weather data, soil data, etc.), and subsequently analyzing these data deliver valuable and helpful information to farmers and agribusinesses. However, massive data in agriculture are generated in every minute through multiple kinds of devices and services such as sensors and agricultural web markets. It leads to the challenges of big data problem including data collection, data storage, and data analysis. In this project, we design and implement a platform for collecting and analyzing agricultural big data. The proposed platform supports (1) multiple methods of collecting data from various data sources using Flume and MapReduce; (2) multiple choices of data storage including HDFS, HBase, and Hive; (3) big data analysis modules with Spark and Hadoop; and (4) visualization module. The figure below shows an overview of the architecture of our platform

Architecture of of ArgiBigData Platform

Publications

  1. Van-Quyet Nguyen, Giang-Truong Nguyen, Sinh-Ngoc Nguyen, Jintae Choi, Kyungbaek Kim, "Big Data Analytics and Visualization Techniques: A Case Study from Agriculture Domain", In Proceedings of KISM Fall Conference October 03-04, 2017, Mokpo University, Mokpo, South Korea. PDF
  2. Van-Quyet Nguyen, Kyungbaek Kim, "Performance Evaluation between Hive on MapReduce and Spark SQL with BigBench and PAT", In Proceedings of KISM Spring Conference April 28-29, 2017, Seoul National University, Seoul, South Korea. PDF
  3. Van-Quyet Nguyen, Sinh Ngoc Nguyen, Kyungbaek Kim, "Design of a Platform for Collecting and Analyzing Agricultural BigData", Journal of Digital Contents Society Vol.18 No.1 pp. 149-158, Feburary 28, 2017. PDF
  4. Van-Quyet Nguyen, Sinh Ngoc Nguyen, Duc Tiep Vu, Kyungbaek Kim, "Design and Implementation of Big Data Platform for Image Processing in Agriculture", In Proceedings of KIPS Fall Conference November 04-05, 2016, Pusan National University, Busan, South Korea. PDF
  5. Van-Quyet Nguyen, Sinh Ngoc Nguyen, Duc Tiep Vu, Kyungbaek Kim, "The Performance Comparison of Hadoop and Spark in Agriculture Big Data Processing", In Proceedings of KISM Fall Conference October 28-29, 2016, Honam University, Gwangju, South Korea. PDF
  6. Van-Quyet Nguyen, Sinh Ngoc Nguyen, Duc Tiep Vu, Kyungbaek Kim, "Design and Implementation of a Crawling Agriculture Product Prices System based on Hadoop and Flume", In Proceedings of KISM Fall Conference October 28-29, 2016, Honam University, Gwangju, South Korea. PDF
  7. Sinh Ngoc Nguyen, Van-Quyet Nguyen, Kyungbaek Kim, "Comparison the query time of searching data on HBase and HDFS", In Proceedings of KISM Fall Conference October 28-29, 2016, Honam University, Gwangju, South Korea. PDF
  8. Ngoc Nguyen-Sinh, Quyet Nguyen-Van, Kyungbaek Kim, "Design of Spark based Agricultural Big Data Analysis Platform", In Proceedings of KISM Spring Conference April 29-30, 2016, Silla University, Busan, South Korea. PDF

Documents

Name Description Download
AgriBigData platform deployment manual v1.5 This user's manual includes three chapters describing about our big data platform.
Chapter 1: Introducing AgriBigData platform
Chapter 2: Walking through a manual deployment
Chapter 3: Deploying applications to AgriBigData platform 		Link

Softwares

Name Description Download
realtime_weather_collector.jar A Java-based application that supports collecting weather data of a city in every hour from the web at URL https://www.wunderground.com Link
historical_weather_collector.jar A Java-based application that supports collecting weather data of a city over a period of date in the past from the web at URL https://www.wunderground.com Link
realtime_price_collector.jar A Java-based application that supports collecting product price data from market websites related to agriculture products. The data is updated in every 30 minutues. The data is crawled from Korean market websites such as http://seobu-market.gwangju.go.kr and http://www.eomgung-market.busan.kr. Link
historical_price_collector.jar A MapReduce-based application that supports collecting agriculture product price data of a year in the past from a region/market The data is crawled from Korean market websites such as http://seobu-market.gwangju.go.kr and http://www.eomgung-market.busan.kr. Link
nongjak_people_plants.jar A MapReduce-based program that finds out all plants which are grown by each person from Korea agricultural data (*) Link
nongjak_people_animals.jar A MapReduce-based program that finds out all animals who are raised by each person from Korea agricultural data (*) Link
nongjak_plants_area.jar A MapReduce-based program that computes total planted area for each type of plant from Korea agricultural data (*) Link
nongjak_people_area_income.jar A MapReduce-based program that computes total planted area and total income for each person from Korea agricultural data (*) Link
nongjak_plants_people.jar A MapReduce-based program that finds out all people who grew on each type of plant from Korea agricultural data (*) Link
Note: Korea agricultural data (*) is the private data, please contact us via email for getting that data.

Datasets

Name Description Download
Gwangju's weather data 2016 This data is stored in *.csv file, where each row contains information about temperature, humudity, event (e.g., rain, snow, etc) Link
Busan's weather data 2016 This data is stored in *.csv file, where each row contains information about temperature, humudity, event (e.g., rain, snow, etc) Link

People


Prof. Kyungbaek Kim
 Research Director

Contact

Prof. Kyungbaek Kim
Distributed Networks and Systems Laboratory,
Dept. Electronics and Computer Engineering,
Chonnam National University
Email: kyungbaekkim@jnu.ac.kr
Phone: +82-62-530-3438