I had heard about BigData couple year ago. First time I heard about it, I was not quite sure what was the difference with database system. It looked very similar. Until last year, I started to take a look on BigData and went to the detail. Oh I found a lot of opportunity to improve the query data system and "data prediction".
Look back to 20 years ago, when I was a junior programmer. My job made software to create data summary report. I still remember, I needed to run my software more than a week to finish one summary report. That took time too long. I had talked to my older friend who is working on IT department for big company in Thailand. He explained to me, he has a lot of new technology to improve data query system to make more easier and faster for report (he mentioned on BI and quick view) but he still needs long time to finish the report, specially yearly report.
I am thinking, will it be better if he will create new BigData System (not replace with database) and copy the data transaction from database to Hadoop, then write the MapReduce software to generate the report. If you know how is Hadoop cluster system doing, you will understand how can be more faster. My friend mention to create new system is not easy because he needs to think about cost, time and maintain. I agreed on this but we need to trade off the benefit, how much money have to pay, how much time have gain. The Hadoop system does not require high performance. I tested with CubieTruck (ARM7 CPU), it works very well (base on not much big data :) )
One thing that I still do not understand, why the user (human) still be a person who generate the report. Why don't they have "a software bot" to auto collect data transaction, once time for create report was triggered. Just let the software bot does by itself, then send the final report to user automatic. This is a routine job that need to do everyday or every month or every year. The software bot should able to do it, no more user action. Do you think so?
A lot of more idea I can imagine what the BigData can do, like billing system, stock system. But I am interesting "data prediction" from BigData. Specially if you know machine learning, statistic or Six Sigma. You can predict data in the future from the history data. Look interest??
TuChay
