Thursday, November 24, 2011
How to make sense of Big data
The advance of database technology and hardware enables us to built larger and larger data warehouses with massive amounts of data - big data. The increasing demands from financial transactions, network connections, and other areas makes this a requirement. Yet, trying to make sense of this data is like trying to take a sip of water from a fire-hose. The quantity of data is just too large to review using tables. Even data-mining requires hypothesis formation to write the query. The only way to access this data in a useful way is through visualization. Initially, this has been through graphs and diagrams. However, these are relatively clumsy and require reference back to the tables and charts. Our knowledge of human perception has advanced, along with algorithms and the capability of the Graphical Processing Unit to the point that Synerscope presents big data in a way which the human eye and brain finds natural. This means that humans can draw meaning from visualizations that hold millions of pieces of data.