Growth in data is attributed to vast improvements in internet technologies. Internet first fueled the Telecom industry which laid the groundworks and fueled other companies in:
- Social Media
- Streaming Services
- IoT
Our traditional enterprise data warehouse (EDW) could not address the rapid growth of data in these verticals. The traditional EDW was designed for relational data and cannot handle unstructured or semi-structured data. The EDW ecosystem also could not scale at both compute and storage. The storage could not be decoupled with the compute layer. The Big Data era brought forth lots of innovative technologies including the the ability to separate compute from storage but also how we consumed unstructured and semi-structured data. Big data also brought forth change in the definition of useable data.
What is the next storage metric after Yottabyte ? Brontobyte. One brontobyte is equal to one quadrillion terabytes. A brontobyte is smaller than a geopbyte; and a thousand brontobytes equasl to one geopbyte .
Historically, we had to be picky with data that we selected to move into the EDW database for analytics. Now data in whatever shape or form can be kept at fraction of the cost. In 2010, the average cost of storing 1 GB of data on a traditional spinning hard drive disk (HDD) was $10 cents. This number has come down to .01 (penny) per gigabyte. Now we live in the paradigm shift where we store first with the intention to analyze later.