Historically, data in warehouses have been periodically loaded in batches:
- Typically daily
- Sometimes weekly
- Rarely monthly.
More and more data analytic repositories are incrementally loaded several times during the business work day or even at an hourly level. In modern data analytics platforms, data is updated in near real time, such as every five minutes, as well as streaming data that may be asynchronously replicated from the source system(s) to the target platform or database(s).
There is a term in the data engineering space called Data Latency which is the time delay between when data is generated and when it is available for use. Based on data latency and the defined SLA for data, when critical data does not arrive within the specified threshold, we have a Data Outage, With Real-time Streaming technologies like:
- Quest SharePlex
- Oracle GoldenGate
- QLIK Replicate
- FiveTran HVR
We can push data from the source system to the target systems and keep the data asynchronously updated.
