Data management. 1) Cleaning the data as it comes in rather than in batches so we can use it sooner, invalid data is discarded, outlier detection, normalizing inputs etc.... 2) Warehousing of the data with proper indexes so you can perform some advanced queries on unstructured data 3) Some data is sent in bulk at the end of day, some of the data is streamed in fire hose style. How can we preprocess the fire hose data…
Check out Striim http://www.striim.com/ It's a native streaming platform so your data will be cleansed, processed, scanned for outliers event-by-event rather than in batches. We have dozens of streaming connectors IT/Enterprise/Web data sources. We also support initial load for your firehose data. For unstructured data, we have support for RegEx based parsers. Shoot me a message if you have any more questions. We hav…
www.watchful.io