Data retention costs are a significant part of your analytics budget. Using Splunk Enterprise we help you implement a solution and architecture to reduce historical data storage costs by up to 80%!
We propose data archiving configuration using options that retain Splunk search functionality, with varying tradeoffs in terms of search speed.
These options are best suited for cold data that will not be searched very frequently and primarily be used for historical trend analysis.
One option is to keep historical data within Splunk and reduce the storage footprint of seldom-analyzed, cold data. This storage reduction is achieved by removing some search performance optimization data (called TSIDX) primarily useful for speeding up “needle in the haystack” type of search. For typical historical data use cases, impact on performance will be limited.
Another option is configuring the platform to roll your data to an existing Hadoop data lake, ESI Technologies Cloud Storage, Azure Storage or Amazon S3. Similar to the TSIDX reduction option, the storage reduction is achieved by removing some Splunk search performance optimization.