Project : Kubernetes log shipping to Splunk

• Design  
• Data Engineering  
• k8s  
• kubernetes  
• splunk  
• fluentbit  

Overview

Requirements

• Platform-wide deployment
• Self-service event to index routing
• Self-service sourcetype definition

Version 1.0.0 - Event shipping using S2S

Notable features,

• Client-side load-balancing with Splunk

https://github.com/michiel/docker-compose-splunk-fluentbit/tree/v1.0.0

Version 1.1.0 - Observability

Notable features,

• Add heartbeat function
• Read out fluentbit Prometheus metrics
• Generate Splunk Forwarder metrics with mtail

https://github.com/michiel/docker-compose-splunk-fluentbit/tree/v1.1.0

Version 1.2.0 - Shipping over HTTP

Notable features,

• Remove Splunk Forwarder from integration
• Client-side load-balancing with envoy proxy

https://github.com/michiel/docker-compose-splunk-fluentbit/tree/v1.2.0

Resources and references

• This implementation was originally presented at the Splunk Melbourne meetup 2018-04.
• The reference for proving out the event shipping is at https://github.com/michiel/docker-compose-splunk-fluentbit