Overview

As a light-weight, efficient replacement for fluentd, fluent-bit met almost all the requirements for lifting Kubernetes logs, transforming them to Splunk HEC format and forwarding them to a Splunk HEC instance.

This solution gives us,

• Resource efficiency
• No further processing before index storage required by Splunk
• Self-service index routing and sourcetype specification via Pod labels

At this point, fluent-bit is missing two key pieces required to complete the transformation to HEC format. Instead of creating a HEC transformation filter, we decide to implement the functionality in a composable, re-usable way.

This results in two new filters. The MVP release had 2 operations that met our requirements. The feature-complete release is based on observations and community feedback. It has 11 operations and 10 conditions, all of which can be combined and repeated as needed.

MVP Requirements

• Add ability to nest key/value pairs inside a new key
  • Use case : Move all keys prefixed with k8s: to a map named fields
• Add ability to rename specific keys
  • Use case : Use pod labels k8s:labels:index and k8s:labels:sourcetype as top-level HEC key/value pairs

Feedback Requirements

• Add ability to lift keys from a nest (based on prefix)
• Optionally add or remove prefixes for both nest and lift operations
• Add pre-conditions for applying modify operations (key_exists, etc)
• Expand set of verbs for modify operations (set, add, remove, remove_wildcard, remove_regex, rename, hard_rename, copy, hard_copy)

Order of events

• Implement new filters (modify and nest) with minimum functionality
• Setup reference implementation using docker-compose
• Transform and ship events to Splunk HEC using new fluent-bit build
• Upstream initial work via PRs (#531, #532) (release 0.13.2)
• Add documentation with examples via PRs
  • Documentation 0.13 modify
  • Documentation 0.13 nest
• Add public docker build for evaluation (travis-ci to docker-hub)
• Gather community requirements for feature-complete filters
• Update documentation with examples via PRs
  • Documentation 0.14 modify
  • Documentation 0.14 nest
• Upstream feature-complete filters via PRs (#660, #656, #654, #618) (release 0.14.0)

Resources and references

• The reference for proving out the event shipping is at https://github.com/michiel/docker-compose-splunk-fluentbit/tree/v1.0.0