As a light-weight, efficient replacement for fluentd, fluent-bit met almost all the requirements for lifting Kubernetes logs, transforming them to Splunk HEC format and forwarding them to a Splunk HEC instance.

This solution gives us,

  • Resource efficiency
  • No further processing before index storage required by Splunk
  • Self-service index routing and sourcetype specification via Pod labels

At this point, fluent-bit is missing two key pieces required to complete the transformation to HEC format. Instead of creating a HEC transformation filter, we decide to implement the functionality in a composable, re-usable way.

This results in two new filters. The MVP release had 2 operations that met our requirements. The feature-complete release is based on observations and community feedback. It has 11 operations and 10 conditions, all of which can be combined and repeated as needed.

MVP Requirements

  • Add ability to nest key/value pairs inside a new key
    • Use case : Move all keys prefixed with k8s: to a map named fields
  • Add ability to rename specific keys
    • Use case : Use pod labels k8s:labels:index and k8s:labels:sourcetype as top-level HEC key/value pairs

Feedback Requirements

  • Add ability to lift keys from a nest (based on prefix)
  • Optionally add or remove prefixes for both nest and lift operations
  • Add pre-conditions for applying modify operations (key_exists, etc)
  • Expand set of verbs for modify operations (set, add, remove, remove_wildcard, remove_regex, rename, hard_rename, copy, hard_copy)

Order of events

Resources and references