vectordotdev/vector

Allow specifying the object key in the `aws_s3` sink

Open

#6,929 opened on Mar 30, 2021

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Rust (2,126 forks)batch import
good first issuesink: aws_s3type: enhancement

Repository metrics

Stars
 (21,837 stars)
PR merge metrics
 (Avg merge 5d 12h) (85 merged PRs in 30d)

Description

Requested by user migrating from fluentd.

Currently, we expose some configuration knobs for the object prefix + key for the aws_s3 sink:

  filename_append_uuid = true # optional, default
  filename_extension = "log" # optional, default
  filename_time_format = "%s" # optional, default
  key_prefix = "date=%F/" # optional, default

However there is no way to directly set the key. Ideally this could be templated from the event (like key = "{{ host }}"). This would affect internal partitioning of events across S3 requests. It is likely we'll still want to support formatting the timestamp as part of that, as well.

Ref: fluentd's s3 output: https://docs.fluentd.org/output/s3#s-3-_object_key_format

Contributor guide

Research direction
The issue requests adding a configurable key template for AWS S3 sink. Start by examining the existing sink configuration in `src/sinks/aws s3/config.rs` and the key formatting logic in `src/sinks/aws s3/sink.rs`. Look at how other sinks implement templated keys, such as the `console` sink. The fluentd documentation linked provides a reference for the expected format. Note that the implementation should support templating from event fields and include timestamp formatting. Consider how this affects partitioning and ensure backward compatibility.
Tech stack
rustaws
Domain
backendclouddata
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Rust programmingAWS S3 basicsVector sink configuration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50