pingcap/tidb

br: open format file generation by backup

Open

#58,611 opened on Dec 30, 2024

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (6,186 forks)batch import
help wantedtype/feature-request

Repository metrics

Stars
 (40,090 stars)
PR merge metrics
 (Avg merge 14d 4h) (369 merged PRs in 30d)

Description

Feature Request

Is your feature request related to a problem? Please describe:

Currently, there is no native tidb tool to export tikv data into open format files, like parquet. Instead, tidb users need to use client like tispark etc to extract data and do the format conversion. And the long tech stack suffers from bad performance.

Describe the feature you'd like:

tidb can provide a native way to dump snapshot data and incremental data to open format files. A preferred way is to let backup to generate open format files directly, in other words, backup can support to generate either log/sst files or parquet files. A simple prototype code is here https://github.com/BornChanger/sampleParquet.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Contributor guide

Research direction
Explore the existing backup codebase and the provided prototype to implement open format file generation.
Tech stack
go
Domain
backenddatabase
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GoTiDB backup tools
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
55