apache/airflow

Inconsistency in S3 Transfer Operators

Closed

#37,213 opened on Feb 7, 2024

View on GitHubContributor guide
 (7 comments) (0 reactions) (1 assignee)Python (16,781 forks)batch import
area:providersgood first issuekind:featureprovider:amazon

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (Avg merge 7d 18h) (834 merged PRs in 30d)

Description

Description

Standardize similar AWS transfer operators to have similar functionality, as well as similar naming conventions.

This would start with updating transfer operators using S3 as a source or destination to have similar behavior and naming.

Use case/motivation

There seems to be substantial discrepancies between transfer operators, especially those using S3 as a source or destination. For example, in the S3ToSFTPOperator, only a single file can be transferred from the SFTP to the S3 bucket. However, in the FTPToS3Operator an arbitrary number of files can be transferred from S3 to an FTP site. To add to this, the S3ToFTPOperator only supports a single file being transferred.

These sort of discrepancies exist for a number of transfer operators. In addition to that, parameter names are quite different from operator to operator, which makes using them quite difficult.

This will allow for a more seamless experience for Airflow users, and ensure common functionality across similar tools.

Related issues

No related issues.

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Research direction
Review the existing S3 transfer operators: S3ToSFTPOperator, FTPToS3Operator, S3ToFTPOperator. Identify inconsistencies in functionality (e.g., single vs. multiple file transfers) and parameter naming. Check comments and maintainer input for guidance on desired unified behavior. Propose and implement changes to standardize these operators, ensuring backward compatibility where possible.
Tech stack
python
Domain
databackend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
PythonApache AirflowAWS S3
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35