drivendataorg/cloudpathlib

MinIO custom S3 endpoint does not support our `touch` implementation

Open

#141 opened on Apr 3, 2021

View on GitHubContributor guide
 (0 comments) (0 reactions) (0 assignees)Python (87 forks)auto 404
S3bughelp wanted

Repository metrics

Stars
 (624 stars)
PR merge metrics
 (PR metrics pending)

Description

Our touch implementation depends on the MetadataDirective="REPLACE": https://github.com/drivendataorg/cloudpathlib/blob/de6b54737014b06982e8c3af54d2da4238fcd8e3/cloudpathlib/s3/s3client.py#L158-L165

However, this fails our test for touch when the rig is custom_s3_rig running against our MinIO test server: https://github.com/drivendataorg/cloudpathlib/blob/de6b54737014b06982e8c3af54d2da4238fcd8e3/tests/test_cloudpath_file_io.py#L61-L66

I suspect that MinIO doesn't support this S3 functionality yet or it is a bug. We potentially could workaround by checking if the endpoint_url is set to a non-AWS URL and potentially do the fallback entire copy in that case.

This needs research to understand if this is a MinIO bug we could file to be fixed upstream or if we should workaround it.

When this is fixed, we should re-enable this test.

Contributor guide

Research direction
Investigate if MinIO supports the 'MetadataDirective=REPLACE' parameter in its copy object API. If not, implement a fallback that performs a full copy when a custom endpoint is detected, and then re enable the test.
Tech stack
python
Domain
backendcloudapi
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonS3testing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65