drivendataorg/cloudpathlib

Methods that move/remove files on the cloud don't remove the cached version

Open

#388 opened on Dec 20, 2023

 (3 comments) (0 reactions) (0 assignees)Python (87 forks)auto 404
bugcachinggood first issue

Repository metrics

Stars
 (624 stars)
PR merge metrics
 (PR metrics pending)

Description

The following code

import cloudpathlib
s3_path = cloudpathlib.S3Path(f"s3://my-bucket/foo.txt")
s3_path.open("x").close() # Create empty file
s3_path.unlink()
s3_path.open("x").close() # Create empty file

does not work as expected. I would expect it to create a file, remove it, and then create it again. Instead, I get

Traceback (most recent call last):
  File "/workspaces/scoutingtool/backend/../cpltest.py", line 9, in <module>
    s3_path.open("x").close() # Create empty file
    ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/cloudpathlib/cloudpath.py", line 503, in open
    buffer = self._local.open(
             ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 1044, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileExistsError: [Errno 17] File exists: '/tmp/tmpg5cu85sn/my-bucket/foo.txt'

This seems to happen because the file is kept in cache, even when it was removed from S3. A simple solution would probably to use the "w" mode, even when "x" was supplied.

Contributor guide