@giampaolo I saw that you are marked as a code owner for shutil, but GitHub did not request your review for some reason.

Would you like to take a look?

@illia-v to my understandingcopy_file_range() on Linux enables the so called "server side copy", not "reflink" / "copy on write". I posted a patch some time ago, but I should update it, see:
#81340

As for reflink / CoW copy, I also submitted a patch some time ago, but I wanted to add Windows support before merging it.
#81338

To my understanding reflink / CoW copy on Linux is achieved by using FICLONE, not copy_file_range().

@giampaolocopy_file_range does try cloning. Take a look athttps://stackoverflow.com/a/65518879 andhttps://github.com/torvalds/linux/blob/1e57930e9f4083ad5854ab6eadffe790a8167fb4/fs/read_write.c#L1491-L1507.

cc@pablogsal

@giampaolo copy_file_range does try cloning. Take a look athttps://stackoverflow.com/a/65518879 andhttps://github.com/torvalds/linux/blob/1e57930e9f4083ad5854ab6eadffe790a8167fb4/fs/read_write.c#L1491-L1507.

Would you mind to create a first PR to mention it in copy_file_range() documentation?https://docs.python.org/dev/library/os.html#os.copy_file_range

copyfile(src, dst, *, follow_symlinks=True, allow_reflink=True)

It terms of API, I would prefer to change the parameter name to justreflink=True. IMO it's clear enough thatreflink=True will not use CoW if it's not supported or cannot be used. Python has a long habit of trying to use the most efficient method, but falls back on the most compatible code if it doesn't work.

For example, socket.sendfile() tries to use sendfile() but falls back on send() if sendfile() doesn't work or is not available.

The Unix cp command has 3 modes for--reflink=WHEN:

• always: "perform a lightweight copy (...) if this is not possible the copy fails"
• never (False in your PR)
• auto (True in your PR)

Here you only propose 2 modes. Do we need to implement the 3rd "always" mode?

I see a lot of confusion between which function/syscall is used, server-side copy, and copy of write. There are expectections, hopes, and ... disillusions :-) IMO the doc should be completed to better explain differences between server-side copy and copy-on-write, and attempt to document when and how one method is attempted, when it can fail, etc.

I am glad the issue has received this attention.

@vstinner thanks for valid points, especially one about the three modes. I think it will be nice to support all of them in Python too.

Butcopy_file_range has no way to specify that reflink is required.

• cp --reflink=always command seems to useioctl(dest_fd, FICLONE, src_fd) (1 &2) as the only copying method or fail if the call is not possible;
• cp --reflink=auto triesioctl(dest_fd, FICLONE, src_fd) firstly, thencopy_file_range, then others;
• cp --reflink=never skips bothioctl(dest_fd, FICLONE, src_fd) andcopy_file_range.

I will create a PR to extendos.copy_file_range documentation. Then I will need to experiment a bit with FICLONE.

Do you think it is possible to launch the three-mode reflink functionality for Linux firstly and add support for other OSes in later pull requests?
I am most interested in supporting this on Linux, but I may consider writing some code for other platforms later.

Do you think it is possible to launch the three-mode reflink functionality for Linux firstly and add support for other OSes in later pull requests?

If possible, the API should be the same and should be usable on all platform. For example, "as an user, I would like to call shutil.copyfile() with the same arguments on all platforms and get a similar behavior on all platforms". For example, the API should attempt to use server-side copy and/or use Copy-on-Write, but silently switch to regular read+write copy if no modern copy function is available on the OS or on the source/destination filesystems.

For the os module, it's fine to have different functions depending on the OS. But the shutil module is more a high-level module with a (mostly) portable API.

@giampaolo copy_file_range does try cloning. Take a look athttps://stackoverflow.com/a/65518879 andhttps://github.com/torvalds/linux/blob/1e57930e9f4083ad5854ab6eadffe790a8167fb4/fs/read_write.c#L1491-L1507.

Would you mind to create a first PR to mention it in copy_file_range() documentation?https://docs.python.org/dev/library/os.html#os.copy_file_range

Done in#93182, please review it.

I am swaying away from the idea in favor of a separatereflink function#81338 (comment).

I am swaying away from the idea in favor of a separatereflink function#81338 (comment).

The problem with that is you don't get the automatic free win, which is contrary to what others are doing. e.g. GNOME's glib (not glibc), KDE's kio, and even GCC's libstdc++ upgrade automatically. GCC 14 willautomatically usecopy_file_range if available forfilesystem::copy_file.

There doesn't appear to be much precedent for carving this out into its own opt-in functionality?

EDIT: It's been pointed out to me thatEmacs, of all things, unconditionally usescopy_file_range if the system supports it forcopy-file.

@illia-v are you still interested in working on this? I wonder if we even need theallow_reflinks argument - perhaps we could enable use ofcopy_file_range() (where available) without adding any new arguments?

@barneygale yes, I am interested and can revise the changes this week

@barneygale I droppedallow_reflinks, please take a look

As I've previously stated in#81338 (comment), personally I consider copy and CoW / clone 2 distinct operations. E.g. one may want to occupy disk spaceright now instead of later. For this reason I thinkshutil.copyfile should give the possibility to disable this behavior.cp command defaults to--reflink=auto, but it allows to disable CoW with--reflink=never. The problem withcopy_file_range() is that it doesn't allow you to opt-out from CoW, which is why back in 2020 I was more keen on providing a separateshutil.reflink function using FICLONE. But then again,copy_file_range() does provide fast server-side copy in some circumstances, which is something youalways want. What a mess...

It appears to me that the tool which solved all these controversies iscp via--reflink=auto/always/never. I have the feeling thatshutil.copyfile (and possibly also pathlib.Path.copy /#73991) should try to follow its lead and replicate that. E.g. given the signature:

def copyfile(..., reflink=None)

We may do:

• reflink=None ==cp --reflink=auto == usecopy_file_range() - on failure usesendfile() orread/write
• reflink=True ==cp --reflink=always == useFICLONE and fail on error
• reflink=False ==cp --reflink=never == usesendfile() orread/write

But if we do this, I think it would make sense to exposeshutil.reflink() first, which would also support macOS and Windows.

My 2 cents and sorry for not replying earlier on this topic.

There are some compelling arguments and examples from other languages in#81338. Personally I'm persuaded that we should enable it (without an opt-out), rather than second-guessing what the OS provides and what other language runtimes are doing. We have plenty of time til 3.14 ships if we need to back it out. But this is by no means my area of expertise, so I'm happy to be overruled.

Personally I'm persuaded that we should enable it (without an opt-out), rather than second-guessing what the OS provides and what other language runtimes are doing.

This seems alright to me, too. We're expecting/hoping Windows will start doing transparent CoW when doing file copies, so that ought to give an idea pretty quickly if there's a genuineneed to avoid it, or if we'd be adding the opt-out without justification.

@barneygale based on your experience of having#119058 present in main for more than a month, can we merge this too?

There have been no releases frommain in that time, and the first alpha of 3.14 isn't due until October, so I wouldn't read too much into the lack ofpathlib.Path.copy() bug reports.

@barneygale thank you for the response. Can we risk merging this PR before the alpha to be able to discover issues (if any) in early stages of 3.14?

Sorry I'm not familiar with the details of CPython development. I see this PR has theawaiting merge, is it waiting on anything else before said merge?

The merge conflicts need addressing, and it sounds like we want someone knowledgeable about the Linux API to chime in on whether this is using it properly.

But I agree, we ought to get it in asap, now that 3.14 is shipping.

The merge conflicts need addressing, and it sounds like we want someone knowledgeable about the Linux API to chime in on whether this is using it properly.

I resolved the conflicts, thanks
The PR has an approval from@barneygale who integratedcopy_file_range inpathlib#119058

Apologies for being slow. Despite the lack of an explicit endorsement from someone (else) claiming to understand the API well, I think the best option here is to have it in the 3.14 alphas.

I likely won't get tagged on any issues that are reported, but anyone who is should feel free to modify or revert the code if needed to fix users. I'm merging as a point of process, not because this is "my" code - I don't need to be consulted on changes.