Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Implement awinsymlinks mode that prefers native symlinks, falling back to the deep copy mode#114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
dscho wants to merge2 commits intomsys2:msys2-3.5.7
base:msys2-3.5.7
Choose a base branch
Loading
fromdscho:native-symlinks-with-deepcopy-fallback

Conversation

@dscho
Copy link
Collaborator

@dschodscho commentedNov 7, 2022
edited
Loading

It surprises new MSYS2 users no end thatln -s does not create symbolic links at all, but deep copies (and with an exit code indicating success!). This did not at all match the expectations of those users who were familiar with Unix' concept of symbolic links and thought that they could rely on MSYS2 providing those, too, or fail with a non-zero exit code.

Historical reasons are at play here: When MSYS2 was started (or was that already the behavior of MSys? I forget...), symbolic links were not supported on Windows, at least not really: you had to have administrator privileges to create them (but not to delete them... 🤷) in Windows Vista, and before that, Windows simply had no idea about symbolic links.

So what about Cygwin? Well, Cygwin had something like support for symbolic links, using.lnk files for the emulation. The only problem? You had to stay within Cygwin's walled garden to make use of them. All non-Cygwin applications would react with a less or more unpleasant "huh?!?" when encountering those "symbolic links".

That's why MSYS2 chose to deep-copy by default. At least that way./configure would still work for those projects that required symbolic links. This was instrumental in getting MSYS2's package ecosystem off the ground.

Even when a Windows 10 update introduced support for creating symbolic links without elevation as long as Windows was run in Developer Mode, the created symbolic links are notcompletely what Unix/Linux/macOS users may be used to, as Windows discerns between directory symlinks and file symlinks.

Be that as it may, now that we've dropped Windows 7 and Windows 8 support, it may be a good time to start switching the default to creating actual symbolic links by default.

Since we still support Windows 8.1 (and a couple of Windows 10 versions that do not allow creating symbolic links in non-elevated operations, even in Developer Mode), we cannot simply switch to a mode where the MSYS2 runtime creates symbolic links when asked for, but we have to have a mode where the MSYS2 runtime first checks whether that is possible with the Windows version on which it is running, and if not, falls back to the deep-copy.

This PR does precisely that: implement that mode, but doesnot yet flip the default away fromdeepcopy. The reason is that I want this to be tested by volunteers (myself included) first, and once it is deemed robust and stable enough, flip the default tonativeordeepcopy.

This addresses#113.

1480c1, TechnoSparks, dimk90, and vanyasem reacted with thumbs up emojilazka and YarBurArt reacted with eyes emoji
@dschodscho self-assigned thisNov 7, 2022
@dschodscho changed the base branch frommsys2-3_3_6-release tomsys2-3.5.3July 24, 2024 12:58
@dschodschoforce-pushed thenative-symlinks-with-deepcopy-fallback branch 2 times, most recently fromcb31af7 tob8e842eCompareJuly 24, 2024 19:51
@dschodscho marked this pull request as ready for reviewJuly 24, 2024 20:28
@dschodscho requested a review fromlazkaJuly 24, 2024 20:29
@jcrben
Copy link

Well, Cygwin had something like support for symbolic links, using .lnk files for the emulation.

Cygwin not onlyhad, it still has symlinks like that - except the docs suggest that the*.lnk are not the default. The cygwin docs say this about their default symlinkshttps://cygwin.com/cygwin-ug-net/using.html#pathnames-symlinks

special reparse points shared with WSL (on NTFS on Windows 10 1607 or later)
plain files with the system attribute, containing a magic cookie followed by the path to which the link points.

The "shortcut" style links are something different.

The new Developer Mode feature is OK, but it's got a big warning on it when you enable it and enterprises therefore don't want to enable it:

Turning on developer mode, including installing and running apps from outside the Microsoft Store, could expose your device and personal data to security risks or harm your device.

https://stackoverflow.com/questions/74613686/what-does-the-warning-in-developer-mode-mean

I setexport MSYS=winsymlinks:sys as I believe this restores the cygwin default special reparse point symlinks, which is fine by me. I also foundhttps://carltonf.github.io/post/symbolic-links-on-windows sort of helpful in dumbing it down

@dscho
Copy link
CollaboratorAuthor

MSYS=winsymlinks:sys

This mode is incompatible with regular Win32 programs; They won't understand those symlinks and misinterpret them all the time. That is an okay stance to take for Cygwin, which wants you to stay within its ecosystem. It is not an okay stance for MSYS2 which wants to integrate with native Win32 programs as much as possible.

The new Developer Mode feature is OK, but it's got a big warning on it when you enable it and enterprises therefore don't want to enable it:

Turning on developer mode, including installing and running apps from outside the Microsoft Store, could expose your device and personal data to security risks or harm your device.

Yes, developer mode comes with a lot of liberties that developing code requires. Nevertheless, it isthe mode in which Windows users can create symbolic links in modern Windows without requiring an elevated process (which would increase the security risk a lot more than Developer Mode).

In any case, the mode I introduce here, and which I propose to promote to eventually be the default, retains backwards-compatibility in a fashion, by falling back to MSYS2's current behavior if symbolic links cannot be created.

mati865 and TechnoSparks reacted with thumbs up emoji

@dschodscho mentioned this pull requestJan 27, 2025
@jeremyd2019
Copy link
Member

I am thinking, would it make sense to try to generalize this? maybe something likewinsymlinks:native:deepcopy or something? There's a sort of ambivalence in me: I don't want to just add another symlink type, but I also don't want to increase the combinatorial complexity of symlinks.

@lazka
Copy link
Member

I am thinking, would it make sense to try to generalize this?

Can you think of any use case where a different config/order would be preferred by users?

@jeremyd2019
Copy link
Member

Not necessarily, it's just now we have a lot of seemingly-poorly-defined fallback cases:

  • native (or nativestrict if target isspecial) falls back todefault
  • default is actually wsl with a fallback to sysfile
  • deepcopy falls back to sysfile if target isspecial

I guess now that I write it out, that's not too bad, everything that falls back eventually falls back to sysfile. Perhaps deepcopy should fall back todefault also now for consistency (the wsl stuff obviously didn't exist when deepcopy was written). Moving deepcopy up into the block where native/strict happens would make it more obvious that it's a fallback case too.

__leave;
}
/* With deepcopy fall back? Let's do that, then*/
if (res == -1 && wsym_type == WSYM_native_or_deepcopy)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Note that if targetisspecial res will be-2, so will still fall back to default. This is probably reasonable, since deepcopy would just fall back to sysfile in that case. default will let it try wsl before falling back to sysfile. I'm thinking it might be best to make deepcopy fall back to default instead of sysfile anyway.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

making deepcopy fall back to wsl instead of sysfile breaks one git test which tries to commit a symlink to /dev/null. The result of doing so is almost certainly not helpful (I bet it commits the sysfile, not a symlink), but when it's a wsl symlink it fails withFunction not implemented

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

when it's a wsl symlink it fails withFunction not implemented

That's my fault, as I have not managed to spend time on upstreaming Git for Windows' symlink support:git/git@master...dscho:git:support-symlinks-on-windows

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I had a quick look at those changes, and I didn't see any handling of wsl symlinks in there, so I don't think that would help.

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Huh. I misremembered. Agit grep IO_REPARSE_TAG_LX_SYMLINK indeed comes up empty...

jeremyd2019 added a commit to jeremyd2019/msys2-runtime that referenced this pull requestFeb 1, 2025
Factor out deepcopy symlink to its own worker function, like wsl,native, and nfs.  Move it up into the beginning switch with them, so thefallback behavior is more obvious.  See alsomsys2#113/msys2#114.
jeremyd2019 added a commit to jeremyd2019/msys2-runtime that referenced this pull requestFeb 1, 2025
Factor out deepcopy symlink to its own worker function, like wsl,native, and nfs.  Move it up into the beginning switch with them, so thefallback behavior is more obvious.  See alsomsys2#113/msys2#114.
jeremyd2019 added a commit to jeremyd2019/msys2-runtime that referenced this pull requestFeb 1, 2025
Factor out deepcopy symlink to its own worker function, like wsl,native, and nfs.  Move it up into the beginning switch with them, so thefallback behavior is more obvious.  See alsomsys2#113/msys2#114.
jeremyd2019 added a commit to jeremyd2019/msys2-runtime that referenced this pull requestFeb 1, 2025
Factor out deepcopy symlink to its own worker function, like wsl,native, and nfs.  Move it up into the beginning switch with them, so thefallback behavior is more obvious.  See alsomsys2#113/msys2#114.
jeremyd2019 added a commit to jeremyd2019/msys2-runtime that referenced this pull requestFeb 1, 2025
Factor out deepcopy symlink to its own worker function, like wsl,native, and nfs.  Move it up into the beginning switch with them, so thefallback behavior is more obvious.  See alsomsys2#113/msys2#114.
jeremyd2019 added a commit to jeremyd2019/msys2-runtime that referenced this pull requestFeb 1, 2025
Factor out deepcopy symlink to its own worker function, like wsl,native, and nfs.  Move it up into the beginning switch with them, so thefallback behavior is more obvious.  See alsomsys2#113/msys2#114.
@dscho
Copy link
CollaboratorAuthor

Perhaps deepcopy should fall back todefault also now for consistency (the wsl stuff obviously didn't exist when deepcopy was written).

My rationale was: deepcopy exists only as a Win32-compatible fall-back if symlink support does not exist, and symlink support predates WSL (and both require Developer Mode to be enabled IIRC).

But then, this entire PR is about movingoff of deepcopy in the first place, preparing for a time when MSYS2 drops support for Windows versions where symlinks can only be created when running elevated.

FWIW@jeremyd2019 I really like the refactoring you did, moving the deepcopy code into its own function, I'd just suggest to do it as a preparatory, separate commit instead of squashing it in.

@jcrben
Copy link

symlink support predates WSL (and both require Developer Mode to be enabled IIRC)

Not to derail into something off-topic - not sure if you're saying WSL2 requires Developer Mode - it doesn't. Probably easier to get WSL2 to a level that meets enterprise security than Developer Mode, altho that's no simple thing either.

There's been a trend for tools to just use junctions instead since they don't require admin privileges - you can see a bit of context atkubernetes-sigs/krew#843 - obv junctions just link a folder, but in many cases that can suffice

@dscho
Copy link
CollaboratorAuthor

There's been a trend for tools to just use junctions instead since they don't require admin privileges

@jcrben It had been a trend, but thankfully it's no longer all that common. See e.g.this insightful comment for an explanation. Here's an excerpt:

FeatureUnixWindows symlinkWindows junction
Target is relative to source directoryyesyesno
Target is relative to current computeryes [1]yesno
Target is absolutenoyes [2]yes [3]
Link is resolved byclientclientserver

[1] Symlinks to absolute paths are resolved by the client and thus relative to the client computer. E.g. if/usr is mounted to some central network drive, the symlink/usr/local/etc -> /etc/local refers to/etc/local on the client.
[2] Via UNC path, e.g.\\server\share\path.
[3] Via local path, e.g.C:\path or\\?\Volume{...}\path.

tl;dr Windows junctions are nowhere close to being a valid symlink replacement.

@jcrben
Copy link

jcrben commentedFeb 4, 2025
edited
Loading

@dscho well, in some ways using junctions is becoming more common - fnm switched to junctions in 2021, volta just did in 2024, jbangdev did it in 2024 at my urging, kubectl plugin manager krew may also do it. avoiding a complex and scary step of enabling Developer Mode is pretty nice. wish we could get symlinks on Windows without enabling Developer Mode. See also:Why do you have to be an admin to create a symlink in Windows?

@sskras
Copy link

@jcrben writes:

wish we could get symlinks on Windows without enabling Developer Mode. See also:Why do you have to be an admin to create a symlink in Windows?

Exactly that. You don't need the Developer Mode or become elevated every time you create a symlink. AssigningSeCreateSymbolicLinkPrivilege to your user once is enough. That's what I did in the past on my w10, and I am running fine since.

@dscho
Copy link
CollaboratorAuthor

in some ways using junctions is becoming more common

@jcrben Yes, and that's true even when mentioning better-known projects (I never heard about your examples, but IIRCpnpm is an example where junctions are used).

The important thing to keep in mind? The usage resembles Unix mount points, much more than symbolic links. I do encourage you to studythe insightful comment about this that I linked earlier.

AssigningSeCreateSymbolicLinkPrivilege to your user once is enough. That's what I did in the past on my w10, and I am running fine since.

@sskras That's enticing, but leaves the reader hanging for a clear description how to do that. In other words, your comment is too terse, saves too much on time at the cost of being less helpful than it could be. Please do fill in the blanks.

@jeremyd2019
Copy link
Member

This adds the missing documentation (in case we ever publish theinformation about the MSYS variable on msys2.org).Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…llbackWhen native symlinks are available, it is a shame to create deep copiesby default.However, since there are many scenarios where symlinks are not available(e.g. when running on FAT, or on older Windows versions, or whenDeveloper Mode is not enabled), we've got to have a fallback.In the regular Cygwin world, it is legitimate to fall back to WSLsymlinks and/or to the system file emulation (where a file is createdthat is marked with the "system" attribute and with content that adheresto a specific, magic form that is recognized specifically by the Cygwinruntime).However, in the world of MSYS2, the assumption is that the result of theoperation should be as interoperable with regular Win32 programs aspossible. Hence the default to "deepcopy".As a "best of both worlds" mode, let's implement one that tries tocreate native symlinks by default, and if that fails, uses the"deepcopy" method as a fallback.This addressesmsys2#113.Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@jeremyd2019jeremyd2019force-pushed thenative-symlinks-with-deepcopy-fallback branch fromb8e842e tofeecd91CompareFebruary 6, 2025 18:22
@jeremyd2019jeremyd2019 changed the base branch frommsys2-3.5.3 tomsys2-3.5.7February 6, 2025 18:22
@sskras
Copy link

sskras commentedFeb 7, 2025
edited
Loading

@dscho commented3 days ago

AssigningSeCreateSymbolicLinkPrivilege to your user once is enough. That's what I did in the past on my w10, and I am running fine since.

@sskras That's enticing, but leaves the reader hanging for a clear description how to do that. [...] Please do fill in the blanks.

OK. There are different ways/tools to configure this:

1, usingsecpol.msc GUI.
2, usingsecedit.exe CLI.
3, usingntrights.exe old CLI (it was present on the old Windows Server Resource Kit only, and it harder to find).
4, using 3rd party tools, eg:

I didn't think earlier a Windows developer would need guidance on that, but now visualized usage of the 1st tool:

image

  1. Open "Run" dialog, typesecpol.msc.
  2. Press OK ("Local Security Policy" opens).
  3. Navigate to "Local Policies", expand it.
  4. Navigate to "User Rights Assignment".
  5. Double-click "Create symbolic links" ("Create symbolic links Properties" dialog opens).
  6. Click "Add User or Group..." ("Select Users or Groups" dialog opens).
  7. Type in the needed username.
  8. Press OK twice.
  9. Check if the username already occurs on the line from step 4.
  10. Maybe rungpupdate after that (I already forgot / didn't test that).

@jeremyd2019 commented3 days ago

https://packages.msys2.org/packages/mingw-w64-x86_64-editrightshttps://cygwin.com/cgit/cygwin-apps/editrights/tree/editrights.README

editrights -u YourUser -a SeCreateSymbolicLinkPrivilege

? (untested)

Thanks, I didn't know about this tool! Might be a nice addition to the list above.

@dscho
Copy link
CollaboratorAuthor

Honestly, I was thinking more about an automatable solution that requires nothing but PowerShell, something like:

# Export the current security settings to a file$SecEditExportFile="$env:TEMP\secedit.inf"secedit/export/cfg$SecEditExportFile/areas USER_RIGHTS# Modify the security settings file to include the privilege for the user$User="DOMAIN\User"$Privilege="SeCreateSymbolicLinkPrivilege"# Read the exported settings file$lines=Get-Content$SecEditExportFile# Add the privilege for the user$index=$lines.IndexOf("[Privilege Rights]")$privilegeLine=$lines[$index..($lines.Length-1)]|Where-Object {$_-match"^$Privilege" }if ($privilegeLine) {$existingValues=$privilegeLine-replace"$Privilege =",""$newValues="$existingValues,$User"$lines[$lines.IndexOf($privilegeLine)]="$Privilege =$newValues"}else {$lines+="$Privilege =$User"}# Write the modified settings back to the file$lines|Set-Content$SecEditExportFile# Import the modified security settings:secedit/configure/db secedit.sdb/cfg$SecEditExportFile/areas USER_RIGHTS# Refresh the policy:gpupdate/force

Since this here is a public bug tracker, one must assume that people might find this and would be delighted by guidance how to achieve the goal they set out to accomplish when trying to find more information.

I didn't think earlier a Windows developer would need guidance on that, but now visualized usage of the 1st tool:

Stating that you didn't think earlier a Windows developer would need guidance on that may very well come over as pretty smug and arrogant, and is therefore potentially something you'd like to avoid.

@sskras
Copy link

sskras commentedFeb 7, 2025
edited
Loading

@jeremyd2019,editrights that you found seems to work just fine:

Thank you.

@sskras
Copy link

sskras commentedFeb 7, 2025
edited
Loading

@dscho wrote:

Honestly, I was thinking more about an automatable solution that requires nothing but PowerShell, something like:

  ...secedit/export/cfg$SecEditExportFile/areas USER_RIGHTS  ...secedit/configure/db secedit.sdb/cfg$SecEditExportFile/areas USER_RIGHTS  ...

It still relies on runningsecedit a couple of times internally. Could be done quickly using sh + AWK too, eg:

image

Easily extendable to add the newSeCreateSymbolicLinkPrivilege = ... line in case it was missing, but unsure if that's needed.

IIUC this would mean that the current install of the supported Windows versions (w10, w11) is misconfigured or pretty broken, eg. having corrupt Security Database. But I am not against that.

And... how is it better than usingeditrights ?

Since this here is a public bug tracker, one must assume that people might find this and would be delighted by guidance how to achieve the goal they set out to accomplish when trying to find more information.

I see your point. For starters, as an ordinary user of MSYS2 I hope to find such info on the installation page, eg:https://www.msys2.org/#installation

Crossing fingers so the ways get discussed properly and then get better publicity. Cheers :)

I didn't think earlier a Windows developer would need guidance on that, but now visualized usage of the 1st tool:

Stating that you didn't think earlier a Windows developer would need guidance on that may very well come over as pretty smug and arrogant, and is therefore potentially something you'd like to avoid.

That's sad. What drove me is not the pride, but the ambiguity of the ways to solve this + my aim at keeping the Signal-to-Noise ratio high. I hope I proved that by illustrating two of these ways.

@culyun
Copy link

Came here looking for answers, read the thread, got good info.

The conversation is fine.
Not smug, just direct.

Probably the main concern is making this work out of the box in the corporate environment.
Coorporate IT tends to be very defensive about exceptions that bubble up in GUI settings like "Developer Mode".

Automatically enabling NTFS symlinks with an opt-out (like Git-Bash) should be the chief goal here.
This is merely codifying the principles of no harm and implicit permission granted through SeCreateSymbolicLinkPrivilege.

dscho added a commit to dscho/git that referenced this pull requestNov 25, 2025
Git for Windows has supported symbolic links for quite some time:Ingit-for-windows#156, this support wasintroduced already into Git for Windows v2.4.2.windows.1 in May 2015.However, the Git for Windows CI never ran the test suite with symboliclink support because the MSYS2 runtime (i.e. the POSIX emulation layerrequired to run Git's test suite because the latter is written in Unixshell script) does not support symbolic links right out of the box. Thisis for historical reasons: Symbolic link support was introduced inWindows 7, but these links could only be created by administrators bydefault, and it took until Windows 10 Build 14972 that at least inDeveloper Mode, non-administrators would be permitted to create them.The MSYS2 runtime _does_ have some sort of support for symbolic links,although with caveats: seeing as it expects the inputs as Unix-likepaths, but the outputs need to be Win32 symbolic links pointing to Win32paths, some normalization has to be performed in the process. This leadsto sometimes surprising behavior e.g. when a link target like `a/b/..`is normalized to `a`.It has been a minute or three since the time when Windows versionswithout symbolic link support were common, therefore there are plans toturn on that support in the MSYS2 runtime on these Windows versions bydefault, seemsys2/msys2-runtime#114 for moredetails about this.To prepare for this, I am working toward upstreaming Git for Windows'own support for symbolic links. And to prepare for that, in turn, I amhereby contributing preemptively the fixes required to eventually letGit's test suite pass when both MSYS2 runtime and Git support symboliclinks.Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@jeremyd2019jeremyd2019jeremyd2019 left review comments

@lazkalazkaAwaiting requested review from lazka

Assignees

@dschodscho

Labels

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@dscho@jcrben@jeremyd2019@lazka@sskras@culyun

[8]ページ先頭

©2009-2025 Movatter.jp