- Notifications
You must be signed in to change notification settings - Fork48
Implement awinsymlinks mode that prefers native symlinks, falling back to the deep copy mode#114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:msys2-3.5.7
Are you sure you want to change the base?
Uh oh!
There was an error while loading.Please reload this page.
Conversation
cb31af7 tob8e842eComparejcrben commentedAug 25, 2024
Cygwin not onlyhad, it still has symlinks like that - except the docs suggest that the
The "shortcut" style links are something different. The new Developer Mode feature is OK, but it's got a big warning on it when you enable it and enterprises therefore don't want to enable it:
https://stackoverflow.com/questions/74613686/what-does-the-warning-in-developer-mode-mean I set |
dscho commentedAug 26, 2024
This mode is incompatible with regular Win32 programs; They won't understand those symlinks and misinterpret them all the time. That is an okay stance to take for Cygwin, which wants you to stay within its ecosystem. It is not an okay stance for MSYS2 which wants to integrate with native Win32 programs as much as possible.
Yes, developer mode comes with a lot of liberties that developing code requires. Nevertheless, it isthe mode in which Windows users can create symbolic links in modern Windows without requiring an elevated process (which would increase the security risk a lot more than Developer Mode). In any case, the mode I introduce here, and which I propose to promote to eventually be the default, retains backwards-compatibility in a fashion, by falling back to MSYS2's current behavior if symbolic links cannot be created. |
jeremyd2019 commentedJan 31, 2025
I am thinking, would it make sense to try to generalize this? maybe something like |
lazka commentedJan 31, 2025
Can you think of any use case where a different config/order would be preferred by users? |
jeremyd2019 commentedJan 31, 2025
Not necessarily, it's just now we have a lot of seemingly-poorly-defined fallback cases:
I guess now that I write it out, that's not too bad, everything that falls back eventually falls back to sysfile. Perhaps deepcopy should fall back to |
| __leave; | ||
| } | ||
| /* With deepcopy fall back? Let's do that, then*/ | ||
| if (res == -1 && wsym_type == WSYM_native_or_deepcopy) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Note that if targetisspecial res will be-2, so will still fall back to default. This is probably reasonable, since deepcopy would just fall back to sysfile in that case. default will let it try wsl before falling back to sysfile. I'm thinking it might be best to make deepcopy fall back to default instead of sysfile anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
making deepcopy fall back to wsl instead of sysfile breaks one git test which tries to commit a symlink to /dev/null. The result of doing so is almost certainly not helpful (I bet it commits the sysfile, not a symlink), but when it's a wsl symlink it fails withFunction not implemented
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
when it's a wsl symlink it fails with
Function not implemented
That's my fault, as I have not managed to spend time on upstreaming Git for Windows' symlink support:git/git@master...dscho:git:support-symlinks-on-windows
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I had a quick look at those changes, and I didn't see any handling of wsl symlinks in there, so I don't think that would help.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Huh. I misremembered. Agit grep IO_REPARSE_TAG_LX_SYMLINK indeed comes up empty...
dscho commentedFeb 2, 2025
My rationale was: deepcopy exists only as a Win32-compatible fall-back if symlink support does not exist, and symlink support predates WSL (and both require Developer Mode to be enabled IIRC). But then, this entire PR is about movingoff of deepcopy in the first place, preparing for a time when MSYS2 drops support for Windows versions where symlinks can only be created when running elevated. FWIW@jeremyd2019 I really like the refactoring you did, moving the deepcopy code into its own function, I'd just suggest to do it as a preparatory, separate commit instead of squashing it in. |
jcrben commentedFeb 3, 2025
Not to derail into something off-topic - not sure if you're saying WSL2 requires Developer Mode - it doesn't. Probably easier to get WSL2 to a level that meets enterprise security than Developer Mode, altho that's no simple thing either. There's been a trend for tools to just use junctions instead since they don't require admin privileges - you can see a bit of context atkubernetes-sigs/krew#843 - obv junctions just link a folder, but in many cases that can suffice |
dscho commentedFeb 3, 2025
@jcrben It had been a trend, but thankfully it's no longer all that common. See e.g.this insightful comment for an explanation. Here's an excerpt:
tl;dr Windows junctions are nowhere close to being a valid symlink replacement. |
jcrben commentedFeb 4, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@dscho well, in some ways using junctions is becoming more common - fnm switched to junctions in 2021, volta just did in 2024, jbangdev did it in 2024 at my urging, kubectl plugin manager krew may also do it. avoiding a complex and scary step of enabling Developer Mode is pretty nice. wish we could get symlinks on Windows without enabling Developer Mode. See also:Why do you have to be an admin to create a symlink in Windows? |
sskras commentedFeb 4, 2025
@jcrben writes:
Exactly that. You don't need the Developer Mode or become elevated every time you create a symlink. Assigning |
dscho commentedFeb 4, 2025
@jcrben Yes, and that's true even when mentioning better-known projects (I never heard about your examples, but IIRC The important thing to keep in mind? The usage resembles Unix mount points, much more than symbolic links. I do encourage you to studythe insightful comment about this that I linked earlier.
@sskras That's enticing, but leaves the reader hanging for a clear description how to do that. In other words, your comment is too terse, saves too much on time at the cost of being less helpful than it could be. Please do fill in the blanks. |
jeremyd2019 commentedFeb 4, 2025
https://packages.msys2.org/packages/mingw-w64-x86_64-editrights editrights -u YourUser -a SeCreateSymbolicLinkPrivilege? (untested) |
This adds the missing documentation (in case we ever publish theinformation about the MSYS variable on msys2.org).Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…llbackWhen native symlinks are available, it is a shame to create deep copiesby default.However, since there are many scenarios where symlinks are not available(e.g. when running on FAT, or on older Windows versions, or whenDeveloper Mode is not enabled), we've got to have a fallback.In the regular Cygwin world, it is legitimate to fall back to WSLsymlinks and/or to the system file emulation (where a file is createdthat is marked with the "system" attribute and with content that adheresto a specific, magic form that is recognized specifically by the Cygwinruntime).However, in the world of MSYS2, the assumption is that the result of theoperation should be as interoperable with regular Win32 programs aspossible. Hence the default to "deepcopy".As a "best of both worlds" mode, let's implement one that tries tocreate native symlinks by default, and if that fails, uses the"deepcopy" method as a fallback.This addressesmsys2#113.Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
b8e842e tofeecd91Comparesskras commentedFeb 7, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@dscho commented3 days ago
OK. There are different ways/tools to configure this: 1, using
I didn't think earlier a Windows developer would need guidance on that, but now visualized usage of the 1st tool:
@jeremyd2019 commented3 days ago
Thanks, I didn't know about this tool! Might be a nice addition to the list above. |
dscho commentedFeb 7, 2025
Honestly, I was thinking more about an automatable solution that requires nothing but PowerShell, something like: # Export the current security settings to a file$SecEditExportFile="$env:TEMP\secedit.inf"secedit/export/cfg$SecEditExportFile/areas USER_RIGHTS# Modify the security settings file to include the privilege for the user$User="DOMAIN\User"$Privilege="SeCreateSymbolicLinkPrivilege"# Read the exported settings file$lines=Get-Content$SecEditExportFile# Add the privilege for the user$index=$lines.IndexOf("[Privilege Rights]")$privilegeLine=$lines[$index..($lines.Length-1)]|Where-Object {$_-match"^$Privilege" }if ($privilegeLine) {$existingValues=$privilegeLine-replace"$Privilege =",""$newValues="$existingValues,$User"$lines[$lines.IndexOf($privilegeLine)]="$Privilege =$newValues"}else {$lines+="$Privilege =$User"}# Write the modified settings back to the file$lines|Set-Content$SecEditExportFile# Import the modified security settings:secedit/configure/db secedit.sdb/cfg$SecEditExportFile/areas USER_RIGHTS# Refresh the policy:gpupdate/force Since this here is a public bug tracker, one must assume that people might find this and would be delighted by guidance how to achieve the goal they set out to accomplish when trying to find more information.
Stating that you didn't think earlier a Windows developer would need guidance on that may very well come over as pretty smug and arrogant, and is therefore potentially something you'd like to avoid. |
sskras commentedFeb 7, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@jeremyd2019, ![]() ![]() Thank you. |
sskras commentedFeb 7, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@dscho wrote:
It still relies on running Easily extendable to add the new IIUC this would mean that the current install of the supported Windows versions (w10, w11) is misconfigured or pretty broken, eg. having corrupt Security Database. But I am not against that. And... how is it better than using
I see your point. For starters, as an ordinary user of MSYS2 I hope to find such info on the installation page, eg:https://www.msys2.org/#installation Crossing fingers so the ways get discussed properly and then get better publicity. Cheers :)
That's sad. What drove me is not the pride, but the ambiguity of the ways to solve this + my aim at keeping the Signal-to-Noise ratio high. I hope I proved that by illustrating two of these ways. |
culyun commentedAug 2, 2025
Came here looking for answers, read the thread, got good info. The conversation is fine. Probably the main concern is making this work out of the box in the corporate environment. Automatically enabling NTFS symlinks with an opt-out (like Git-Bash) should be the chief goal here. |
Git for Windows has supported symbolic links for quite some time:Ingit-for-windows#156, this support wasintroduced already into Git for Windows v2.4.2.windows.1 in May 2015.However, the Git for Windows CI never ran the test suite with symboliclink support because the MSYS2 runtime (i.e. the POSIX emulation layerrequired to run Git's test suite because the latter is written in Unixshell script) does not support symbolic links right out of the box. Thisis for historical reasons: Symbolic link support was introduced inWindows 7, but these links could only be created by administrators bydefault, and it took until Windows 10 Build 14972 that at least inDeveloper Mode, non-administrators would be permitted to create them.The MSYS2 runtime _does_ have some sort of support for symbolic links,although with caveats: seeing as it expects the inputs as Unix-likepaths, but the outputs need to be Win32 symbolic links pointing to Win32paths, some normalization has to be performed in the process. This leadsto sometimes surprising behavior e.g. when a link target like `a/b/..`is normalized to `a`.It has been a minute or three since the time when Windows versionswithout symbolic link support were common, therefore there are plans toturn on that support in the MSYS2 runtime on these Windows versions bydefault, seemsys2/msys2-runtime#114 for moredetails about this.To prepare for this, I am working toward upstreaming Git for Windows'own support for symbolic links. And to prepare for that, in turn, I amhereby contributing preemptively the fixes required to eventually letGit's test suite pass when both MSYS2 runtime and Git support symboliclinks.Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>




Uh oh!
There was an error while loading.Please reload this page.
It surprises new MSYS2 users no end that
ln -sdoes not create symbolic links at all, but deep copies (and with an exit code indicating success!). This did not at all match the expectations of those users who were familiar with Unix' concept of symbolic links and thought that they could rely on MSYS2 providing those, too, or fail with a non-zero exit code.Historical reasons are at play here: When MSYS2 was started (or was that already the behavior of MSys? I forget...), symbolic links were not supported on Windows, at least not really: you had to have administrator privileges to create them (but not to delete them... 🤷) in Windows Vista, and before that, Windows simply had no idea about symbolic links.
So what about Cygwin? Well, Cygwin had something like support for symbolic links, using
.lnkfiles for the emulation. The only problem? You had to stay within Cygwin's walled garden to make use of them. All non-Cygwin applications would react with a less or more unpleasant "huh?!?" when encountering those "symbolic links".That's why MSYS2 chose to deep-copy by default. At least that way
./configurewould still work for those projects that required symbolic links. This was instrumental in getting MSYS2's package ecosystem off the ground.Even when a Windows 10 update introduced support for creating symbolic links without elevation as long as Windows was run in Developer Mode, the created symbolic links are notcompletely what Unix/Linux/macOS users may be used to, as Windows discerns between directory symlinks and file symlinks.
Be that as it may, now that we've dropped Windows 7 and Windows 8 support, it may be a good time to start switching the default to creating actual symbolic links by default.
Since we still support Windows 8.1 (and a couple of Windows 10 versions that do not allow creating symbolic links in non-elevated operations, even in Developer Mode), we cannot simply switch to a mode where the MSYS2 runtime creates symbolic links when asked for, but we have to have a mode where the MSYS2 runtime first checks whether that is possible with the Windows version on which it is running, and if not, falls back to the deep-copy.
This PR does precisely that: implement that mode, but doesnot yet flip the default away from
deepcopy. The reason is that I want this to be tested by volunteers (myself included) first, and once it is deemed robust and stable enough, flip the default tonativeordeepcopy.This addresses#113.