When building Coder Desktop for distribution for the first time, and running the app on a device with System Integrity Protection, we hit a bit of a roadblock:

By default, when writing executable files in sandboxed apps, the files are quarantined. Gatekeeper prevents quarantined executable files and other similar files (shell scripts, web archives, and so on) from opening or executing unless the user explicitly launches them from Finder.

If your app needs to create executable files that are typically executed in some way other than through Launch Services (shell scripts, for example), you should also specify the com.apple.security.files.user-selected.executable entitlement.

If those executables are tools that are intended to run from the command line, such as shell scripts, this presents a problem. With this flag, the file quarantine system allows the app to write non-quarantined executables so that Gatekeeper does not prevent them from executing.

As a result, the CoderVPN.dylib downloaded from the Coder deployment has thecom.apple.quarantine xattr set, prompting Gatekeeper to check it. Presumably because the.dylib isn't notarized, it gets blocked. This wasn't noticed during earlier prototyping as the file was never created within the System Extension sandbox, and so it never had the xattr.

Unfortunately, System Network Extensionsmust run in a sandbox. I was also unable to abuse thecom.apple.security.files.user-selected.executable entitlement to remove the quarantinexattr from the file - all signs point to theuser-selected part of the entitlement being very much crucial.

Fortunately, we're not required to run the application itself sandboxed. When not sandboxed, an app is able to remove the quarantine xattr.
Of note here is that our System Network Extensions runs asroot, whilst the applications runs as the logged-in user. This means there is no overlap in files writable by both the Network Extension, and the app[1].

Given this, one thing I explored was if we could somehow notarize the.dylib, and have Gatekeeper use that notarization ticket to remove the quarantine. Unfortunately, notarizing the dylib alone (or within a framework bundle) had no effect when attempting to open the.dylib on an internet-connected computer. Furthermore, there's no way to staple a notarization ticket to an unbundled.dylib. Placing it into a container that can be stapled, such as a.dmg or a.pkg would make it unsbale by the sandboxed network extension - it cannot mount the disk image (which was the source of other sandbox escapes), and it cannot work with apkg installer.

I've sent in an Apple Support ticket asking if we've missed anything on this topic, and if there's potentially a better way, though from what I've read online that would appear unlikely - if we were able to execute the dylib without the user's explicit consent, it would constitute a sandbox escape CVE[2].

Employed Workaround

To deal with this issue, this PR removes the app sandbox from the Coder Desktop application, and uses a privilegedAppleScript command to run/usr/bin/xattr -d com.apple.quarantine /path/to/dylib - this is requested over XPC, and the Network Extension waits for the user to complete thesudo prompt before continuing.

Some downsides to this workaround are:

• We're no longer sandboxing the application, which is the 'right' thing to do.
• The user is prompted for their login password each time a new instance of the.dylib is downloaded. (This is annoying for us at Coder - our Coder deployment updates on each push tocoder/coder/main.)
• If the quarantine flag is present, but the app is not running (such as when the VPN is started from System Settings), the VPN won't start.