Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-102255: Improve build support on xbox#102256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
zooba merged 104 commits intopython:mainfrommaxbachmann:xbox
Mar 9, 2023
Merged

gh-102255: Improve build support on xbox#102256

zooba merged 104 commits intopython:mainfrommaxbachmann:xbox
Mar 9, 2023

Conversation

@maxbachmann
Copy link
Contributor

@maxbachmannmaxbachmann commentedFeb 25, 2023
edited by bedevere-bot
Loading

Co-authored-by: Eryk Sun <eryksun@gmail.com>
@zooba
Copy link
Member

This looks like it's ready, or very close.@maxbachmann,@eryksun is there anything left to do here? I don't mind it being done in future PRs if they're necessary.

@maxbachmann
Copy link
ContributorAuthor

I am done with everything I planned for this PR. As suggested by@eryksun I plan to extend thesys module to retrieve the api partition. I will add this in a separate PR.

zooba reacted with thumbs up emoji

@zooba
Copy link
Member

I plan to extend thesys module to retrieve the api partition. I will add this in a separate PR.

Give that a new issue as well. We really ought to add the platform tosys.implementation as well, since currently it's a bit weird to check for 32-bit vs 64-bit vs ARM64 as well (sys.winver is the only way). I assume that would be where the API partition is indicated, too.

maxbachmann reacted with thumbs up emoji

maxbachmannand others added2 commitsMarch 6, 2023 20:07
Co-authored-by: Eryk Sun <eryksun@gmail.com>
Co-authored-by: Eryk Sun <eryksun@gmail.com>
maxbachmannand others added2 commitsMarch 6, 2023 21:00
Co-authored-by: Eryk Sun <eryksun@gmail.com>
Co-authored-by: Eryk Sun <eryksun@gmail.com>
size_tfile_len=relfile ?wcslen(relfile) :0;
/* path is at max dirname + filename + backslash + \0 */
size_tnew_len=dir_len+file_len+2;
if (bufsize >=MAXPATHLEN||new_len>bufsize) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The value ofbufsize isn't limited. The result length may be limited. IfPATHCCH_ALLOW_LONG_PATHS isn't set inflags, andnew_len >= MAX_PATH (notMAXPATHLEN), the returned error should beHRESULT_FROM_WIN32(ERROR_FILENAME_EXCED_RANGE).

Else ifnew_len is greater thanbufsize, the returned error should beHRESULT_FROM_WIN32(ERROR_INSUFFICIENT_BUFFER).

PATHCCH_ALLOW_LONG_PATHS also determines whether to convert to or from an extended path, depending on whether the current process supports long paths (Python 3.6+ does). But it's okay to omit that behavior forGAMES.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Oh yeah, we shouldn't have references to MAXPATHLEN in here.

I'm not so concerned about error codes. They're only going to be reported to users, and really we ought to be getting the buffer right ourselves before calling.

I think it's fine for this implementation to assumePATHCCH_ALLOW_LONG_PATHS. Don't worry about mimicking every possible behaviour of the original.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
if (bufsize >=MAXPATHLEN||new_len>bufsize) {
if (new_len>bufsize) {

Don't worry about mimicking every possible behaviour of the original.

I agree, if getting the correct behavior is excessively complicated with no tangible benefit. What about my other comment on this function? The wayPathCchStripToRoot() is used, whenrelfile is a rooted path, leads to incorrect and inconsistent behavior. If I know I'm combining a rooted path, I may use a smaller buffer because I know that most ofdirname won't be copied -- just the drive part. The current implementation requires using a buffer that's big enough to holddirname,relname and a joining backslash. OTOH, ifrelname has a drive or UNC share and thus replacesdirname, then the length of the passed-indirname is ignored.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Firstly, the suggested change is good.

I'm not particularly worried about the case whererelname would shorten the overall drive. It seems something youcould take advantage of, but generally we don't control either half of the paths that will come in here, which means the caller is going to have to assume worst case of plain-old concatenation.

Maybe just add a comment like/* assuming worst case result length, so our caller should as well */.

The only reasonable alternative (without complicating everything) would involve modifying the result buffer before potentially returning an error, which I think is worse. I'm pretty sure the original API uses temporary buffers internally, but there's no real need to go there.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yes I would assume the implementation to use something likePathAllocCombine internally

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

If we really wanted to I could come up with an implementation that behaves closer toPathCchCombineEx. I was simply not sure it was worth the complexity (I kind of hope that at some point microsoft makes these APIs available on the xbox, since there does not appear to be any inherent reason to provide them on every platform except the xbox)

zooba reacted with thumbs up emoji
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The only reasonable alternative (without complicating everything) would involve modifying the result buffer before potentially returning an error, which I think is worse. I'm pretty sure the original API uses temporary buffers internally, but there's no real need to go there.

It would calculatedir_len based onPathCchSkipRoot() ifrelfile begins with a backslash. Then it's the same initial test based onnew_len = dir_len + file_len + 2, requiringnew_len < bufsize. Copyingdirname would now usewcsncpy(buffer, dirname, dir_len), andPathCchStripToRoot() would not have to be implemented. So you get better behavior AND it's simpler overall.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hold off on any more changes on this function for a day or two - I'm having some interesting conversations with colleagues right now. Might have a more suitable approach (or at least more information about it).

maxbachmann reacted with thumbs up emoji
Co-authored-by: Eryk Sun <eryksun@gmail.com>
@zooba
Copy link
Member

Okay, thanks for holding. What I'm hearing from my colleagues at Microsoft is that thePathCch* APIs are implemented in the Xbox OS, just not exposed through the partition or their import lib. It sounds like both of these will be fixed for an update later this year, at which point our original code will be fine. The only restriction seems to be that the APIs are not implemented on Win7, so even though games are meant to be compatible all the way back, if we were to use the API then it would not work.

Until the GDK update arrives, itseems like we can probably use code like what we used to have to load the API dynamically (fromgetpathp.c in the 3.7 branch):

static int _PathCchCombineEx_Initialized = 0;typedef HRESULT(__stdcall *PPathCchCombineEx) (PWSTR pszPathOut, size_t cchPathOut,                                               PCWSTR pszPathIn, PCWSTR pszMore,                                               unsigned long dwFlags);static PPathCchCombineEx _PathCchCombineEx;static voidjoin(wchar_t *buffer, const wchar_t *stuff){    if (_PathCchCombineEx_Initialized == 0) {        HMODULE pathapi = LoadLibraryExW(L"api-ms-win-core-path-l1-1-0.dll", NULL,                                         LOAD_LIBRARY_SEARCH_SYSTEM32);        if (pathapi) {            _PathCchCombineEx = (PPathCchCombineEx)GetProcAddress(pathapi, "PathCchCombineEx");        }        else {            _PathCchCombineEx = NULL;        }        _PathCchCombineEx_Initialized = 1;    }    if (_PathCchCombineEx) {        if (FAILED(_PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0))) {            Py_FatalError("buffer overflow in getpathp.c's join()");        }    } else {        if (!PathCombineW(buffer, buffer, stuff)) {            Py_FatalError("buffer overflow in getpathp.c's join()");        }    }}

@maxbachmann Would you be able to give this approach a try and see if it works? Unfortunately, I don't think there's any way to detect the version of the games SDK involved, so we'd just have to keep this until we assume everyone's on the fixed update.

If it helps (and I suspect it won't), it also ought to be okay to temporarily define the PARTITION constants needed when includingpathcch.h. It's more likely going to be better toexclude the include statement in GAMES when we're defining a GetProcAddress wrapper, since that way we can define our ownPathCchCombineEx implementation and it shouldn't collide with the real one until we choose to let it. You can probably even keep the existing implementation as a fallback for the (hopefully rare) cases where games may be running on Win7, but I'm happy to defer to you on that one, as you've likely got a better view of the gamedev landscape than I do.

@maxbachmann
Copy link
ContributorAuthor

Okay, thanks for holding. What I'm hearing from my colleagues at Microsoft is that the PathCch* APIs are implemented in the Xbox OS, just not exposed through the partition or their import lib. It sounds like both of these will be fixed for an update later this year, at which point our original code will be fine.

Yes I did read in the developer forums, that there is already an implementation as well. Great to hear that they are likely going to get exposed at some point.

Would you be able to give this approach a try and see if it works? Unfortunately, I don't think there's any way to detect the version of the games SDK involved, so we'd just have to keep this until we assume everyone's on the fixed update.

just gave this a quick test and it works both on the Xbox One and Xbox Scarlett.

You can probably even keep the existing implementation as a fallback for the (hopefully rare) cases where games may be running on Win7, but I'm happy to defer to you on that one, as you've likely got a better view of the gamedev landscape than I do.

I do not think we need to take Windows 7 into consideration here:

  1. Cpython overall does not support Windows 7 anymore
  2. when building games for the PC you can always build against the desktop partition. Right now you only really need to use the games partition on the xbox, which does not use Windows 7 anyways.
  3. even on PC the share of Windows 7 users is really slim. According to the current steam hardware survey around 1.5% of Windows users is on Windows 7:https://store.steampowered.com/hwsurvey/directx/
zooba reacted with rocket emoji

@pythonpython deleted a comment frombedevere-botMar 9, 2023
@zooba
Copy link
Member

!buildbot .Windows.

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@zooba for commit331b1f4 🤖

The command will test the builders whose names match following regular expression:.*Windows.*

The builders matched are:

  • AMD64 Windows10 PR
  • ARM64 Windows Non-Debug PR
  • AMD64 Windows10 Pro PR
  • ARM64 Windows PR

@zooba
Copy link
Member

There we go. I expect there'll be no buildbot issues, but worth double checking (there are some unrelated stack issues, so may still fail, but we can ignore those).

Other than that, I'm happy to merge this whenever.@eryksun I'll wait for a positive signal from you.

externPyObject*PyInit_gc(void);
externPyObject*PyInit_nt(void);
externPyObject*PyInit__signal(void);
#if defined(MS_WINDOWS_DESKTOP)|| defined(MS_WINDOWS_SYSTEM)|| defined(MS_WINDOWS_GAMES)
Copy link
Contributor

@eryksuneryksunMar 9, 2023
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@zooba, since the check for building winreg and mmap occurs in several places (here, in "PC/config.c", and the respective modules), maybeHAVE_WINDOWS_REG_API andHAVE_WINDOWS_MMAP_API should be defined in "PC/pyconfig.h". And to match that, maybe renameHAVE_WINDOWS_CONSOLE_IO toHAVE_WINDOWS_CONSOLE_API. That's a better name anyway, sinceGenerateConsoleCtrlEvent() is a console API, but not one that's particularly related to I/O.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The mmap API should lose the definitions fairly soon, we just spun that work out to another issue.

For winreg I wouldn't worry aboutpyconfig.h, but the definition could be calculated at the top of the file. Those aren't C API functions, so they can be detected byAttributeError.

None of these feel important enough to block on, but I wouldn't oppose doing them later. Let's call this PR done and let Max focus on the other changes that have been lined up

@zoobazooba merged commitc6858d1 intopython:mainMar 9, 2023
@zooba
Copy link
Member

Thanks@maxbachmann! This was a huge effort, but I think it's worth it. Don't shy away from adjusting things we decided here if it seems important - nothing we changed should count as a public API change, so we can revert or modify if needed.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@eryksuneryksuneryksun left review comments

@zoobazoobazooba left review comments

@gpsheadgpsheadAwaiting requested review from gpsheadgpshead is a code owner

@pablogsalpablogsalAwaiting requested review from pablogsalpablogsal is a code owner

@lysnikolaoulysnikolaouAwaiting requested review from lysnikolaoulysnikolaou is a code owner

@pgansslepganssleAwaiting requested review from pgansslepganssle is a code owner

@abalkinabalkinAwaiting requested review from abalkinabalkin is a code owner

@tirantiranAwaiting requested review from tiran

@iritkatrieliritkatrielAwaiting requested review from iritkatriel

Assignees

No one assigned

Labels

buildThe build process and cross-buildextension-modulesC modules in the Modules dirinterpreter-core(Objects, Python, Grammar, and Parser dirs)OS-windows

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@maxbachmann@zooba@bedevere-bot@eryksun@arhadthedev

[8]ページ先頭

©2009-2025 Movatter.jp