Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
GH-113464: A copy-and-patch JIT compiler#113465
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Closed
Uh oh!
There was an error while loading.Please reload this page.
Closed
Changes from1 commit
Commits
Show all changes
459 commits Select commitHold shift + click to select a range
8f6c5fe Remove the ip_offset hack and executor local
brandtbuchere0888de Merge branch 'main' into justin
brandtbucher365f7a2 Get rid of ip_offset
brandtbucher7fb79b8 Catch up with main
brandtbucher66a5659 fixup
brandtbucherf90bd1e Catch up with main
brandtbuchera98d4fe Crank up the trace length for nbody
brandtbucherdc308ba Catch up with main
brandtbucherf2b18d7 Add rough assembly dumps to jit_stencils.h
brandtbuchere2ae7bc Make machine code disassemblies more robust
brandtbucherbe88965 Go back to llvm-objdump like the good 'ol days
brandtbucher56976b3 Remove old debugging struct
brandtbuchere33b4ab -fno-pic
brandtbucher3ac1946 Take a triple (and debug flag) instead of guessing
brandtbucher50731b5 Use more regular configs and fix debug builds
brandtbuchere559898 Try using cross-compiles for aarch64
brandtbucher317ba79 Fix quotes
brandtbucher07e03a1 apt update
brandtbuchere649059 No pkg-config
brandtbucher11c7f44 Fix PGO and disable multiprocessing
brandtbucher4981573 Add missing excludes
brandtbucher1688830 Switch to enums
brandtbucher29936e6 Catch up with main
brandtbuchered75bd6 More cleanup
brandtbucher8488def Rename some things, and don't constantly regen
brandtbucher28351a0 Clean up a bunch of stuff
brandtbuchercc5db79 Patch opargs and operands in the continuation
brandtbucherf2817bb Fix GHCCC hack
brandtbucherfb96638 Blacken
brandtbucher77e1487 Fix incorrect signature
brandtbucher5e9259d -fpic -mcmodel=small
brandtbucherff1d6a7 -fno-pic -mcmodel=medium
brandtbucheraf9f114 Catch up with main
brandtbucherc8b5073 Catch up with justin
brandtbucher57a8019 Switch to -mcmodel=small -fpic on two platforms
brandtbucher68a87a1 Ditto for Intel Macs
brandtbucherf44767b ELF cleanup
brandtbucher60d6295 More cleanup
brandtbucherd634cd3 Catch up with main
brandtbucherca4956f Add R_X86_64_PC32
brandtbucher092c795 Fixes for aarch64 and i686
brandtbucher15ba80d ELF everywhere!
brandtbucherf2927ae Fix some leftover bugs
brandtbucher26abedf More cleanup
brandtbucher1cb4a1f Catch up with main
brandtbucherb2e47f9 Catch up with justin
brandtbucher1799b79 Get rid of an old workaround
brandtbucheraef1e42 Not my fault
brandtbucher7b33e21 More cleanup
brandtbucher32b2708 Fix relocations
brandtbucher94448ed Get rid of dynamic lookup
brandtbucherd08d53c Clean things up for merge
brandtbucher502ab9c Get rid of some extra padding
brandtbucherf87aab2 Clean up the diff
brandtbucher83492dd Fix 64-bit Windows
brandtbucherc4b166f Clean up formatting
brandtbucher0c4e147 fixup
brandtbucher0f23be3 More cleanup
brandtbucher8c62663 Merge branch 'main' into justin
brandtbucher3497f8c Add a short README
brandtbucher0be489a More cleanup
brandtbucher73eb3ce Catch up with main
brandtbuchere529809 Reenable more tests
brandtbucher1a71c1a Add ip_offset back
brandtbucherb13056f Remove some comments
brandtbucher898dcc2 Comments
brandtbucher720878f Load opargs and operands only when needed
brandtbucher07fb485 Be smarter about opargs and operands
brandtbucheraa9d3b3 Rerun CI
brandtbucher8db6838 Add error stub and clean things up more
brandtbuchere0a69d1 Split stuff into instructions and data
brandtbucherd33cc95 Fix Windows?
brandtbucher2c78f70 ...fix Windows?
brandtbuchera62e82d More fixes...
brandtbucher878dc80 Last try tonight
brandtbucherc440e55 Okay, for real
brandtbucher169acb3 Fix missing ghccc
brandtbucherd6fa4ad Clean up opargs and operands
brandtbucher4ffe087 Reuse static const data
brandtbucherff195e2 Clean up relocations
brandtbucher7de5664 Get the addend from the correct segment on i686
brandtbucher1963c66 Don't mprotect 0 bytes of memory
brandtbucher3b54bb6 Catch up with main
brandtbucherfb3afb5 Fix tier two
brandtbucherd934ddf Clean up the diff
brandtbucher21a9efd Don't overallocate data
brandtbucher7eaec09 Fix alignment stuff
brandtbuchercdf8b1d Catch up with main
brandtbucher5f4e950 Catch up with main
brandtbucherc4bf874 Catch up with main
brandtbuchereda424d Catch up with main
brandtbucher3c816f8 Make BEFORE_WITH a uop
brandtbucher824cb77 Catch up with main
brandtbucher7aa253c Fix bad merge
brandtbucher4f60977 Clean up the diff a tiny bit
brandtbucherd65308a Disable BEFORE_ASYNC_WITH/BEFORE_WITH/INSERT
brandtbucherfb676a6 Disable the GHCCC hack
brandtbucher0e8a50a Don't use ELF everywhere (WIP)
brandtbucherc997b66 Fix burn-in on LLVM 16
brandtbucherc26012b Increase data sharing
brandtbucher82579d2 Get rid of ghccc hack (for now)
brandtbucher3930170 Add COFF support back
brandtbucher71dd8be Catch up with justin
brandtbucherd2687e4 Only compile once
brandtbucher5774f6d Mach-O!
brandtbuchercd05c17 Re-enable macOS job
brandtbucher241ce0f Catch up with main
brandtbucher5137145 Catch up with justin
brandtbucherffac31b Fix types and require LLVM 16
brandtbucher888d9ca Un-skip some tests
brandtbucher4611265 Catch up with main
brandtbucher4d3a081 Catch up with justin
brandtbucher96c3499 Catch up with main
brandtbuchercc201a9 Catch up with justin
brandtbucher0446e98 Catch up with main
brandtbucher6f86a0f Remove unnecessary relocation types
brandtbucher7a7e995 Start cleaning up the allocator
brandtbucherc4904e4 Clean up and rename stencil stuff
brandtbucher8021c98 Remove groups and excludes, just for fun
brandtbucherf17495b More tweaks
brandtbucherc2b5b57 fixup
brandtbucherd39d915 fixup
brandtbucher249e12a fixup
brandtbucherf142c2f Rework build.py to match jit.c a bit more closely
brandtbucher6490a71 Fix M1 builds
brandtbucher27a50cf Rename stuff and protect readable memory
brandtbucher89996d9 Catch up with main
brandtbucher58bd20c Make the jit an optional option
brandtbucher7bb54b5 Catch up with main
brandtbucher7a592f0 Catch up with main
brandtbucher1a8a63b Catch up with justin
brandtbucher40614ee Skip some more aarch64 tests
brandtbucher5bbda58 More cleanup
brandtbucher8342f07 black + mypy
brandtbucher925f140 Refactor JIT code: Remove deoptimize and error
brandtbucher4ae9371 Rename trampoline to wrapper, and only warn once
brandtbuchere295407 Add schema for JSON, and clean things up
brandtbucher3b64124 Remove deoptimize and error stubs
brandtbucher4cb7a87 Catch up with main
brandtbucher5bf5187 Catch up with main
brandtbucherdfface9 Enable CI temporarily
brandtbuchera9637a8 Catch up with man
brandtbucher0466d7a Stop using the static data hack
brandtbucher00abf88 Comment out memory debugging code
brandtbuchera9c613e Clean up JIT initialization and memory allocation
brandtbucher094581a Fix Windows madness
brandtbucherbc5ba12 Clean up some types
brandtbucher42d96de Get rid of runtime trampolines
brandtbucher1fa6d7b Fix byteorder issue
brandtbucher01ffefc Remove trampoline.c and update STUBS list
brandtbuchered95f11 Catch up with main
brandtbucher772d868 Catch up with justin
brandtbucher7809cce Get rid of trampolines
brandtbucherdcb290a Use dataclasses.replace
brandtbucher86ed6a1 Catch up with main
brandtbucher8349218 Lots of cleanup
brandtbuchere8176b8 Don't leak memory, and don't emit wrappers!
brandtbucherc3e2aab Clean up relocations and fix Windows nojit builds
brandtbucher3af4a4b More Windows fixes
brandtbuchera4a2862 More cleanup
brandtbucher780b678 Make it possible to enable the tier 2 interpreter
brandtbucher9901843 A bunch of cleanup (and proper debug compiles)
brandtbucherab556e7 Don't emit unwind info (and fix __bzero)
brandtbucherc458df4 Catch up with main
brandtbucher9c1867b asm not needed for large -mcmodel=large
brandtbucher882263c Catch up with main
brandtbucher522adc5 Catch up with main
brandtbucher86d4fd7 fixup
brandtbucher94f0877 Tons of cleanup... prepare to revert!
brandtbucherbfa6533 fixup
brandtbucherc5c44c8 fixup
brandtbucher56fbb07 fixup
brandtbuchera8931fe ...fixup?
brandtbucher1fcdc94 ...fixup.
brandtbucher64f903c Put pyconfig.h where it needs to be
brandtbucher4e67db4 Fix Windows/out-of-tree/hashing/verbosity
brandtbucher78f01ae Fix bit manipulation bugs in jit.c
brandtbucher73ed82b Catch up with main
brandtbucher53c35f3 Clean up configure
brandtbucherdc55796 Catch up with main
brandtbucher770a480 Clean up jit.vcxproj
brandtbucher8998952 fixup
brandtbucher3d04346 Put jit_stencils.h in the build root
brandtbucher46063fb fixup
brandtbucher8addbbe Catch up with main
brandtbucher6b19d7c Clean up the AArch64 stuff
brandtbucherb63610e Break stuff up
brandtbucher44a024a More cleanup
brandtbucheraf7606d Merge Parser and Target
brandtbucherf238057 Turn relocations into holes quickly
brandtbucher50b0df8 Fix AArch64 Macs.
brandtbucher0d358ef Final-ish cleanup
brandtbucher323a9d6 blurb add
brandtbucherd36c9de Enable JIT tests on pull_request
brandtbucher7026d0c Nevermind
brandtbucher412c68d Catch up with main
brandtbucherb1f1c9f Add Include/internal/mimalloc to include path
brandtbuchere9faa27 Catch up with main
brandtbucher4229327 Re-enable JIT CI
brandtbucher2643439 Lots of little cleanups from code review
brandtbucher3342775 Catch up with main
brandtbucherc1b7007 Move some logic out of try blocks
brandtbucher20ad5f5 Clean up some type annotations
brandtbucherc518c09 Catch up with main
brandtbucher5e229ed Consider target when caching the JIT stencils
brandtbucherd01a2c7 Clarify why LLVM is required
brandtbucher4266f5a Catch up with main (and never interpret tier two on JIT builds)
brandtbucher6ebd085 fixup
brandtbucher2d17e22 JIT everything (even the counter thing)
brandtbucherafbc1f5 fixup
brandtbucher5315b1d Wrap machine code bytes.
brandtbucher6b84187 Use better hex formatting
brandtbucher675dd0c Catch up with main
brandtbucherbc6763e Clean up the LLVM utilities
brandtbucherfb62bc8 Remove duplicated code (and unnecessary sorting)
brandtbucher344683f Fix machine code formatting
brandtbucher2636ece Clean up the build script (mostly comments and renaming)
brandtbucherefbe31a Split things up a bit more
brandtbucher23fad98 Add missing braces to Hole's C dump
brandtbucher44b194f Move relocation processing to StencilGroup
brandtbuchereb5f21d Move symbols to StencilGroup
brandtbucher0dca530 Make some nested schema items private
brandtbuchera249170 Move more logic to from Target to StencilGroup
brandtbucher1a956bd Move the version check to where it's needed
brandtbuchere78341d Catch up with main
brandtbucher0bba20f Remove bad asserts
brandtbucherb4474d0 Catch up with main
brandtbucher41162ca Rework PCbuild stuff based on PR review
brandtbucherdebb24d Make JIT workflow conditional on files changed, add a dispatch button…
brandtbucher4f656e0 Make the README more future-proof
brandtbucher0c9c806 Set up incremental builds and fix PGO
brandtbucher79b1bbf Raise instead of warning, rename bit-shifting arguments, and move pat…
brandtbucher6643773 Fix error message
brandtbucher663ba31 Catch up with main
brandtbucher21ac0f2 Add const qualifier
brandtbucherb352c6b Catch up with main
brandtbucher763f527 Actions runners are beefier now
brandtbucherb15529c Fix some typos
brandtbucher1cee739 Reorder ELF and COFF
brandtbucher81d0fb6 Don't include Python/, add a --force option, and unify section/symbol…
brandtbucher437bc53 Make _PyJIT_Compile take a trace and length
brandtbucher26e248a Move header comment to build.py
brandtbucherf3984e6 Clean up the schema
brandtbucher113a1f1 Reorder some keys
brandtbucher6648d84 Rearrange (and add) comments for patch function
brandtbucher46bee49 Use generated pyconfig.h directory
brandtbucherc33c78f Catch up with main
brandtbucher4d3e207 Add back Python/ include
brandtbucher581036e Catch up with main
brandtbucherb00e021 Merge branch 'main' into justin
brandtbucherFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
Lots of cleanup
- Loading branch information
Uh oh!
There was an error while loading.Please reload this page.
commit8349218a7f1bda379b900d77a0edc8cf7d271f76
There are no files selected for viewing
222 changes: 75 additions & 147 deletionsPython/jit.c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -23,116 +23,94 @@ | ||
| #include <sys/mman.h> | ||
| #endif | ||
| #define FITS_IN_BITS(U, B) \ | ||
| (((int64_t)(U) >= -(1LL << ((B) - 1))) && ((int64_t)(U) < (1LL << ((B) - 1)))) | ||
| staticuint64_t | ||
| get_page_size(void) | ||
| { | ||
| #ifdef MS_WINDOWS | ||
| SYSTEM_INFO si; | ||
| GetSystemInfo(&si); | ||
| return si.dwPageSize; | ||
| #else | ||
| return sysconf(_SC_PAGESIZE); | ||
| #endif | ||
| } | ||
| static void | ||
| warn(const char *message) | ||
| { | ||
| #ifdef MS_WINDOWS | ||
| int errno = GetLastError(); | ||
| #endif | ||
| PyErr_WarnFormat(PyExc_RuntimeWarning, 0, "JIT %s (%d)", message, errno); | ||
| } | ||
| static char * | ||
| alloc(uint64_tsize) | ||
| { | ||
| assert(size); | ||
| assert(size % get_page_size() == 0); | ||
| #ifdef MS_WINDOWS | ||
brandtbucher marked this conversation as resolved. Show resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| char *memory = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE); | ||
| int failed = memory == NULL; | ||
| #else | ||
| char *memory = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); | ||
| int failed = memory == MAP_FAILED; | ||
| #endif | ||
| if (failed) { | ||
| warn("unable to allocate memory"); | ||
| return NULL; | ||
| } | ||
| return memory; | ||
| } | ||
| static int | ||
| mark_executable(char *memory, uint64_tsize) | ||
| { | ||
| if (size == 0) { | ||
| return 0; | ||
| } | ||
| assert(size% get_page_size() == 0); | ||
| #ifdef MS_WINDOWS | ||
| if (!FlushInstructionCache(GetCurrentProcess(), memory, size)) { | ||
| warn("unable to flush instruction cache"); | ||
| return -1; | ||
| } | ||
| DWORD old; | ||
| int failed = !VirtualProtect(memory, size, PAGE_EXECUTE, &old); | ||
| #else | ||
| __builtin___clear_cache((char *)memory, (char *)memory + size); | ||
| int failed = mprotect(memory, size, PROT_EXEC); | ||
| #endif | ||
| if (failed) { | ||
| warn("unable to protect executable memory"); | ||
| return -1; | ||
| } | ||
| return 0; | ||
| } | ||
| static int | ||
| mark_readable(char *memory, uint64_tsize) | ||
| { | ||
| if (size == 0) { | ||
| return 0; | ||
| } | ||
| assert(size% get_page_size() == 0); | ||
| #ifdef MS_WINDOWS | ||
| DWORD old; | ||
| int failed = !VirtualProtect(memory, size, PAGE_READONLY, &old); | ||
| #else | ||
| int failed = mprotect(memory, size, PROT_READ); | ||
| #endif | ||
| if (failed) { | ||
| warn("unable to protect readable memory"); | ||
| return -1; | ||
| } | ||
| return 0; | ||
| } | ||
| static void | ||
| patch(char *base, const Hole *hole, uint64_t *patches) | ||
brandtbucher marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| { | ||
| @@ -145,16 +123,11 @@ patch(char *base, const Hole *hole, uint64_t *patches) | ||
| return; | ||
| case HoleKind_IMAGE_REL_AMD64_REL32: | ||
| case HoleKind_IMAGE_REL_I386_REL32: | ||
| case HoleKind_X86_64_RELOC_BRANCH: | ||
| case HoleKind_X86_64_RELOC_GOT: | ||
| case HoleKind_X86_64_RELOC_GOT_LOAD: | ||
| value -= (uint64_t)location; | ||
| assert(FITS_IN_BITS(value, 32)); | ||
| *addr = (uint32_t)value; | ||
| return; | ||
| case HoleKind_ARM64_RELOC_UNSIGNED: | ||
| @@ -174,14 +147,11 @@ patch(char *base, const Hole *hole, uint64_t *patches) | ||
| return; | ||
| case HoleKind_R_AARCH64_CALL26: | ||
| case HoleKind_R_AARCH64_JUMP26: | ||
| value -= (uint64_t)location; | ||
| assert(((*addr & 0xFC000000) == 0x14000000) || | ||
| ((*addr & 0xFC000000) == 0x94000000)); | ||
| assert((value & 0x3) == 0); | ||
| assert(FITS_IN_BITS(value, 28)); | ||
| *addr = (*addr & 0xFC000000) | ((uint32_t)(value >> 2) & 0x03FFFFFF); | ||
| return; | ||
| case HoleKind_ARM64_RELOC_GOT_LOAD_PAGEOFF12: | ||
| @@ -236,101 +206,59 @@ emit(const StencilGroup *stencil_group, uint64_t patches[]) | ||
| copy_and_patch(text, &stencil_group->text, patches); | ||
| } | ||
| _PyJITFunction | ||
| _PyJIT_CompileTrace(_PyUOpExecutorObject *executor, _PyUOpInstruction *trace, int size) | ||
| { | ||
| uint64_t text_size = wrapper_stencil_group.text.body_size; | ||
| uint64_t data_size = wrapper_stencil_group.data.body_size; | ||
| for (int i = 0; i < size; i++) { | ||
| _PyUOpInstruction *instruction = &trace[i]; | ||
| const StencilGroup *stencil_group = &stencil_groups[instruction->opcode]; | ||
| text_size += stencil_group->text.body_size; | ||
| data_size += stencil_group->data.body_size; | ||
| } | ||
| uint64_t page_size = get_page_size(); | ||
| assert((page_size & (page_size - 1)) == 0); | ||
| text_size += page_size - (text_size & (page_size - 1)); | ||
| data_size += page_size - (data_size & (page_size - 1)); | ||
| char *memory = alloc(text_size + data_size); | ||
| if (memory == NULL) { | ||
| return NULL; | ||
| } | ||
| char *text = memory; | ||
| char *data = memory + text_size; | ||
| const StencilGroup *stencil_group = &wrapper_stencil_group; | ||
| uint64_t patches[] = GET_PATCHES(); | ||
| patches[HoleValue_CONTINUE] = (uint64_t)text + stencil_group->text.body_size; | ||
| patches[HoleValue_DATA] = (uint64_t)data; | ||
| patches[HoleValue_TEXT] = (uint64_t)text; | ||
| patches[HoleValue_ZERO] = 0; | ||
| emit(stencil_group, patches); | ||
| text += stencil_group->text.body_size; | ||
| data += stencil_group->data.body_size; | ||
| for (int i = 0; i < size; i++) { | ||
| _PyUOpInstruction *instruction = &trace[i]; | ||
| const StencilGroup *stencil_group = &stencil_groups[instruction->opcode]; | ||
| uint64_t patches[] = GET_PATCHES(); | ||
| patches[HoleValue_CONTINUE] = (uint64_t)text + stencil_group->text.body_size; | ||
| patches[HoleValue_CURRENT_EXECUTOR] = (uint64_t)executor; | ||
| patches[HoleValue_OPARG] = instruction->oparg; | ||
| patches[HoleValue_OPERAND] = instruction->operand; | ||
| patches[HoleValue_TARGET] = instruction->target; | ||
| patches[HoleValue_DATA] = (uint64_t)data; | ||
| patches[HoleValue_TEXT] = (uint64_t)text; | ||
| patches[HoleValue_TOP] = (uint64_t)memory + wrapper_stencil_group.text.body_size; | ||
| patches[HoleValue_ZERO] = 0; | ||
| emit(stencil_group, patches); | ||
| text += stencil_group->text.body_size; | ||
| data += stencil_group->data.body_size; | ||
| } | ||
| if (mark_executable(memory, text_size) || | ||
| mark_readable(memory + text_size, data_size)) | ||
| { | ||
| return NULL; | ||
| } | ||
| return (_PyJITFunction)memory; | ||
| } | ||
| #endif | ||
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.