- Notifications
You must be signed in to change notification settings - Fork328
fix pgml conflicts with plpython#931
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Merged
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
pgml is not compatible with plpython, if using both pgml and plpython in thesame session, postgresql will crash.minimum reproducible code:```sqlSELECT pgml.embed('intfloat/e5-small', 'hi mom');create or replace function pyudf()returns int as$$return 0$$ language 'plpython3u';```the call stack:``` Stack trace of thread 161970: #0 0x00007efc1429edb8 PyImport_Import (libpython3.9.so.1.0 + 0x9edb8)postgresml#1 0x00007efc1429f125 PyImport_ImportModule (libpython3.9.so.1.0 + 0x9f125)postgresml#2 0x00007efb04b0f496 n/a (plpython3.so + 0x10496)postgresml#3 0x00007efb04b1039d plpython3_validator (plpython3.so + 0x1139d)postgresml#4 0x0000559d0cdbc5c2 OidFunctionCall1Coll (postgres + 0x6465c2)postgresml#5 0x0000559d0c9d68bb ProcedureCreate (postgres + 0x2608bb)postgresml#6 0x0000559d0ca5030c CreateFunction (postgres + 0x2da30c)postgresml#7 0x0000559d0ce1c730 n/a (postgres + 0x6a6730)postgresml#8 0x0000559d0cc5a030 standard_ProcessUtility (postgres + 0x4e4030)postgresml#9 0x0000559d0cc545ed n/a (postgres + 0x4de5ed)postgresml#10 0x0000559d0cc546e7 n/a (postgres + 0x4de6e7)postgresml#11 0x0000559d0cc54beb PortalRun (postgres + 0x4debeb)postgresml#12 0x0000559d0cc55249 n/a (postgres + 0x4df249)postgresml#13 0x0000559d0cc576f0 PostgresMain (postgres + 0x4e16f0)postgresml#14 0x0000559d0cbc3e9c n/a (postgres + 0x44de9c)postgresml#15 0x0000559d0cbc50aa PostmasterMain (postgres + 0x44f0aa)postgresml#16 0x0000559d0c8ce7d2 main (postgres + 0x1587d2)postgresml#17 0x00007efc18427cd0 n/a (libc.so.6 + 0x27cd0)postgresml#18 0x00007efc18427d8a __libc_start_main (libc.so.6 + 0x27d8a)postgresml#19 0x0000559d0c8cee15 _start (postgres + 0x158e15)```this is because PostgreSQL is using dlopen(RTLD_GLOBAL). this will parse someof symbols into the previous opened .so file, but the others will use arelative offset in pgml.so, and will cause a null-pointer crash.this commit hide all symbols except the UDF symbols (ends with `_wrapper`) andthe magic symbols (`_PG_init` `Pg_magic_func`). so dlopen(RTLD_GLOBAL) willparse the symbols to the correct position.for the ld change: gold and mold only support one version script. only lld support the version script append mode. rustup will distribute ldd, I think that will not be a problem. but need some documents and CI changes. |
I get this error when trying to compile (Ubuntu 22.04): |
ContributorAuthor
Sasasu commentedAug 20, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
need install lld. the current setup steps need to install |
ContributorAuthor
Sasasu commentedAug 23, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
If anyone wants to reproduce this try Python 3.9 with glibc ld the root cause is because those in-function static variable init function is a weak function. and this weak function does not execute correctly. https://github.com/python/cpython/blob/3.9/Python/import.c#L2019-L2029 but for other python versions, there should have another problem. |
Finally had a moment to test this on my end. Seems to work. In my original attempt, I confused We just need to make sure to include this linker in our CI & our docs: |
I can not veriy the CI. but I think the modification is suitable.
|
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Labels
None yet
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading.Please reload this page.
pgml is not compatible with plpython, if using both pgml and plpython in the same session, postgresql will crash.
minimum reproducible code:
the call stack:
this is because PostgreSQL is using dlopen(RTLD_GLOBAL). this will parse some of symbols into the previous opened .so file, but the others will use a relative offset in plpython.so, and will cause a null-pointer crash.
this commit hide all symbols except the UDF symbols (ends with
_wrapper) and the magic symbols (_PG_initPg_magic_func). so dlopen(RTLD_GLOBAL) will parse the symbols to the correct position.