AMD64 RHEL8 LTO 3.x:https://buildbot.python.org/all/#/builders/312/builds/4FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)FAIL: test_NULL_ptr (test.test_gdb.PrettyPrintTests)FAIL: test_builtin_method (test.test_gdb.PrettyPrintTests)FAIL: test_builtins_help (test.test_gdb.PrettyPrintTests)FAIL: test_bytes (test.test_gdb.PrettyPrintTests)FAIL: test_corrupt_ob_type (test.test_gdb.PrettyPrintTests)FAIL: test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests)FAIL: test_corrupt_tp_name (test.test_gdb.PrettyPrintTests)FAIL: test_dicts (test.test_gdb.PrettyPrintTests)FAIL: test_exceptions (test.test_gdb.PrettyPrintTests)FAIL: test_frozensets (test.test_gdb.PrettyPrintTests)FAIL: test_int (test.test_gdb.PrettyPrintTests)FAIL: test_lists (test.test_gdb.PrettyPrintTests)FAIL: test_modern_class (test.test_gdb.PrettyPrintTests)FAIL: test_selfreferential_dict (test.test_gdb.PrettyPrintTests)FAIL: test_selfreferential_list (test.test_gdb.PrettyPrintTests)FAIL: test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests)FAIL: test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests)FAIL: test_sets (test.test_gdb.PrettyPrintTests)FAIL: test_singletons (test.test_gdb.PrettyPrintTests)FAIL: test_strings (test.test_gdb.PrettyPrintTests)FAIL: test_subclassing_list (test.test_gdb.PrettyPrintTests)FAIL: test_subclassing_tuple (test.test_gdb.PrettyPrintTests)FAIL: test_truncation (test.test_gdb.PrettyPrintTests)FAIL: test_tuples (test.test_gdb.PrettyPrintTests)Example of failure:======================================================================FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)Ensure that a PyObject* with NULL ob_type is handled gracefully----------------------------------------------------------------------Traceback (most recent call last):  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 515, in test_NULL_ob_type    self.assertSane('id(42)',  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 486, in assertSane    self.get_gdb_repr(source,  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 261, in get_gdb_repr    self.fail('Unexpected gdb output: %r\n%s' % (gdb_output, gdb_output))AssertionError: Unexpected gdb output: 'Breakpoint 1 at 0x565b60: fileObjects/longobject.c, line 1113.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, builtin_id (self=, v=42) atObjects/longobject.c:1113\n1113\tPyLong_FromVoidPtr(void *p)\n#0  builtin_id (self=, v=<unknown at remote 0x94e000>) atObjects/longobject.c:1113\n#1  cfunction_vectorcall_O (func=<built-in method id of module object at remote 0x7ffff7f6eea0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) atObjects/methodobject.c:442\n#2  _PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=<built-in method id of module object at remote 0x7ffff7f6eea0>) at ./Include/cpython/abstract.h:96\n#3  call_function (tstate=, pp_stack=, oparg=<optimized out>, kwnames=0x0) atPython/ceval.c:4984\n#4  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) atPython/ceval.c:3496\n#5  _PyEval_EvalCodeWithName (_co=<code at remote 0x7ffff7f13920>, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, args=, argcount=0, kwnames=, kwargs=, kwcount=0, kwstep=2, defs=, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) atPython/ceval.c:4296\n#6  PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=, kwcount=0, kws=, argcount=0, args=, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, _co=<code at remote 0x7ffff7f13920>) atPython/ceval.c:712\n#7  PyEval_EvalCode (locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, co=<code at remote 0x7ffff7f13920>) atPython/ceval.c:714\n#8  run_eval_code_obj (locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, co=) atPython/pythonrun.c:1117\n#9  run_mod (mod=<optimized out>, filename=\'<string>\', globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, flags=, arena=) atPython/pythonrun.c:1139\n#10 PyRun_StringFlags (flags=, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, start=257, str="id(42)\\n") atPython/pythonrun.c:1008\n#11 PyRun_SimpleStringFlags (command="id(42)\\n", flags=) atPython/pythonrun.c:460\n#12 pymain_run_command (cf=, command=<optimized out>) atModules/main.c:260\n#13 pymain_run_python (exitcode=) atModules/main.c:558\n#14 Py_RunMain () atModules/main.c:646\n#15 pymain_main (args=) atModules/main.c:676\n#16 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) atModules/main.c:700\n#17 __libc_start_main () from /lib64/libc.so.6\n#18 _start () atPython/ceval.c:5444\n'Breakpoint 1 at 0x565b60: fileObjects/longobject.c, line 1113.[Thread debugging using libthread_db enabled]Using host libthread_db library "/lib64/libthread_db.so.1".Breakpoint 1, builtin_id (self=, v=42) atObjects/longobject.c:11131113PyLong_FromVoidPtr(void *p)#0  builtin_id (self=, v=<unknown at remote 0x94e000>) atObjects/longobject.c:1113#1  cfunction_vectorcall_O (func=<built-in method id of module object at remote 0x7ffff7f6eea0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) atObjects/methodobject.c:442#2  _PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=<built-in method id of module object at remote 0x7ffff7f6eea0>) at ./Include/cpython/abstract.h:96#3  call_function (tstate=, pp_stack=, oparg=<optimized out>, kwnames=0x0) atPython/ceval.c:4984#4  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) atPython/ceval.c:3496#5  _PyEval_EvalCodeWithName (_co=<code at remote 0x7ffff7f13920>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, args=, argcount=0, kwnames=, kwargs=, kwcount=0, kwstep=2, defs=, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) atPython/ceval.c:4296#6  PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=, kwcount=0, kws=, argcount=0, args=, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, _co=<code at remote 0x7ffff7f13920>) atPython/ceval.c:712#7  PyEval_EvalCode (locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, co=<code at remote 0x7ffff7f13920>) atPython/ceval.c:714#8  run_eval_code_obj (locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, co=) atPython/pythonrun.c:1117#9  run_mod (mod=<optimized out>, filename='<string>', globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, flags=, arena=) atPython/pythonrun.c:1139#10 PyRun_StringFlags (flags=, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, start=257, str="id(42)\n") atPython/pythonrun.c:1008#11 PyRun_SimpleStringFlags (command="id(42)\n", flags=) atPython/pythonrun.c:460#12 pymain_run_command (cf=, command=<optimized out>) atModules/main.c:260#13 pymain_run_python (exitcode=) atModules/main.c:558#14 Py_RunMain () atModules/main.c:646#15 pymain_main (args=) atModules/main.c:676#16 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) atModules/main.c:700#17 __libc_start_main () from /lib64/libc.so.6#18 _start () atPython/ceval.c:5444

Similar errors on:AMD64 RHEL8 LTO + PGO 3.7:https://buildbot.python.org/all/#/builders/255/builds/13AMD64 RHEL8 LTO 3.7:https://buildbot.python.org/all/#/builders/293/builds/13AMD64 RHEL8 LTO 3.8:https://buildbot.python.org/all/#/builders/265/builds/12

Similar issue on AMD64 Fedora Stable LTO 3.8:https://buildbot.python.org/all/#/builders/235/builds/2

Shouldn't this test be suppressed on LTO builds?#0  builtin_id (self=, v=<unknown at remote 0x94e000>) atObjects/longobject.c:1113builtid_id isn't in longobject.c, so this is incorrect debug information. The test expects to find it in bltinmodule.c:re.match(r'.*#0\s+builtin_id\s+\(self\=.*,\s+v=\s*(.*?)?\)\s+at\s+\S*Python/bltinmodule.c.*' ...

AMD64 Fedora Stable LTO + PGO 3.7 is also affected:https://buildbot.python.org/all/#/builders/252/builds/2And AMD64 Fedora Stable LTO 3.7:https://buildbot.python.org/all/#/builders/270/builds/2

> builtid_id isn't in longobject.c, so this is incorrect debug information. The test expects to find it in bltinmodule.c:gdb may be confused by LTO. I'm not sure if how LTO works to debug symbols.> Shouldn't this test be suppressed on LTO builds?Maybe. I don't know.

Greg seems to have done LTO related things in the past, maybe he knows? (All my PGO contributions only apply to Windows :) )

The test probably has a fragile regex.  I don't spend time with gdb things much myself.

Um, okay then.The test already includes this:if ((sysconfig.get_config_var('PGO_PROF_USE_FLAG') or 'xxx') in    (sysconfig.get_config_var('PY_CORE_CFLAGS') or '')):    raise unittest.SkipTest("test_gdb is not reliable on PGO builds")Apparently this no longer works. Who understands these variables enough to fix the issue?

> Apparently this no longer works. Who understands these variables enough to fix the issue?Some failures are LTO + PGO builds, but test_gdb also fails on LTO without PGO builds.The PGO_PROF_USE_FLAG check only checks for PGO.

New changeset64b4a3a2deabcd4103fac2759a311fe94159b4d1 by Victor Stinner in branch 'master':bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)https://github.com/python/cpython/commit/64b4a3a2deabcd4103fac2759a311fe94159b4d1

Python 2.7 doesn't seem to be affected: I failed to reproduce the test_gdb failure on Python 2.7 with LTO.

New changesetc9893400652f38804aed0be8d8f70feda9465c47 by Miss Islington (bot) in branch '3.8':bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)https://github.com/python/cpython/commit/c9893400652f38804aed0be8d8f70feda9465c47

New changesete6b5ed1fbdbb7b44c7ab2a353fa0bc726b073740 by Victor Stinner in branch '3.7':bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422) (GH-16426)https://github.com/python/cpython/commit/e6b5ed1fbdbb7b44c7ab2a353fa0bc726b073740

AMD64 RHEL8 LTO 3.x is green again. I will keep the issue open until all mentionned buildbots are back to green (when test_gdb pass on these workers).

> AMD64 RHEL8 LTO 3.x is green again. I will keep the issue open until all mentionned buildbots are back to green (when test_gdb pass on these workers).I checked an all buildbots mentioned in this issues are back to green (success). I close the issue.> AMD64 RHEL8 LTO 3.x:>https://buildbot.python.org/all/#/builders/312/builds/4Green> AMD64 RHEL8 LTO + PGO 3.7:>https://buildbot.python.org/all/#/builders/255/builds/13Green> AMD64 RHEL8 LTO 3.7:>https://buildbot.python.org/all/#/builders/293/builds/13Green> AMD64 RHEL8 LTO 3.8:>https://buildbot.python.org/all/#/builders/265/builds/12Green> Similar issue on AMD64 Fedora Stable LTO 3.8:>https://buildbot.python.org/all/#/builders/235/builds/2Green> AMD64 Fedora Stable LTO + PGO 3.7 is also affected:>https://buildbot.python.org/all/#/builders/252/builds/2Green> And AMD64 Fedora Stable LTO 3.7:>https://buildbot.python.org/all/#/builders/270/builds/2Green