Rate this Page
★★★★★
Skipped Functions#
Created On: Jul 28, 2025 | Last Updated On: Jul 29, 2025
Summary:
Sometimes,
torch.compilecompletely gives up compiling a function and runs it eagerly instead,resulting in potentially lost optimization opportunities.There are ways to work around skipped functions in order to re-enable tracing around the problematic code.
Sometimes,torch.compile withfullgraph=False is unable to resume tracing when encountering a graph breakor other compiler error. In many of these cases,torch.compile will skip compiling the function entirely and run it eagerly.
Note that the skip is only applied to the current function and NOT any nested function calls.torch.compile will still attempt to compile nested calls.
definner1(x):returnx+1definner2(x):returnx+2@torch.compiledeffn(x):x=inner1(x)torch._dynamo.skip_frame()x=inner2(x)fn(torch.randn(3))
ChromiumEventLogger initialized with id 23519bca-29f9-476e-84a4-2aecfc50de2dtorchdynamo start compiling fn /tmp/ipykernel_532/2126697152.py:5, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2126697152.py", line 10, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/2126697152.py:5create_envTRACE starts_line /tmp/ipykernel_532/2126697152.py:7 in fn x = inner1(x)TRACE LOAD_GLOBAL inner1 []TRACE LOAD_FAST x [LazyVariableTracker()]TRACE CALL_FUNCTION 1 [LazyVariableTracker(), LazyVariableTracker()]INLINING <code object inner1 at 0x7fabe826a340, file "/tmp/ipykernel_532/2126697152.py", line 1>, inlined according trace_rules.lookup inlined by defaultwrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE starts_line /tmp/ipykernel_532/2126697152.py:2 in inner1 (inline depth: 1) return x + 1TRACE LOAD_FAST x []TRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE RETURN_VALUE None [TensorVariable()]DONE INLINING <code object inner1 at 0x7fabe826a340, file "/tmp/ipykernel_532/2126697152.py", line 1>TRACE STORE_FAST x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/2126697152.py:8 in fn torch._dynamo.skip_frame()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR skip_frame [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]Skipping frame Skip frame due to `torch._dynamo.skip_frame()`. Message: None fn /tmp/ipykernel_532/2126697152.py 5put_code_state: no cache key, skippingrun_gc_after_compile: running gctorchdynamo start compiling inner1 /tmp/ipykernel_532/2126697152.py:1, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2126697152.py", line 10, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing inner1 /tmp/ipykernel_532/2126697152.py:1create_envTRACE starts_line /tmp/ipykernel_532/2126697152.py:2 in inner1 return x + 1TRACE LOAD_FAST x []TRACE LOAD_CONST 1 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE RETURN_VALUE None [TensorVariable()]Step 1: torchdynamo done tracing inner1 (RETURN_VALUE)RETURN_VALUE triggered compileCOMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_532/2126697152.py, line 2 in inner1>], graph_break=False)TRACED GRAPH ===== __compiled_fn_2_9a47d8a0_0722_48b1_8b6d_8892f4ce7d34 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/2126697152.py:2 in inner1, code: return x + 1 add: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None return (add,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 1 # mp/ipykernel_532/2126697152.py:2 in inner1| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 1 # mp/ipykernel_532/2126697152.py:2 in inner1Guard eval latency = 72.12 usput_code_state: no cache key, skippingrun_gc_after_compile: running gcskipping: inner (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_compile.py)skipping: disable (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/decorators.py)skipping: innermost_fn (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)skipping: __init__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)skipping: __init__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)skipping: nothing (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)skipping: __call__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)skipping: _fn (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)skipping: skip_frame (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/decorators.py)torchdynamo start compiling inner2 /tmp/ipykernel_532/2126697152.py:3, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2126697152.py", line 10, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing inner2 /tmp/ipykernel_532/2126697152.py:3create_envTRACE starts_line /tmp/ipykernel_532/2126697152.py:4 in inner2 return x + 2TRACE LOAD_FAST x []TRACE LOAD_CONST 2 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 2)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE RETURN_VALUE None [TensorVariable()]Step 1: torchdynamo done tracing inner2 (RETURN_VALUE)RETURN_VALUE triggered compileCOMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_532/2126697152.py, line 4 in inner2>], graph_break=False)TRACED GRAPH ===== __compiled_fn_4_c777bb3e_3a95_481d_9fa6_aca3bc776228 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/2126697152.py:4 in inner2, code: return x + 2 add: "f32[3][1]cpu" = l_x_ + 2; l_x_ = None return (add,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 2 # mp/ipykernel_532/2126697152.py:4 in inner2| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 2 # mp/ipykernel_532/2126697152.py:4 in inner2Guard eval latency = 66.29 usput_code_state: no cache key, skippingrun_gc_after_compile: running gcskipping: remove (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/weak.py)skipping: __hash__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/weak.py)In the above example,torch.compile will tracefn (includinginner1) up until theskip_frame.Thenfn is skipped and run eagerly -inner1 andinner2 are compiled when they are called.
Skipping functions may result in lost optimization opportunities,so it is important to check if code you want compiled is being skipped, and if so, to work around the skip.
Graph Break in a Loop#
torch.compile cannot resume tracing if a graph break occurs in a loop:
@torch.compiledeffn(x):foriinrange(5):x=x+1ifi==3:torch._dynamo.graph_break()returnxfn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_532/2044822433.py:1, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2044822433.py", line 8, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/2044822433.py:1create_envTRACE starts_line /tmp/ipykernel_532/2044822433.py:3 in fn for i in range(5):TRACE LOAD_GLOBAL range []TRACE LOAD_CONST 5 [LazyVariableTracker()]TRACE CALL_FUNCTION 1 [LazyVariableTracker(), ConstantVariable(int: 5)]TRACE GET_ITER None [RangeVariable()]TRACE FOR_ITER 40 [RangeIteratorVariable()]TRACE STORE_FAST i [RangeIteratorVariable(), ConstantVariable(int: 0)]TRACE starts_line /tmp/ipykernel_532/2044822433.py:4 in fn x = x + 1TRACE LOAD_FAST x [RangeIteratorVariable()]TRACE LOAD_CONST 1 [RangeIteratorVariable(), LazyVariableTracker()]TRACE BINARY_ADD None [RangeIteratorVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [RangeIteratorVariable(), TensorVariable()]TRACE starts_line /tmp/ipykernel_532/2044822433.py:5 in fn if i == 3:TRACE LOAD_FAST i [RangeIteratorVariable()]TRACE LOAD_CONST 3 [RangeIteratorVariable(), ConstantVariable(int: 0)]TRACE COMPARE_OP == [RangeIteratorVariable(), ConstantVariable(int: 0), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 38 [RangeIteratorVariable(), ConstantVariable(bool: False)]TRACE starts_line /tmp/ipykernel_532/2044822433.py:6 in fn torch._dynamo.graph_break()TRACE JUMP_ABSOLUTE 8 [RangeIteratorVariable()]TRACE starts_line /tmp/ipykernel_532/2044822433.py:3 in fn for i in range(5):TRACE FOR_ITER 40 [RangeIteratorVariable()]TRACE STORE_FAST i [RangeIteratorVariable(), ConstantVariable(int: 1)]TRACE starts_line /tmp/ipykernel_532/2044822433.py:4 in fn x = x + 1TRACE LOAD_FAST x [RangeIteratorVariable()]TRACE LOAD_CONST 1 [RangeIteratorVariable(), TensorVariable()]TRACE BINARY_ADD None [RangeIteratorVariable(), TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_FAST x [RangeIteratorVariable(), TensorVariable()]TRACE starts_line /tmp/ipykernel_532/2044822433.py:5 in fn if i == 3:TRACE LOAD_FAST i [RangeIteratorVariable()]TRACE LOAD_CONST 3 [RangeIteratorVariable(), ConstantVariable(int: 1)]TRACE COMPARE_OP == [RangeIteratorVariable(), ConstantVariable(int: 1), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 38 [RangeIteratorVariable(), ConstantVariable(bool: False)]TRACE starts_line /tmp/ipykernel_532/2044822433.py:6 in fn torch._dynamo.graph_break()TRACE JUMP_ABSOLUTE 8 [RangeIteratorVariable()]TRACE starts_line /tmp/ipykernel_532/2044822433.py:3 in fn for i in range(5):TRACE FOR_ITER 40 [RangeIteratorVariable()]TRACE STORE_FAST i [RangeIteratorVariable(), ConstantVariable(int: 2)]TRACE starts_line /tmp/ipykernel_532/2044822433.py:4 in fn x = x + 1TRACE LOAD_FAST x [RangeIteratorVariable()]TRACE LOAD_CONST 1 [RangeIteratorVariable(), TensorVariable()]TRACE BINARY_ADD None [RangeIteratorVariable(), TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_FAST x [RangeIteratorVariable(), TensorVariable()]TRACE starts_line /tmp/ipykernel_532/2044822433.py:5 in fn if i == 3:TRACE LOAD_FAST i [RangeIteratorVariable()]TRACE LOAD_CONST 3 [RangeIteratorVariable(), ConstantVariable(int: 2)]TRACE COMPARE_OP == [RangeIteratorVariable(), ConstantVariable(int: 2), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 38 [RangeIteratorVariable(), ConstantVariable(bool: False)]TRACE starts_line /tmp/ipykernel_532/2044822433.py:6 in fn torch._dynamo.graph_break()TRACE JUMP_ABSOLUTE 8 [RangeIteratorVariable()]TRACE starts_line /tmp/ipykernel_532/2044822433.py:3 in fn for i in range(5):TRACE FOR_ITER 40 [RangeIteratorVariable()]TRACE STORE_FAST i [RangeIteratorVariable(), ConstantVariable(int: 3)]TRACE starts_line /tmp/ipykernel_532/2044822433.py:4 in fn x = x + 1TRACE LOAD_FAST x [RangeIteratorVariable()]TRACE LOAD_CONST 1 [RangeIteratorVariable(), TensorVariable()]TRACE BINARY_ADD None [RangeIteratorVariable(), TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_FAST x [RangeIteratorVariable(), TensorVariable()]TRACE starts_line /tmp/ipykernel_532/2044822433.py:5 in fn if i == 3:TRACE LOAD_FAST i [RangeIteratorVariable()]TRACE LOAD_CONST 3 [RangeIteratorVariable(), ConstantVariable(int: 3)]TRACE COMPARE_OP == [RangeIteratorVariable(), ConstantVariable(int: 3), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 38 [RangeIteratorVariable(), ConstantVariable(bool: True)]TRACE starts_line /tmp/ipykernel_532/2044822433.py:6 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch [RangeIteratorVariable()]TRACE LOAD_ATTR _dynamo [RangeIteratorVariable(), LazyVariableTracker()]TRACE LOAD_ATTR graph_break [RangeIteratorVariable(), LazyVariableTracker()]TRACE CALL_FUNCTION 0 [RangeIteratorVariable(), LazyVariableTracker()]Graph break in user code at /tmp/ipykernel_532/2044822433.py:6Graph Break Reason: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlUser code traceback: File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2044822433.py", line 8, in <module> fn(torch.randn(3)) File "/tmp/ipykernel_532/2044822433.py", line 6, in fn torch._dynamo.graph_break()Skipping frame because there is a graph break in a for/while loop<FrameSummary file /tmp/ipykernel_532/2044822433.py, line 6 in fn>Skipping frame Skipping frame because there is a graph break in a for/while loop<FrameSummary file /tmp/ipykernel_532/2044822433.py, line 6 in fn> fn /tmp/ipykernel_532/2044822433.py 1put_code_state: no cache key, skippingrun_gc_after_compile: running gcskipping: graph_break (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/decorators.py)tensor([4.9228, 6.1472, 2.6696])
In this example, we can avoid skipping by unrolling the loop:
@torch.compiledeffn(x):definner(i):nonlocalxx=x+1ifi==3:torch._dynamo.graph_break()inner(0)inner(1)inner(2)inner(3)inner(4)returnxfn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_532/617960493.py:1, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/617960493.py", line 14, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/617960493.py:1create_envTRACE starts_line /tmp/ipykernel_532/617960493.py:3 in fn def inner(i):TRACE LOAD_CLOSURE x []TRACE BUILD_TUPLE 1 [CellVariable()]TRACE LOAD_CONST <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3> [TupleVariable(length=1)]TRACE LOAD_CONST fn.<locals>.inner [TupleVariable(length=1), ConstantVariable(code: <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>)]TRACE MAKE_FUNCTION 8 [TupleVariable(length=1), ConstantVariable(code: <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>), ConstantVariable(str: 'fn.<locals>.inner')]TRACE STORE_FAST inner [NestedUserFunctionVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:8 in fn inner(0)TRACE LOAD_FAST inner []TRACE LOAD_CONST 0 [NestedUserFunctionVariable()]TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 0)]INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner (inline depth: 1) x = x + 1TRACE LOAD_DEREF x []wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner (inline depth: 1) if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [ConstantVariable(int: 0)]TRACE COMPARE_OP == [ConstantVariable(int: 0), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/617960493.py:9 in fn inner(1)TRACE LOAD_FAST inner []TRACE LOAD_CONST 1 [NestedUserFunctionVariable()]TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 1)]INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner (inline depth: 1) x = x + 1TRACE LOAD_DEREF x []TRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner (inline depth: 1) if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [ConstantVariable(int: 1)]TRACE COMPARE_OP == [ConstantVariable(int: 1), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/617960493.py:10 in fn inner(2)TRACE LOAD_FAST inner []TRACE LOAD_CONST 2 [NestedUserFunctionVariable()]TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 2)]INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner (inline depth: 1) x = x + 1TRACE LOAD_DEREF x []TRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner (inline depth: 1) if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [ConstantVariable(int: 2)]TRACE COMPARE_OP == [ConstantVariable(int: 2), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/617960493.py:11 in fn inner(3)TRACE LOAD_FAST inner []TRACE LOAD_CONST 3 [NestedUserFunctionVariable()]TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 3)]INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner (inline depth: 1) x = x + 1TRACE LOAD_DEREF x []TRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner (inline depth: 1) if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [ConstantVariable(int: 3)]TRACE COMPARE_OP == [ConstantVariable(int: 3), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: True)]TRACE starts_line /tmp/ipykernel_532/617960493.py:7 in inner (inline depth: 1) torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR graph_break [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]empty checkpointFAILED INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>Graph break in user code at /tmp/ipykernel_532/617960493.py:7Graph Break Reason: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlUser code traceback: File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/617960493.py", line 14, in <module> fn(torch.randn(3)) File "/tmp/ipykernel_532/617960493.py", line 11, in fn inner(3) File "/tmp/ipykernel_532/617960493.py", line 7, in inner torch._dynamo.graph_break()Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysisStep 1: torchdynamo start tracing fn /tmp/ipykernel_532/617960493.py:1create_envTRACE starts_line /tmp/ipykernel_532/617960493.py:3 in fn def inner(i):TRACE LOAD_CLOSURE x []TRACE BUILD_TUPLE 1 [CellVariable()]TRACE LOAD_CONST <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3> [TupleVariable(length=1)]TRACE LOAD_CONST fn.<locals>.inner [TupleVariable(length=1), ConstantVariable(code: <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>)]TRACE MAKE_FUNCTION 8 [TupleVariable(length=1), ConstantVariable(code: <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>), ConstantVariable(str: 'fn.<locals>.inner')]TRACE STORE_FAST inner [NestedUserFunctionVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:8 in fn inner(0)TRACE LOAD_FAST inner []TRACE LOAD_CONST 0 [NestedUserFunctionVariable()]TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 0)]INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner (inline depth: 1) x = x + 1TRACE LOAD_DEREF x []wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner (inline depth: 1) if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [ConstantVariable(int: 0)]TRACE COMPARE_OP == [ConstantVariable(int: 0), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/617960493.py:9 in fn inner(1)TRACE LOAD_FAST inner []TRACE LOAD_CONST 1 [NestedUserFunctionVariable()]TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 1)]INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner (inline depth: 1) x = x + 1TRACE LOAD_DEREF x []TRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner (inline depth: 1) if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [ConstantVariable(int: 1)]TRACE COMPARE_OP == [ConstantVariable(int: 1), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/617960493.py:10 in fn inner(2)TRACE LOAD_FAST inner []TRACE LOAD_CONST 2 [NestedUserFunctionVariable()]TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 2)]INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner (inline depth: 1) x = x + 1TRACE LOAD_DEREF x []TRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner (inline depth: 1) if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [ConstantVariable(int: 2)]TRACE COMPARE_OP == [ConstantVariable(int: 2), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/617960493.py:11 in fn inner(3)TRACE LOAD_FAST inner []TRACE LOAD_CONST 3 [NestedUserFunctionVariable()]TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 3)]COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_532/617960493.py, line 11 in fn>, <FrameSummary file /tmp/ipykernel_532/617960493.py, line 7 in inner>], graph_break=True)TRACED GRAPH ===== __compiled_fn_8_0472935d_d975_47e2_b34e_9d149161b952 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/617960493.py:5 in inner, code: x = x + 1 x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None # File: /tmp/ipykernel_532/617960493.py:5 in inner, code: x = x + 1 x_1: "f32[3][1]cpu" = x + 1; x = None # File: /tmp/ipykernel_532/617960493.py:5 in inner, code: x = x + 1 x_2: "f32[3][1]cpu" = x_1 + 1; x_1 = None return (x_2,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=2), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_532/617960493.py:5 in inner| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_532/617960493.py:5 in innerGuard eval latency = 53.83 usput_code_state: no cache key, skippingrun_gc_after_compile: running gcskipping: _create_nested_fn (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py)torchdynamo start compiling inner /tmp/ipykernel_532/617960493.py:3, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/617960493.py", line 14, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing inner /tmp/ipykernel_532/617960493.py:3create_envTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner x = x + 1TRACE LOAD_DEREF x []TRACE LOAD_CONST 1 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=False, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [LazyVariableTracker()]TRACE COMPARE_OP == [LazyVariableTracker(), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: True)]TRACE starts_line /tmp/ipykernel_532/617960493.py:7 in inner torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR graph_break [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]Graph break (user stack suppressed due to duplicate graph break) in user code at /tmp/ipykernel_532/617960493.py:7Graph Break Reason: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlRestarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysisStep 1: torchdynamo start tracing inner /tmp/ipykernel_532/617960493.py:3create_envTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner x = x + 1TRACE LOAD_DEREF x []TRACE LOAD_CONST 1 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=False, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [LazyVariableTracker()]TRACE COMPARE_OP == [LazyVariableTracker(), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: True)]TRACE starts_line /tmp/ipykernel_532/617960493.py:7 in inner torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR graph_break [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_532/617960493.py, line 7 in inner>], graph_break=True)TRACED GRAPH ===== __compiled_fn_12_c9702b2e_a86b_41a2_9a4d_e0823204f4ae ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/617960493.py:5 in inner, code: x = x + 1 x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None return (x,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['i'], accessed_by=FrameLocalsGuardAccessor(key='i', framelocals_idx=0), type=<class 'int'>, tag_safe=(True, False)| | +- EQUALS_MATCH: L['i'] == 3 # if i == 3: # mp/ipykernel_532/617960493.py:6 in inner| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=1), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_532/617960493.py:5 in inner| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_532/617960493.py:5 in inner| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(False, False)| | +- GuardManager: source=G['torch'], accessed_by=DictGetItemGuardAccessor('torch'), type=<class 'module'>, tag_safe=(False, False)| | | +- ID_MATCH: ___check_obj_id(G['torch'], 140376310990704) # torch._dynamo.graph_break() # mp/ipykernel_532/617960493.py:7 in inner| | | +- GuardManager: source=G['torch']._dynamo, accessed_by=GetAttrGuardAccessor(_dynamo), type=<class 'module'>, tag_safe=(False, False)| | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo, 140374889071936) # torch._dynamo.graph_break() # mp/ipykernel_532/617960493.py:7 in inner| | | | +- GuardManager: source=G['torch']._dynamo.graph_break, accessed_by=GetAttrGuardAccessor(graph_break), type=<class 'function'>, tag_safe=(True, False)| | | | | +- GuardManager: source=G['torch']._dynamo.graph_break.__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)| | | | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo.graph_break.__code__, 140374703207360) # torch._dynamo.graph_break() # mp/ipykernel_532/617960493.py:7 in innerGuard eval latency = 368.80 usput_code_state: no cache key, skippingrun_gc_after_compile: running gctorchdynamo start compiling torch_dynamo_resume_in_inner_at_7 /tmp/ipykernel_532/617960493.py:7, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/617960493.py", line 14, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs) File "/tmp/ipykernel_532/617960493.py", line 11, in fn inner(3)Step 1: torchdynamo start tracing torch_dynamo_resume_in_inner_at_7 /tmp/ipykernel_532/617960493.py:7create_envTRACE starts_line /tmp/ipykernel_532/617960493.py:7 in torch_dynamo_resume_in_inner_at_7 torch._dynamo.graph_break()TRACE LOAD_CONST True []TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]TRACE LOAD_FAST ___stack0 []TRACE LOAD_CONST False [LazyVariableTracker()]TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]TRACE JUMP_ABSOLUTE 36 [LazyVariableTracker()]TRACE POP_TOP None [LazyVariableTracker()]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]Skipping frame because no content in function call torch_dynamo_resume_in_inner_at_7 /tmp/ipykernel_532/617960493.py 7put_code_state: no cache key, skippingrun_gc_after_compile: running gctorchdynamo start compiling torch_dynamo_resume_in_fn_at_11 /tmp/ipykernel_532/617960493.py:11, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/617960493.py", line 14, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_11 /tmp/ipykernel_532/617960493.py:11create_envTRACE starts_line /tmp/ipykernel_532/617960493.py:11 in torch_dynamo_resume_in_fn_at_11 inner(3)TRACE LOAD_CONST True []TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]TRACE LOAD_FAST ___stack0 []TRACE LOAD_CONST False [LazyVariableTracker()]TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]TRACE JUMP_ABSOLUTE 54 [LazyVariableTracker()]TRACE POP_TOP None [LazyVariableTracker()]TRACE starts_line /tmp/ipykernel_532/617960493.py:12 in torch_dynamo_resume_in_fn_at_11 inner(4)TRACE LOAD_FAST inner []TRACE LOAD_CONST 4 [LazyVariableTracker()]TRACE CALL_FUNCTION 1 [LazyVariableTracker(), ConstantVariable(int: 4)]INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/617960493.py:5 in inner (inline depth: 1) x = x + 1TRACE LOAD_DEREF x []TRACE LOAD_CONST 1 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=False, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_DEREF x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/617960493.py:6 in inner (inline depth: 1) if i == 3:TRACE LOAD_FAST i []TRACE LOAD_CONST 3 [ConstantVariable(int: 4)]TRACE COMPARE_OP == [ConstantVariable(int: 4), ConstantVariable(int: 3)]TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object inner at 0x7fab86007aa0, file "/tmp/ipykernel_532/617960493.py", line 3>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/617960493.py:13 in torch_dynamo_resume_in_fn_at_11 return xTRACE LOAD_DEREF x []TRACE RETURN_VALUE None [TensorVariable()]Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_11 (RETURN_VALUE)RETURN_VALUE triggered compileCOMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_532/617960493.py, line 13 in torch_dynamo_resume_in_fn_at_11>], graph_break=False)TRACED GRAPH ===== __compiled_fn_16_240112e8_d62a_444c_a9ef_5bde0a078bbd ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/617960493.py:5 in inner, code: x = x + 1 x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None return (x,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=6), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_532/617960493.py:5 in inner| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_532/617960493.py:5 in inner| +- GuardManager: source=L['inner'], accessed_by=FrameLocalsGuardAccessor(key='inner', framelocals_idx=3), type=<class 'function'>, tag_safe=(True, False)| | +- GuardManager: source=L['inner'].__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)| | | +- ID_MATCH: ___check_obj_id(L['inner'].__code__, 140374664313504) # inner(4) # mp/ipykernel_532/617960493.py:12 in torch_dynamo_resume_in_fn_at_11Guard eval latency = 1147.51 usput_code_state: no cache key, skippingrun_gc_after_compile: running gcskipping because no torch.* remove /opt/conda/envs/py_3.10/lib/python3.10/weakref.py 370tensor([4.3159, 5.4236, 5.5645])
In general, resolving the graph break causing the skip will also resolve the skip.
Graph Break in a Context Manager#
Another common example of an unresumable graph break is a graph break in most context managers:
classCustomCtxManager:def__enter__(self):passdef__exit__(self,exc_type,exc_value,traceback):pass@torch.compiledeffn(x):withCustomCtxManager():x=x+1torch._dynamo.graph_break()returnx+1fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_532/4148913404.py:6, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/4148913404.py", line 12, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/4148913404.py:6create_envTRACE starts_line /tmp/ipykernel_532/4148913404.py:8 in fn with CustomCtxManager():TRACE LOAD_GLOBAL CustomCtxManager []TRACE CALL_FUNCTION 0 [LazyVariableTracker()]TRACE SETUP_WITH 48 [GenericContextWrappingVariable(CustomCtxManager)]INLINING <code object __enter__ at 0x7fabee5323f0, file "/tmp/ipykernel_532/4148913404.py", line 2>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/4148913404.py:3 in __enter__ (inline depth: 1) passTRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object __enter__ at 0x7fabee5323f0, file "/tmp/ipykernel_532/4148913404.py", line 2>TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/4148913404.py:9 in fn x = x + 1TRACE LOAD_FAST x [WithExitFunctionVariable()]TRACE LOAD_CONST 1 [WithExitFunctionVariable(), LazyVariableTracker()]TRACE BINARY_ADD None [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [WithExitFunctionVariable(), TensorVariable()]TRACE starts_line /tmp/ipykernel_532/4148913404.py:10 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]TRACE LOAD_ATTR _dynamo [WithExitFunctionVariable(), LazyVariableTracker()]TRACE LOAD_ATTR graph_break [WithExitFunctionVariable(), LazyVariableTracker()]TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), LazyVariableTracker()]empty checkpointrun_gc_after_compile: running gcGraph break: skip: from user code at: File "/tmp/ipykernel_532/4148913404.py", line 10, in fn torch._dynamo.graph_break()Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2333, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward return getattr(self.realize(), name)(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2 raise Unsupported(msg)torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlThe above exception was the direct cause of the following exception:Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1625, in __call__ result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__ result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1434, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner return _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 910, in wrapper unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 595, in unimplemented_v2 raise Unsupported(msg) from from_exctorch._dynamo.exc.Unsupported: Graph break under GenericContextWrappingVariable Explanation: Attempted to graph break in an active context manager(s) that doesn't support graph breaking. Hint: Move the offending context manager(s) to outside the compiled region. Hint: This graph break may have been caused by an earlier graph break. Resolving the earlier graph break may resolve this one. Developer debug context: Active generic context managers: [GenericContextWrappingVariable(CustomCtxManager)] For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0066.htmlfrom user code: File "/tmp/ipykernel_532/4148913404.py", line 10, in fn torch._dynamo.graph_break()Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"WON'T CONVERT fn /tmp/ipykernel_532/4148913404.py line 6 due to: Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2333, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward return getattr(self.realize(), name)(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2 raise Unsupported(msg)torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlThe above exception was the direct cause of the following exception:Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1625, in __call__ result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__ result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1434, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner return _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 910, in wrapper unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 595, in unimplemented_v2 raise Unsupported(msg) from from_exctorch._dynamo.exc.Unsupported: Graph break under GenericContextWrappingVariable Explanation: Attempted to graph break in an active context manager(s) that doesn't support graph breaking. Hint: Move the offending context manager(s) to outside the compiled region. Hint: This graph break may have been caused by an earlier graph break. Resolving the earlier graph break may resolve this one. Developer debug context: Active generic context managers: [GenericContextWrappingVariable(CustomCtxManager)] For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0066.htmlfrom user code: File "/tmp/ipykernel_532/4148913404.py", line 10, in fn torch._dynamo.graph_break()Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2333, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward return getattr(self.realize(), name)(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2 raise Unsupported(msg)torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlThe above exception was the direct cause of the following exception:Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1625, in __call__ result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__ result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1434, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner return _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 910, in wrapper unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 595, in unimplemented_v2 raise Unsupported(msg) from from_exctorch._dynamo.exc.Unsupported: Graph break under GenericContextWrappingVariable Explanation: Attempted to graph break in an active context manager(s) that doesn't support graph breaking. Hint: Move the offending context manager(s) to outside the compiled region. Hint: This graph break may have been caused by an earlier graph break. Resolving the earlier graph break may resolve this one. Developer debug context: Active generic context managers: [GenericContextWrappingVariable(CustomCtxManager)] For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0066.htmlfrom user code: File "/tmp/ipykernel_532/4148913404.py", line 10, in fn torch._dynamo.graph_break()Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"skipping because no torch.* __enter__ /tmp/ipykernel_532/4148913404.py 2skipping because no torch.* __exit__ /tmp/ipykernel_532/4148913404.py 4tensor([2.9449, 2.7372, 1.0708])
We can avoid skipping by moving the graph break outside of the context manager:
@torch.compiledeffn(x):withCustomCtxManager():x=x+1torch._dynamo.graph_break()withCustomCtxManager():returnx+1fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_532/2124425154.py:1, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2124425154.py", line 8, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/2124425154.py:1create_envTRACE starts_line /tmp/ipykernel_532/2124425154.py:3 in fn with CustomCtxManager():TRACE LOAD_GLOBAL CustomCtxManager []TRACE CALL_FUNCTION 0 [LazyVariableTracker()]TRACE SETUP_WITH 30 [GenericContextWrappingVariable(CustomCtxManager)]INLINING <code object __enter__ at 0x7fabee5323f0, file "/tmp/ipykernel_532/4148913404.py", line 2>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/4148913404.py:3 in __enter__ (inline depth: 1) passTRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object __enter__ at 0x7fabee5323f0, file "/tmp/ipykernel_532/4148913404.py", line 2>TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/2124425154.py:4 in fn x = x + 1TRACE LOAD_FAST x [WithExitFunctionVariable()]TRACE LOAD_CONST 1 [WithExitFunctionVariable(), LazyVariableTracker()]TRACE BINARY_ADD None [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [WithExitFunctionVariable(), TensorVariable()]TRACE POP_BLOCK None [WithExitFunctionVariable()]TRACE starts_line /tmp/ipykernel_532/2124425154.py:3 in fn with CustomCtxManager():TRACE LOAD_CONST None [WithExitFunctionVariable()]TRACE DUP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE DUP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]TRACE CALL_FUNCTION 3 [WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]INLINING <code object __exit__ at 0x7fabe8268c90, file "/tmp/ipykernel_532/4148913404.py", line 4>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/4148913404.py:5 in __exit__ (inline depth: 1) passTRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object __exit__ at 0x7fabe8268c90, file "/tmp/ipykernel_532/4148913404.py", line 4>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE JUMP_FORWARD 46 []TRACE starts_line /tmp/ipykernel_532/2124425154.py:5 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR graph_break [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]Graph break in user code at /tmp/ipykernel_532/2124425154.py:5Graph Break Reason: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlUser code traceback: File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2124425154.py", line 8, in <module> fn(torch.randn(3)) File "/tmp/ipykernel_532/2124425154.py", line 5, in fn torch._dynamo.graph_break()Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysisStep 1: torchdynamo start tracing fn /tmp/ipykernel_532/2124425154.py:1create_envTRACE starts_line /tmp/ipykernel_532/2124425154.py:3 in fn with CustomCtxManager():TRACE LOAD_GLOBAL CustomCtxManager []TRACE CALL_FUNCTION 0 [LazyVariableTracker()]TRACE SETUP_WITH 30 [GenericContextWrappingVariable(CustomCtxManager)]INLINING <code object __enter__ at 0x7fabee5323f0, file "/tmp/ipykernel_532/4148913404.py", line 2>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/4148913404.py:3 in __enter__ (inline depth: 1) passTRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object __enter__ at 0x7fabee5323f0, file "/tmp/ipykernel_532/4148913404.py", line 2>TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/2124425154.py:4 in fn x = x + 1TRACE LOAD_FAST x [WithExitFunctionVariable()]TRACE LOAD_CONST 1 [WithExitFunctionVariable(), LazyVariableTracker()]TRACE BINARY_ADD None [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [WithExitFunctionVariable(), TensorVariable()]TRACE POP_BLOCK None [WithExitFunctionVariable()]TRACE starts_line /tmp/ipykernel_532/2124425154.py:3 in fn with CustomCtxManager():TRACE LOAD_CONST None [WithExitFunctionVariable()]TRACE DUP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE DUP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]TRACE CALL_FUNCTION 3 [WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]INLINING <code object __exit__ at 0x7fabe8268c90, file "/tmp/ipykernel_532/4148913404.py", line 4>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/4148913404.py:5 in __exit__ (inline depth: 1) passTRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object __exit__ at 0x7fabe8268c90, file "/tmp/ipykernel_532/4148913404.py", line 4>TRACE POP_TOP None [ConstantVariable(NoneType: None)]TRACE JUMP_FORWARD 46 []TRACE starts_line /tmp/ipykernel_532/2124425154.py:5 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR graph_break [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_532/2124425154.py, line 5 in fn>], graph_break=True)TRACED GRAPH ===== __compiled_fn_20_763153f3_7654_433a_8e55_bfa1b1276b54 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/2124425154.py:4 in fn, code: x = x + 1 x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None return (x,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_532/2124425154.py:4 in fn| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_532/2124425154.py:4 in fn| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(False, False)| | +- GuardManager: source=G['torch'], accessed_by=DictGetItemGuardAccessor('torch'), type=<class 'module'>, tag_safe=(False, False)| | | +- ID_MATCH: ___check_obj_id(G['torch'], 140376310990704) # torch._dynamo.graph_break() # mp/ipykernel_532/2124425154.py:5 in fn| | | +- GuardManager: source=G['torch']._dynamo, accessed_by=GetAttrGuardAccessor(_dynamo), type=<class 'module'>, tag_safe=(False, False)| | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo, 140374889071936) # torch._dynamo.graph_break() # mp/ipykernel_532/2124425154.py:5 in fn| | | | +- GuardManager: source=G['torch']._dynamo.graph_break, accessed_by=GetAttrGuardAccessor(graph_break), type=<class 'function'>, tag_safe=(True, False)| | | | | +- GuardManager: source=G['torch']._dynamo.graph_break.__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)| | | | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo.graph_break.__code__, 140374703207360) # torch._dynamo.graph_break() # mp/ipykernel_532/2124425154.py:5 in fn| | +- GuardManager: source=G['CustomCtxManager'], accessed_by=DictGetItemGuardAccessor('CustomCtxManager'), type=<class 'type'>, tag_safe=(True, False)| | | +- ID_MATCH: ___check_obj_id(G['CustomCtxManager'], 93956540677728) # with CustomCtxManager(): # mp/ipykernel_532/2124425154.py:3 in fnGuard eval latency = 48.80 usput_code_state: no cache key, skippingrun_gc_after_compile: running gctorchdynamo start compiling torch_dynamo_resume_in_fn_at_5 /tmp/ipykernel_532/2124425154.py:5, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2124425154.py", line 8, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_5 /tmp/ipykernel_532/2124425154.py:5create_envTRACE starts_line /tmp/ipykernel_532/2124425154.py:5 in torch_dynamo_resume_in_fn_at_5 torch._dynamo.graph_break()TRACE LOAD_CONST True []TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]TRACE LOAD_FAST ___stack0 []TRACE LOAD_CONST False [LazyVariableTracker()]TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]TRACE JUMP_ABSOLUTE 66 [LazyVariableTracker()]TRACE POP_TOP None [LazyVariableTracker()]TRACE starts_line /tmp/ipykernel_532/2124425154.py:6 in torch_dynamo_resume_in_fn_at_5 with CustomCtxManager():TRACE LOAD_GLOBAL CustomCtxManager []TRACE CALL_FUNCTION 0 [LazyVariableTracker()]TRACE SETUP_WITH 98 [GenericContextWrappingVariable(CustomCtxManager)]INLINING <code object __enter__ at 0x7fabee5323f0, file "/tmp/ipykernel_532/4148913404.py", line 2>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/4148913404.py:3 in __enter__ (inline depth: 1) passTRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object __enter__ at 0x7fabee5323f0, file "/tmp/ipykernel_532/4148913404.py", line 2>TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/2124425154.py:7 in torch_dynamo_resume_in_fn_at_5 return x + 1TRACE LOAD_FAST x [WithExitFunctionVariable()]TRACE LOAD_CONST 1 [WithExitFunctionVariable(), LazyVariableTracker()]TRACE BINARY_ADD None [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE starts_line /tmp/ipykernel_532/2124425154.py:6 in torch_dynamo_resume_in_fn_at_5 with CustomCtxManager():TRACE POP_BLOCK None [WithExitFunctionVariable(), TensorVariable()]TRACE ROT_TWO None [WithExitFunctionVariable(), TensorVariable()]TRACE LOAD_CONST None [TensorVariable(), WithExitFunctionVariable()]TRACE DUP_TOP None [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE DUP_TOP None [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]TRACE CALL_FUNCTION 3 [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]INLINING <code object __exit__ at 0x7fabe8268c90, file "/tmp/ipykernel_532/4148913404.py", line 4>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/4148913404.py:5 in __exit__ (inline depth: 1) passTRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]DONE INLINING <code object __exit__ at 0x7fabe8268c90, file "/tmp/ipykernel_532/4148913404.py", line 4>TRACE POP_TOP None [TensorVariable(), ConstantVariable(NoneType: None)]TRACE RETURN_VALUE None [TensorVariable()]Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_5 (RETURN_VALUE)RETURN_VALUE triggered compileCOMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_532/2124425154.py, line 6 in torch_dynamo_resume_in_fn_at_5>], graph_break=False)TRACED GRAPH ===== __compiled_fn_23_df525bd6_d853_4d00_8aac_8dccfb55e2ea ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/2124425154.py:7 in torch_dynamo_resume_in_fn_at_5, code: return x + 1 add: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None return (add,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=3), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 1 # mp/ipykernel_532/2124425154.py:7 in torch_dynamo_resume_in_fn_at_5| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 1 # mp/ipykernel_532/2124425154.py:7 in torch_dynamo_resume_in_fn_at_5| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(True, False)| | +- GuardManager: source=G['CustomCtxManager'], accessed_by=DictGetItemGuardAccessor('CustomCtxManager'), type=<class 'type'>, tag_safe=(True, False)| | | +- ID_MATCH: ___check_obj_id(G['CustomCtxManager'], 93956540677728) # with CustomCtxManager(): # mp/ipykernel_532/2124425154.py:6 in torch_dynamo_resume_in_fn_at_5Guard eval latency = 1087.49 usput_code_state: no cache key, skippingrun_gc_after_compile: running gctensor([1.5408, 1.1547, 1.4015])
There are some context managers where Dynamo can resume after a graph break.Some of these can be found insupported_ctx_manager_classes intorch/_dynamo/variables/torch.py.In general, any context manager represented by aContextWrappingVariable subclass intorch/_dynamo/variables/ctx_manager.py support resuming after a graph break. For example:
importcontextlib@torch.compiledeffn(x):withcontextlib.nullcontext():withtorch.no_grad():x=x+1torch._dynamo.graph_break()returnx+1fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_532/3152636365.py:2, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/3152636365.py", line 9, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/3152636365.py:2create_envTRACE starts_line /tmp/ipykernel_532/3152636365.py:4 in fn with contextlib.nullcontext():TRACE LOAD_GLOBAL contextlib []TRACE LOAD_ATTR nullcontext [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]TRACE SETUP_WITH 106 [NullContextVariable()]TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/3152636365.py:5 in fn with torch.no_grad():TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]TRACE LOAD_ATTR no_grad [WithExitFunctionVariable(), LazyVariableTracker()]TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), LazyVariableTracker()]TRACE SETUP_WITH 74 [WithExitFunctionVariable(), GradModeVariable()]TRACE POP_TOP None [WithExitFunctionVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/3152636365.py:6 in fn x = x + 1TRACE LOAD_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable()]TRACE LOAD_CONST 1 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE BINARY_ADD None [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable(), TensorVariable()]TRACE starts_line /tmp/ipykernel_532/3152636365.py:7 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch [WithExitFunctionVariable(), WithExitFunctionVariable()]TRACE LOAD_ATTR _dynamo [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE LOAD_ATTR graph_break [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]Graph break in user code at /tmp/ipykernel_532/3152636365.py:7Graph Break Reason: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlUser code traceback: File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/3152636365.py", line 9, in <module> fn(torch.randn(3)) File "/tmp/ipykernel_532/3152636365.py", line 7, in fn torch._dynamo.graph_break()Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysisStep 1: torchdynamo start tracing fn /tmp/ipykernel_532/3152636365.py:2create_envTRACE starts_line /tmp/ipykernel_532/3152636365.py:4 in fn with contextlib.nullcontext():TRACE LOAD_GLOBAL contextlib []TRACE LOAD_ATTR nullcontext [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]TRACE SETUP_WITH 106 [NullContextVariable()]TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/3152636365.py:5 in fn with torch.no_grad():TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]TRACE LOAD_ATTR no_grad [WithExitFunctionVariable(), LazyVariableTracker()]TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), LazyVariableTracker()]TRACE SETUP_WITH 74 [WithExitFunctionVariable(), GradModeVariable()]TRACE POP_TOP None [WithExitFunctionVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/3152636365.py:6 in fn x = x + 1TRACE LOAD_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable()]TRACE LOAD_CONST 1 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE BINARY_ADD None [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable(), TensorVariable()]TRACE starts_line /tmp/ipykernel_532/3152636365.py:7 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch [WithExitFunctionVariable(), WithExitFunctionVariable()]TRACE LOAD_ATTR _dynamo [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE LOAD_ATTR graph_break [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_532/3152636365.py, line 7 in fn>], graph_break=True)TRACED GRAPH ===== __compiled_fn_26_b931600f_47fa_4e1d_8c6b_321de5fcc179 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # No stacktrace found for following nodes _set_grad_enabled = torch._C._set_grad_enabled(False); _set_grad_enabled = None # File: /tmp/ipykernel_532/3152636365.py:6 in fn, code: x = x + 1 x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None # No stacktrace found for following nodes _set_grad_enabled_1 = torch._C._set_grad_enabled(True); _set_grad_enabled_1 = None return (x,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_532/3152636365.py:6 in fn| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_532/3152636365.py:6 in fn| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(False, False)| | +- GuardManager: source=G['torch'], accessed_by=DictGetItemGuardAccessor('torch'), type=<class 'module'>, tag_safe=(False, False)| | | +- ID_MATCH: ___check_obj_id(G['torch'], 140376310990704) # with torch.no_grad(): # mp/ipykernel_532/3152636365.py:5 in fn| | | +- GuardManager: source=G['torch']._dynamo, accessed_by=GetAttrGuardAccessor(_dynamo), type=<class 'module'>, tag_safe=(False, False)| | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo, 140374889071936) # torch._dynamo.graph_break() # mp/ipykernel_532/3152636365.py:7 in fn| | | | +- GuardManager: source=G['torch']._dynamo.graph_break, accessed_by=GetAttrGuardAccessor(graph_break), type=<class 'function'>, tag_safe=(True, False)| | | | | +- GuardManager: source=G['torch']._dynamo.graph_break.__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)| | | | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo.graph_break.__code__, 140374703207360) # torch._dynamo.graph_break() # mp/ipykernel_532/3152636365.py:7 in fn| | | +- GuardManager: source=G['torch'].no_grad, accessed_by=GetAttrGuardAccessor(no_grad), type=<class 'type'>, tag_safe=(True, False)| | | | +- ID_MATCH: ___check_obj_id(G['torch'].no_grad, 93956503623376) # with torch.no_grad(): # mp/ipykernel_532/3152636365.py:5 in fn| | +- GuardManager: source=G['contextlib'], accessed_by=DictGetItemGuardAccessor('contextlib'), type=<class 'module'>, tag_safe=(False, False)| | | +- ID_MATCH: ___check_obj_id(G['contextlib'], 140376423420832) # with contextlib.nullcontext(): # mp/ipykernel_532/3152636365.py:4 in fn| | | +- GuardManager: source=G['contextlib'].nullcontext, accessed_by=GetAttrGuardAccessor(nullcontext), type=<class 'abc.ABCMeta'>, tag_safe=(True, False)| | | | +- ID_MATCH: ___check_obj_id(G['contextlib'].nullcontext, 93956428443184) # with contextlib.nullcontext(): # mp/ipykernel_532/3152636365.py:4 in fnGuard eval latency = 261.20 usput_code_state: no cache key, skippingrun_gc_after_compile: running gcskipping because no torch.* __init__ /opt/conda/envs/py_3.10/lib/python3.10/contextlib.py 732skipping because no torch.* __enter__ /opt/conda/envs/py_3.10/lib/python3.10/contextlib.py 735skipping: __init__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/grad_mode.py)skipping: __enter__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/grad_mode.py)skipping: __exit__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/grad_mode.py)skipping because no torch.* __exit__ /opt/conda/envs/py_3.10/lib/python3.10/contextlib.py 738torchdynamo start compiling torch_dynamo_resume_in_fn_at_7 /tmp/ipykernel_532/3152636365.py:7, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/3152636365.py", line 9, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_7 /tmp/ipykernel_532/3152636365.py:7create_envTRACE starts_line /tmp/ipykernel_532/3152636365.py:7 in torch_dynamo_resume_in_fn_at_7 torch._dynamo.graph_break()TRACE LOAD_CONST True []TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]TRACE LOAD_FAST ___stack0 []TRACE CALL_FUNCTION 0 [LazyVariableTracker()]TRACE NOP None [NullContextVariable()]TRACE SETUP_WITH 96 [NullContextVariable()]TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE NOP None [WithExitFunctionVariable()]TRACE LOAD_FAST ___stack1 [WithExitFunctionVariable()]TRACE LOAD_CONST False [WithExitFunctionVariable(), LazyVariableTracker()]TRACE CALL_FUNCTION 1 [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(bool: False)]TRACE NOP None [WithExitFunctionVariable(), GradModeVariable()]TRACE SETUP_WITH 56 [WithExitFunctionVariable(), GradModeVariable()]TRACE POP_TOP None [WithExitFunctionVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE NOP None [WithExitFunctionVariable(), WithExitFunctionVariable()]TRACE LOAD_FAST ___stack2 [WithExitFunctionVariable(), WithExitFunctionVariable()]TRACE LOAD_CONST False [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE STORE_FAST __is_tracing_resume_prologue [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(bool: False)]TRACE JUMP_ABSOLUTE 158 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE POP_TOP None [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE starts_line /tmp/ipykernel_532/3152636365.py:8 in torch_dynamo_resume_in_fn_at_7 return x + 1TRACE LOAD_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable()]TRACE LOAD_CONST 1 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]TRACE BINARY_ADD None [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE starts_line /tmp/ipykernel_532/3152636365.py:5 in torch_dynamo_resume_in_fn_at_7 with torch.no_grad():TRACE POP_BLOCK None [WithExitFunctionVariable(), WithExitFunctionVariable(), TensorVariable()]TRACE ROT_TWO None [WithExitFunctionVariable(), WithExitFunctionVariable(), TensorVariable()]TRACE LOAD_CONST None [WithExitFunctionVariable(), TensorVariable(), WithExitFunctionVariable()]TRACE DUP_TOP None [WithExitFunctionVariable(), TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE DUP_TOP None [WithExitFunctionVariable(), TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]TRACE CALL_FUNCTION 3 [WithExitFunctionVariable(), TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]TRACE POP_TOP None [WithExitFunctionVariable(), TensorVariable(), ConstantVariable(NoneType: None)]TRACE starts_line /tmp/ipykernel_532/3152636365.py:4 in torch_dynamo_resume_in_fn_at_7 with contextlib.nullcontext():TRACE POP_BLOCK None [WithExitFunctionVariable(), TensorVariable()]TRACE ROT_TWO None [WithExitFunctionVariable(), TensorVariable()]TRACE LOAD_CONST None [TensorVariable(), WithExitFunctionVariable()]TRACE DUP_TOP None [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]TRACE DUP_TOP None [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]TRACE CALL_FUNCTION 3 [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]TRACE POP_TOP None [TensorVariable(), ConstantVariable(NoneType: None)]TRACE RETURN_VALUE None [TensorVariable()]Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_7 (RETURN_VALUE)RETURN_VALUE triggered compileCOMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_532/3152636365.py, line 4 in torch_dynamo_resume_in_fn_at_7>], graph_break=False)TRACED GRAPH ===== __compiled_fn_31_b2a5c450_5d0e_4895_be05_d0def17b67d7 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # No stacktrace found for following nodes _set_grad_enabled = torch._C._set_grad_enabled(False); _set_grad_enabled = None # File: /tmp/ipykernel_532/3152636365.py:8 in torch_dynamo_resume_in_fn_at_7, code: return x + 1 add: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None # No stacktrace found for following nodes _set_grad_enabled_1 = torch._C._set_grad_enabled(True); _set_grad_enabled_1 = None return (add,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=5), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 1 # mp/ipykernel_532/3152636365.py:8 in torch_dynamo_resume_in_fn_at_7| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 1 # mp/ipykernel_532/3152636365.py:8 in torch_dynamo_resume_in_fn_at_7| +- GuardManager: source=L['___stack0'], accessed_by=FrameLocalsGuardAccessor(key='___stack0', framelocals_idx=2), type=<class 'abc.ABCMeta'>, tag_safe=(True, False)| | +- ID_MATCH: ___check_obj_id(L['___stack0'], 93956428443184) # torch._dynamo.graph_break() # mp/ipykernel_532/3152636365.py:7 in torch_dynamo_resume_in_fn_at_7| +- GuardManager: source=L['___stack1'], accessed_by=FrameLocalsGuardAccessor(key='___stack1', framelocals_idx=3), type=<class 'type'>, tag_safe=(True, False)| | +- ID_MATCH: ___check_obj_id(L['___stack1'], 93956503628432) # torch._dynamo.graph_break() # mp/ipykernel_532/3152636365.py:7 in torch_dynamo_resume_in_fn_at_7Guard eval latency = 79.83 usput_code_state: no cache key, skippingrun_gc_after_compile: running gctensor([1.3265, 1.4369, 2.3154])
Graph Break in a Try Block#
A graph break in a try block cannot be resumed:
@torch.compiledeffn(x):try:x=x+1torch._dynamo.graph_break()returnx+1exceptExceptionase:passfn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_532/2546874632.py:1, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/2546874632.py", line 9, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/2546874632.py:1create_envTRACE starts_line /tmp/ipykernel_532/2546874632.py:3 in fn try:TRACE SETUP_FINALLY 30 []TRACE starts_line /tmp/ipykernel_532/2546874632.py:4 in fn x = x + 1TRACE LOAD_FAST x []TRACE LOAD_CONST 1 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/2546874632.py:5 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR graph_break [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]empty checkpointrun_gc_after_compile: running gcGraph break: skip: from user code at: File "/tmp/ipykernel_532/2546874632.py", line 5, in fn torch._dynamo.graph_break()Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1625, in __call__ result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__ result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1434, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner return _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2333, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward return getattr(self.realize(), name)(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2 raise Unsupported(msg)torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlfrom user code: File "/tmp/ipykernel_532/2546874632.py", line 5, in fn torch._dynamo.graph_break()Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"WON'T CONVERT fn /tmp/ipykernel_532/2546874632.py line 1 due to: Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1625, in __call__ result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__ result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1434, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner return _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2333, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward return getattr(self.realize(), name)(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2 raise Unsupported(msg)torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlfrom user code: File "/tmp/ipykernel_532/2546874632.py", line 5, in fn torch._dynamo.graph_break()Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1625, in __call__ result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__ result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1434, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner return _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1500, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2333, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward return getattr(self.realize(), name)(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function unimplemented_v2( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2 raise Unsupported(msg)torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlfrom user code: File "/tmp/ipykernel_532/2546874632.py", line 5, in fn torch._dynamo.graph_break()Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"tensor([1.4041, 2.4074, 2.5800])
We can avoid skipping by moving the graph break outside of the try block:
@torch.compiledeffn(x):try:x=x+1exceptExceptionase:passtorch._dynamo.graph_break()try:returnx+1exceptExceptionase:passfn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_532/3015389759.py:1, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/3015389759.py", line 12, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/3015389759.py:1create_envTRACE starts_line /tmp/ipykernel_532/3015389759.py:3 in fn try:TRACE SETUP_FINALLY 14 []TRACE starts_line /tmp/ipykernel_532/3015389759.py:4 in fn x = x + 1TRACE LOAD_FAST x []TRACE LOAD_CONST 1 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [TensorVariable()]TRACE POP_BLOCK None []TRACE JUMP_FORWARD 50 []TRACE starts_line /tmp/ipykernel_532/3015389759.py:7 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR graph_break [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]Graph break in user code at /tmp/ipykernel_532/3015389759.py:7Graph Break Reason: Call to `torch._dynamo.graph_break()` Explanation: User-inserted graph break. Message: None Hint: Remove the `torch._dynamo.graph_break()` call. Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}` For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.htmlUser code traceback: File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/3015389759.py", line 12, in <module> fn(torch.randn(3)) File "/tmp/ipykernel_532/3015389759.py", line 7, in fn torch._dynamo.graph_break()Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysisStep 1: torchdynamo start tracing fn /tmp/ipykernel_532/3015389759.py:1create_envTRACE starts_line /tmp/ipykernel_532/3015389759.py:3 in fn try:TRACE SETUP_FINALLY 14 []TRACE starts_line /tmp/ipykernel_532/3015389759.py:4 in fn x = x + 1TRACE LOAD_FAST x []TRACE LOAD_CONST 1 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE STORE_FAST x [TensorVariable()]TRACE POP_BLOCK None []TRACE JUMP_FORWARD 50 []TRACE starts_line /tmp/ipykernel_532/3015389759.py:7 in fn torch._dynamo.graph_break()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR graph_break [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_532/3015389759.py, line 7 in fn>], graph_break=True)TRACED GRAPH ===== __compiled_fn_35_9cf6feae_253b_490a_9640_de9cdf09ab17 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/3015389759.py:4 in fn, code: x = x + 1 x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None return (x,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_532/3015389759.py:4 in fn| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_532/3015389759.py:4 in fn| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(False, False)| | +- GuardManager: source=G['torch'], accessed_by=DictGetItemGuardAccessor('torch'), type=<class 'module'>, tag_safe=(False, False)| | | +- ID_MATCH: ___check_obj_id(G['torch'], 140376310990704) # torch._dynamo.graph_break() # mp/ipykernel_532/3015389759.py:7 in fn| | | +- GuardManager: source=G['torch']._dynamo, accessed_by=GetAttrGuardAccessor(_dynamo), type=<class 'module'>, tag_safe=(False, False)| | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo, 140374889071936) # torch._dynamo.graph_break() # mp/ipykernel_532/3015389759.py:7 in fn| | | | +- GuardManager: source=G['torch']._dynamo.graph_break, accessed_by=GetAttrGuardAccessor(graph_break), type=<class 'function'>, tag_safe=(True, False)| | | | | +- GuardManager: source=G['torch']._dynamo.graph_break.__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)| | | | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo.graph_break.__code__, 140374703207360) # torch._dynamo.graph_break() # mp/ipykernel_532/3015389759.py:7 in fnGuard eval latency = 78.68 usput_code_state: no cache key, skippingrun_gc_after_compile: running gctorchdynamo start compiling torch_dynamo_resume_in_fn_at_7 /tmp/ipykernel_532/3015389759.py:7, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/3015389759.py", line 12, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_7 /tmp/ipykernel_532/3015389759.py:7create_envTRACE starts_line /tmp/ipykernel_532/3015389759.py:7 in torch_dynamo_resume_in_fn_at_7 torch._dynamo.graph_break()TRACE LOAD_CONST True []TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]TRACE LOAD_FAST ___stack0 []TRACE LOAD_CONST False [LazyVariableTracker()]TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]TRACE JUMP_ABSOLUTE 70 [LazyVariableTracker()]TRACE POP_TOP None [LazyVariableTracker()]TRACE starts_line /tmp/ipykernel_532/3015389759.py:8 in torch_dynamo_resume_in_fn_at_7 try:TRACE SETUP_FINALLY 84 []TRACE starts_line /tmp/ipykernel_532/3015389759.py:9 in torch_dynamo_resume_in_fn_at_7 return x + 1TRACE LOAD_FAST x []TRACE LOAD_CONST 1 [LazyVariableTracker()]TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE POP_BLOCK None [TensorVariable()]TRACE RETURN_VALUE None [TensorVariable()]Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_7 (RETURN_VALUE)RETURN_VALUE triggered compileCOMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_532/3015389759.py, line 9 in torch_dynamo_resume_in_fn_at_7>], graph_break=False)TRACED GRAPH ===== __compiled_fn_38_5cdd28f0_bd4c_41fe_864e_fe626167383a ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/3015389759.py:9 in torch_dynamo_resume_in_fn_at_7, code: return x + 1 add: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None return (add,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=3), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 1 # mp/ipykernel_532/3015389759.py:9 in torch_dynamo_resume_in_fn_at_7| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 1 # mp/ipykernel_532/3015389759.py:9 in torch_dynamo_resume_in_fn_at_7Guard eval latency = 61.02 usput_code_state: no cache key, skippingrun_gc_after_compile: running gctensor([1.5745, 2.3647, 1.8617])
Hitting a Recompilation Limit#
Compiler Errors#
Some compiler errors will result in skipped functions.Other compiler errors will result in a hard error rather than a skipped function.
Dealing with Skipped Functions#
In general, you can resolve a skipped function by fixing the underlying graph break or error thatis causing the function to be skipped.
If the graph break/error causing the skipped function is difficult to fix,then consider isolating the graph break/error in its own function so that minimal things are skipped.
definner1(x):returnx+1definner2(x):returnx+2@torch.compiledeffn(x):x=inner1(x)defproblematic_code():torch._dynamo.skip_frame()problematic_code()x=inner2(x)fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_532/273153676.py:5, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/273153676.py", line 12, in <module> fn(torch.randn(3))Step 1: torchdynamo start tracing fn /tmp/ipykernel_532/273153676.py:5create_envTRACE starts_line /tmp/ipykernel_532/273153676.py:7 in fn x = inner1(x)TRACE LOAD_GLOBAL inner1 []TRACE LOAD_FAST x [LazyVariableTracker()]TRACE CALL_FUNCTION 1 [LazyVariableTracker(), LazyVariableTracker()]INLINING <code object inner1 at 0x7fabee5514d0, file "/tmp/ipykernel_532/273153676.py", line 1>, inlined according trace_rules.lookup inlined by defaultwrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE starts_line /tmp/ipykernel_532/273153676.py:2 in inner1 (inline depth: 1) return x + 1TRACE LOAD_FAST x []TRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE RETURN_VALUE None [TensorVariable()]DONE INLINING <code object inner1 at 0x7fabee5514d0, file "/tmp/ipykernel_532/273153676.py", line 1>TRACE STORE_FAST x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/273153676.py:8 in fn def problematic_code():TRACE LOAD_CONST <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8> []TRACE LOAD_CONST fn.<locals>.problematic_code [ConstantVariable(code: <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8>)]TRACE MAKE_FUNCTION 0 [ConstantVariable(code: <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8>), ConstantVariable(str: 'fn.<locals>.problematic_code')]TRACE STORE_FAST problematic_code [NestedUserFunctionVariable()]TRACE starts_line /tmp/ipykernel_532/273153676.py:10 in fn problematic_code()TRACE LOAD_FAST problematic_code []TRACE CALL_FUNCTION 0 [NestedUserFunctionVariable()]INLINING <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8>, inlined according trace_rules.lookup inlined by defaultTRACE starts_line /tmp/ipykernel_532/273153676.py:9 in problematic_code (inline depth: 1) torch._dynamo.skip_frame()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR skip_frame [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]SKIPPED INLINING <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8>: Skip frame due to `torch._dynamo.skip_frame()`. Message: NoneGraph break in user code at /tmp/ipykernel_532/273153676.py:10Graph Break Reason: SKIPPED INLINING <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8>: Skip frame due to `torch._dynamo.skip_frame()`. Message: NoneUser code traceback: File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/273153676.py", line 12, in <module> fn(torch.randn(3)) File "/tmp/ipykernel_532/273153676.py", line 10, in fn problematic_code()Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysisStep 1: torchdynamo start tracing fn /tmp/ipykernel_532/273153676.py:5create_envTRACE starts_line /tmp/ipykernel_532/273153676.py:7 in fn x = inner1(x)TRACE LOAD_GLOBAL inner1 []TRACE LOAD_FAST x [LazyVariableTracker()]TRACE CALL_FUNCTION 1 [LazyVariableTracker(), LazyVariableTracker()]INLINING <code object inner1 at 0x7fabee5514d0, file "/tmp/ipykernel_532/273153676.py", line 1>, inlined according trace_rules.lookup inlined by defaultwrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE starts_line /tmp/ipykernel_532/273153676.py:2 in inner1 (inline depth: 1) return x + 1TRACE LOAD_FAST x []TRACE LOAD_CONST 1 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]TRACE RETURN_VALUE None [TensorVariable()]DONE INLINING <code object inner1 at 0x7fabee5514d0, file "/tmp/ipykernel_532/273153676.py", line 1>TRACE STORE_FAST x [TensorVariable()]TRACE starts_line /tmp/ipykernel_532/273153676.py:8 in fn def problematic_code():TRACE LOAD_CONST <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8> []TRACE LOAD_CONST fn.<locals>.problematic_code [ConstantVariable(code: <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8>)]TRACE MAKE_FUNCTION 0 [ConstantVariable(code: <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8>), ConstantVariable(str: 'fn.<locals>.problematic_code')]TRACE STORE_FAST problematic_code [NestedUserFunctionVariable()]TRACE starts_line /tmp/ipykernel_532/273153676.py:10 in fn problematic_code()TRACE LOAD_FAST problematic_code []TRACE CALL_FUNCTION 0 [NestedUserFunctionVariable()]COMPILING GRAPH due to GraphCompileReason(reason='SKIPPED INLINING <code object problematic_code at 0x7fab85cc7e10, file "/tmp/ipykernel_532/273153676.py", line 8>: Skip frame due to `torch._dynamo.skip_frame()`. Message: None', user_stack=[<FrameSummary file /tmp/ipykernel_532/273153676.py, line 10 in fn>], graph_break=True)TRACED GRAPH ===== __compiled_fn_41_48d91d4a_4ea8_4f47_a3d3_109b14872333 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/273153676.py:2 in inner1, code: return x + 1 x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None return (x,) Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = inner1(x) # mp/ipykernel_532/273153676.py:7 in fn| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = inner1(x) # mp/ipykernel_532/273153676.py:7 in fn| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(True, False)| | +- GuardManager: source=G['inner1'], accessed_by=DictGetItemGuardAccessor('inner1'), type=<class 'function'>, tag_safe=(True, False)| | | +- GuardManager: source=G['inner1'].__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)| | | | +- ID_MATCH: ___check_obj_id(G['inner1'].__code__, 140376414688464) # x = inner1(x) # mp/ipykernel_532/273153676.py:7 in fnGuard eval latency = 1067.03 usput_code_state: no cache key, skippingrun_gc_after_compile: running gctorchdynamo start compiling problematic_code /tmp/ipykernel_532/273153676.py:8, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/273153676.py", line 12, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing problematic_code /tmp/ipykernel_532/273153676.py:8create_envTRACE starts_line /tmp/ipykernel_532/273153676.py:9 in problematic_code torch._dynamo.skip_frame()TRACE LOAD_GLOBAL torch []TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]TRACE LOAD_ATTR skip_frame [LazyVariableTracker()]TRACE CALL_FUNCTION 0 [LazyVariableTracker()]Skipping frame Skip frame due to `torch._dynamo.skip_frame()`. Message: None problematic_code /tmp/ipykernel_532/273153676.py 8put_code_state: no cache key, skippingrun_gc_after_compile: running gctorchdynamo start compiling torch_dynamo_resume_in_fn_at_10 /tmp/ipykernel_532/273153676.py:10, stack (elided 5 frames): File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module> app.launch_new_instance() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance app.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start self.io_loop.start() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start self.asyncio_loop.run_forever() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context return await f(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 614, in shell_main await self.dispatch_shell(msg, subshell_id=subshell_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_shell await result File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 366, in execute_request await super().execute_request(stream, ident, parent) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 827, in execute_request reply_content = await reply_content File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 458, in do_execute res = shell.run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell return super().run_cell(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell result = self._run_cell( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell result = runner(coro) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_532/273153676.py", line 12, in <module> fn(torch.randn(3)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper return fn(*args, **kwargs)Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_10 /tmp/ipykernel_532/273153676.py:10create_envTRACE starts_line /tmp/ipykernel_532/273153676.py:10 in torch_dynamo_resume_in_fn_at_10 problematic_code()TRACE LOAD_CONST True []TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]TRACE LOAD_FAST ___stack0 []TRACE LOAD_CONST False [LazyVariableTracker()]TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]TRACE JUMP_ABSOLUTE 32 [LazyVariableTracker()]TRACE POP_TOP None [LazyVariableTracker()]TRACE starts_line /tmp/ipykernel_532/273153676.py:11 in torch_dynamo_resume_in_fn_at_10 x = inner2(x)TRACE LOAD_GLOBAL inner2 []TRACE LOAD_FAST x [LazyVariableTracker()]TRACE CALL_FUNCTION 1 [LazyVariableTracker(), LazyVariableTracker()]INLINING <code object inner2 at 0x7fab86277b50, file "/tmp/ipykernel_532/273153676.py", line 3>, inlined according trace_rules.lookup inlined by defaultwrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=FalseTRACE starts_line /tmp/ipykernel_532/273153676.py:4 in inner2 (inline depth: 1) return x + 2TRACE LOAD_FAST x []TRACE LOAD_CONST 2 [TensorVariable()]TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 2)]TRACE RETURN_VALUE None [TensorVariable()]DONE INLINING <code object inner2 at 0x7fab86277b50, file "/tmp/ipykernel_532/273153676.py", line 3>TRACE STORE_FAST x [TensorVariable()]TRACE LOAD_CONST None []TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_10 (RETURN_VALUE)RETURN_VALUE triggered compileCOMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_532/273153676.py, line 11 in torch_dynamo_resume_in_fn_at_10>], graph_break=False)TRACED GRAPH ===== __compiled_fn_45_b13efbe6_821e_4bde_a769_2f883cd09f46 ===== /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module): def forward(self, L_x_: "f32[3][1]cpu"): l_x_ = L_x_ # File: /tmp/ipykernel_532/273153676.py:4 in inner2, code: return x + 2 x: "f32[3][1]cpu" = l_x_ + 2; l_x_ = x = None return () Step 2: calling compiler function eagerStep 2: done compiler function eagerproduce_guardstrack_symint L['x'].size()[0] 3 Nonetrack_symint L['x'].stride()[0] 1 Nonetrack_symint L['x'].storage_offset() 0 NoneSkipping guard L['x'].size()[0] == 3Skipping guard L['x'].stride()[0] == 1Skipping guard L['x'].storage_offset() == 0GUARDS:TREE_GUARD_MANAGER:+- RootGuardManager| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards| +- GLOBAL_STATE: ___check_global_state()| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=3), type=<class 'torch.Tensor'>, tag_safe=(True, False)| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = inner2(x) # mp/ipykernel_532/273153676.py:11 in torch_dynamo_resume_in_fn_at_10| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = inner2(x) # mp/ipykernel_532/273153676.py:11 in torch_dynamo_resume_in_fn_at_10| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(True, False)| | +- GuardManager: source=G['inner2'], accessed_by=DictGetItemGuardAccessor('inner2'), type=<class 'function'>, tag_safe=(True, False)| | | +- GuardManager: source=G['inner2'].__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)| | | | +- ID_MATCH: ___check_obj_id(G['inner2'].__code__, 140374666869584) # x = inner2(x) # mp/ipykernel_532/273153676.py:11 in torch_dynamo_resume_in_fn_at_10Guard eval latency = 1082.61 usput_code_state: no cache key, skippingrun_gc_after_compile: running gc On this page
