Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit784190a

Browse files
Benoit Hudsonfilmor
Benoit Hudson
authored andcommitted
Enable pythonnet to survive in the Unity3d editor environment.
The Unity editor unloads the C# domain and creates a new domain every time it recompiles C# sources.This caused two crashes.1. After a domain unload, python is now pointing to a bunch of objects whose C# side is now garbage. Solution: Shutdown() the engine on domain reload, which calls Py_Finalize.2. After a domain unload, Py_Finalize, and a new Py_Intialize, python still keeps pointers in gc for any python objects that were leaked. And pythonnet leaks. This means that python's gc will be calling tp_traverse, tp_clear, and tp_is_gc on types that are implemented in C#. This crashes because the implementation was freed. Solution: implement those calls in code that is *not* released on domain unload. Side effect: we leak a page on every domain unload. I suspect but didn't test that python gc performance will be slightly faster.Changes required to implement and test:3. Use python's platform package to determine what we're running on, so we can use the right mmap/mprotect or VirtualAlloc/VirtualProtect routines, the right flags for mmap, and (theoretically) the right native code. Side effect: to port to a new platform requires updating some additional code now.4. New unit tests. (a) A new test for domain reload. It doesn't run under .NET Core because you can't create and unload a domain in .NET Core. (b) New unit tests to help us find where to add support for new platforms.
1 parent05a1451 commit784190a

13 files changed

+725
-53
lines changed

‎AUTHORS.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414

1515
- Alexandre Catarino([@AlexCatarino](https://github.com/AlexCatarino))
1616
- Arvid JB ([@ArvidJB](https://github.com/ArvidJB))
17+
- Benoît Hudson ([@benoithudson](https://github.com/benoithudson))
1718
- Bradley Friedman ([@leith-bartrich](https://github.com/leith-bartrich))
1819
- Callum Noble ([@callumnoble](https://github.com/callumnoble))
1920
- Christian Heimes ([@tiran](https://github.com/tiran))
@@ -22,6 +23,7 @@
2223
- Daniel Fernandez ([@fdanny](https://github.com/fdanny))
2324
- Daniel Santana ([@dgsantana](https://github.com/dgsantana))
2425
- Dave Hirschfeld ([@dhirschfeld](https://github.com/dhirschfeld))
26+
- David Lassonde ([@lassond](https://github.com/lassond))
2527
- David Lechner ([@dlech](https://github.com/dlech))
2628
- Dmitriy Se ([@dmitriyse](https://github.com/dmitriyse))
2729
- He-chien Tsai ([@t3476](https://github.com/t3476))
@@ -40,6 +42,7 @@
4042
- Sam Winstanley ([@swinstanley](https://github.com/swinstanley))
4143
- Sean Freitag ([@cowboygneox](https://github.com/cowboygneox))
4244
- Serge Weinstock ([@sweinst](https://github.com/sweinst))
45+
- Viktoria Kovescses ([@vkovec](https://github.com/vkovec))
4346
- Ville M. Vainio ([@vivainio](https://github.com/vivainio))
4447
- Virgil Dupras ([@hsoft](https://github.com/hsoft))
4548
- Wenguang Yang ([@yagweb](https://github.com/yagweb))

‎CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ This document follows the conventions laid out in [Keep a CHANGELOG][].
2626
###Fixed
2727

2828
- Fixed Visual Studio 2017 compat ([#434][i434]) for setup.py
29+
- Fixed crashes when integrating pythonnet in Unity3d ([#714][i714]),
30+
related to unloading the Application Domain
2931
- Fixed crash on exit of the Python interpreter if a python class
3032
derived from a .NET class has a`__namespace__` or`__assembly__`
3133
attribute ([#481][i481])

‎src/embed_tests/Python.EmbeddingTest.15.csproj

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
<BaseDefineConstants>XPLAT</BaseDefineConstants>
3030
<DefineConstants>$(DefineConstants);$(CustomDefineConstants);$(BaseDefineConstants);</DefineConstants>
3131
<DefineConstantsCondition="'$(TargetFramework)'=='netcoreapp2.0'">$(DefineConstants);NETCOREAPP</DefineConstants>
32+
<DefineConstantsCondition="'$(TargetFramework)'=='netstandard2.0'">$(DefineConstants);NETSTANDARD</DefineConstants>
3233
<DefineConstantsCondition="'$(BuildingInsideVisualStudio)' == 'true' AND '$(CustomDefineConstants)' != '' AND $(Configuration.Contains('Debug'))">$(DefineConstants);TRACE;DEBUG</DefineConstants>
3334
<FrameworkPathOverrideCondition="'$(TargetFramework)'=='net40' AND $(Configuration.Contains('Mono'))">$(NuGetPackageRoot)\microsoft.targetingpack.netframework.v4.5\1.0.1\lib\net45\</FrameworkPathOverride>
3435
</PropertyGroup>

‎src/embed_tests/Python.EmbeddingTest.csproj

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@
8686
<CompileInclude="pyrunstring.cs" />
8787
<CompileInclude="TestConverter.cs" />
8888
<CompileInclude="TestCustomMarshal.cs" />
89+
<CompileInclude="TestDomainReload.cs" />
8990
<CompileInclude="TestExample.cs" />
9091
<CompileInclude="TestPyAnsiString.cs" />
9192
<CompileInclude="TestPyFloat.cs" />
@@ -103,6 +104,7 @@
103104
<CompileInclude="TestPyWith.cs" />
104105
<CompileInclude="TestRuntime.cs" />
105106
<CompileInclude="TestPyScope.cs" />
107+
<CompileInclude="TestTypeManager.cs" />
106108
</ItemGroup>
107109
<ItemGroup>
108110
<ProjectReferenceInclude="..\runtime\Python.Runtime.csproj">
@@ -122,4 +124,4 @@
122124
<CopySourceFiles="$(TargetAssembly)"DestinationFolder="$(PythonBuildDir)" />
123125
<!--Copy SourceFiles="$(TargetAssemblyPdb)" Condition="Exists('$(TargetAssemblyPdb)')" DestinationFolder="$(PythonBuildDir)" /-->
124126
</Target>
125-
</Project>
127+
</Project>

‎src/embed_tests/TestDomainReload.cs

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
usingSystem;
2+
usingSystem.CodeDom.Compiler;
3+
usingSystem.Reflection;
4+
usingNUnit.Framework;
5+
usingPython.Runtime;
6+
7+
//
8+
// This test case is disabled on .NET Standard because it doesn't have all the
9+
// APIs we use. We could work around that, but .NET Core doesn't implement
10+
// domain creation, so it's not worth it.
11+
//
12+
// Unfortunately this means no continuous integration testing for this case.
13+
//
14+
#if!NETSTANDARD&&!NETCOREAPP
15+
namespacePython.EmbeddingTest
16+
{
17+
classTestDomainReload
18+
{
19+
/// <summary>
20+
/// Test that the python runtime can survive a C# domain reload without crashing.
21+
///
22+
/// At the time this test was written, there was a very annoying
23+
/// seemingly random crash bug when integrating pythonnet into Unity.
24+
///
25+
/// The repro steps that David Lassonde, Viktoria Kovecses and
26+
/// Benoit Hudson eventually worked out:
27+
/// 1. Write a HelloWorld.cs script that uses Python.Runtime to access
28+
/// some C# data from python: C# calls python, which calls C#.
29+
/// 2. Execute the script (e.g. make it a MenuItem and click it).
30+
/// 3. Touch HelloWorld.cs on disk, forcing Unity to recompile scripts.
31+
/// 4. Wait several seconds for Unity to be done recompiling and
32+
/// reloading the C# domain.
33+
/// 5. Make python run the gc (e.g. by calling gc.collect()).
34+
///
35+
/// The reason:
36+
/// A. In step 2, Python.Runtime registers a bunch of new types with
37+
/// their tp_traverse slot pointing to managed code, and allocates
38+
/// some objects of those types.
39+
/// B. In step 4, Unity unloads the C# domain. That frees the managed
40+
/// code. But at the time of the crash investigation, pythonnet
41+
/// leaked the python side of the objects allocated in step 1.
42+
/// C. In step 5, python sees some pythonnet objects in its gc list of
43+
/// potentially-leaked objects. It calls tp_traverse on those objects.
44+
/// But tp_traverse was freed in step 3 => CRASH.
45+
///
46+
/// This test distills what's going on without needing Unity around (we'd see
47+
/// similar behaviour if we were using pythonnet on a .NET web server that did
48+
/// a hot reload).
49+
/// </summary>
50+
[Test]
51+
publicstaticvoidDomainReloadAndGC()
52+
{
53+
// We're set up to run in the directory that includes the bin directory.
54+
System.IO.Directory.SetCurrentDirectory(AppDomain.CurrentDomain.BaseDirectory);
55+
56+
AssemblypythonRunner1=BuildAssembly("test1");
57+
RunAssemblyAndUnload(pythonRunner1,"test1");
58+
59+
// Verify that python is not initialized even though we ran it.
60+
Assert.That(Runtime.Runtime.Py_IsInitialized(),Is.Zero);
61+
62+
// This caused a crash because objects allocated in pythonRunner1
63+
// still existed in memory, but the code to do python GC on those
64+
// objects is gone.
65+
AssemblypythonRunner2=BuildAssembly("test2");
66+
RunAssemblyAndUnload(pythonRunner2,"test2");
67+
}
68+
69+
//
70+
// The code we'll test. All that really matters is
71+
// using GIL { Python.Exec(pyScript); }
72+
// but the rest is useful for debugging.
73+
//
74+
// What matters in the python code is gc.collect and clr.AddReference.
75+
//
76+
// Note that the language version is 2.0, so no $"foo{bar}" syntax.
77+
//
78+
conststringTestCode=@"
79+
using Python.Runtime;
80+
using System;
81+
class PythonRunner {
82+
public static void RunPython() {
83+
AppDomain.CurrentDomain.DomainUnload += OnDomainUnload;
84+
string name = AppDomain.CurrentDomain.FriendlyName;
85+
Console.WriteLine(string.Format(""[{0} in .NET] In PythonRunner.RunPython"", name));
86+
using (Py.GIL()) {
87+
try {
88+
var pyScript = string.Format(""import clr\n""
89+
+ ""print('[{0} in python] imported clr')\n""
90+
+ ""clr.AddReference('System')\n""
91+
+ ""print('[{0} in python] allocated a clr object')\n""
92+
+ ""import gc\n""
93+
+ ""gc.collect()\n""
94+
+ ""print('[{0} in python] collected garbage')\n"",
95+
name);
96+
PythonEngine.Exec(pyScript);
97+
} catch(Exception e) {
98+
Console.WriteLine(string.Format(""[{0} in .NET] Caught exception: {1}"", name, e));
99+
}
100+
}
101+
}
102+
static void OnDomainUnload(object sender, EventArgs e) {
103+
System.Console.WriteLine(string.Format(""[{0} in .NET] unloading"", AppDomain.CurrentDomain.FriendlyName));
104+
}
105+
}";
106+
107+
108+
/// <summary>
109+
/// Build an assembly out of the source code above.
110+
///
111+
/// This creates a file <paramref name="assemblyName"/>.dll in order
112+
/// to support the statement "proxy.theAssembly = assembly" below.
113+
/// That statement needs a file, can't run via memory.
114+
/// </summary>
115+
staticAssemblyBuildAssembly(stringassemblyName)
116+
{
117+
varprovider=CodeDomProvider.CreateProvider("CSharp");
118+
119+
varcompilerparams=newCompilerParameters();
120+
compilerparams.ReferencedAssemblies.Add("Python.Runtime.dll");
121+
compilerparams.GenerateExecutable=false;
122+
compilerparams.GenerateInMemory=false;
123+
compilerparams.IncludeDebugInformation=false;
124+
compilerparams.OutputAssembly=assemblyName;
125+
126+
varresults=provider.CompileAssemblyFromSource(compilerparams,TestCode);
127+
if(results.Errors.HasErrors)
128+
{
129+
varerrors=newSystem.Text.StringBuilder("Compiler Errors:\n");
130+
foreach(CompilerErrorerrorinresults.Errors)
131+
{
132+
errors.AppendFormat("Line {0},{1}\t: {2}\n",
133+
error.Line,error.Column,error.ErrorText);
134+
}
135+
thrownewException(errors.ToString());
136+
}
137+
else
138+
{
139+
returnresults.CompiledAssembly;
140+
}
141+
}
142+
143+
/// <summary>
144+
/// This is a magic incantation required to run code in an application
145+
/// domain other than the current one.
146+
/// </summary>
147+
classProxy:MarshalByRefObject
148+
{
149+
AssemblytheAssembly=null;
150+
151+
publicvoidInitAssembly(stringassemblyPath)
152+
{
153+
theAssembly=Assembly.LoadFile(System.IO.Path.GetFullPath(assemblyPath));
154+
}
155+
156+
publicvoidRunPython()
157+
{
158+
Console.WriteLine("[Proxy] Entering RunPython");
159+
160+
// Call into the new assembly. Will execute Python code
161+
varpythonrunner=theAssembly.GetType("PythonRunner");
162+
varrunPythonMethod=pythonrunner.GetMethod("RunPython");
163+
runPythonMethod.Invoke(null,newobject[]{});
164+
165+
Console.WriteLine("[Proxy] Leaving RunPython");
166+
}
167+
}
168+
169+
/// <summary>
170+
/// Create a domain, run the assembly in it (the RunPython function),
171+
/// and unload the domain.
172+
/// </summary>
173+
staticvoidRunAssemblyAndUnload(Assemblyassembly,stringassemblyName)
174+
{
175+
Console.WriteLine($"[Program.Main] === creating domain for assembly{assembly.FullName}");
176+
177+
// Create the domain. Make sure to set PrivateBinPath to a relative
178+
// path from the CWD (namely, 'bin').
179+
// See https://stackoverflow.com/questions/24760543/createinstanceandunwrap-in-another-domain
180+
varcurrentDomain=AppDomain.CurrentDomain;
181+
vardomainsetup=newAppDomainSetup()
182+
{
183+
ApplicationBase=currentDomain.SetupInformation.ApplicationBase,
184+
ConfigurationFile=currentDomain.SetupInformation.ConfigurationFile,
185+
LoaderOptimization=LoaderOptimization.SingleDomain,
186+
PrivateBinPath="."
187+
};
188+
vardomain=AppDomain.CreateDomain(
189+
$"My Domain{assemblyName}",
190+
currentDomain.Evidence,
191+
domainsetup);
192+
193+
// Create a Proxy object in the new domain, where we want the
194+
// assembly (and Python .NET) to reside
195+
Typetype=typeof(Proxy);
196+
System.IO.Directory.SetCurrentDirectory(AppDomain.CurrentDomain.BaseDirectory);
197+
vartheProxy=(Proxy)domain.CreateInstanceAndUnwrap(
198+
type.Assembly.FullName,
199+
type.FullName);
200+
201+
// From now on use the Proxy to call into the new assembly
202+
theProxy.InitAssembly(assemblyName);
203+
theProxy.RunPython();
204+
205+
Console.WriteLine($"[Program.Main] Before Domain Unload on{assembly.FullName}");
206+
AppDomain.Unload(domain);
207+
Console.WriteLine($"[Program.Main] After Domain Unload on{assembly.FullName}");
208+
209+
// Validate that the assembly does not exist anymore
210+
try
211+
{
212+
Console.WriteLine($"[Program.Main] The Proxy object is valid ({theProxy}). Unexpected domain unload behavior");
213+
}
214+
catch(Exception)
215+
{
216+
Console.WriteLine("[Program.Main] The Proxy object is not valid anymore, domain unload complete.");
217+
}
218+
}
219+
220+
/// <summary>
221+
/// Resolves the assembly. Why doesn't this just work normally?
222+
/// </summary>
223+
staticAssemblyResolveAssembly(objectsender,ResolveEventArgsargs)
224+
{
225+
varloadedAssemblies=AppDomain.CurrentDomain.GetAssemblies();
226+
227+
foreach(varassemblyinloadedAssemblies)
228+
{
229+
if(assembly.FullName==args.Name)
230+
{
231+
returnassembly;
232+
}
233+
}
234+
235+
returnnull;
236+
}
237+
}
238+
}
239+
#endif

‎src/embed_tests/TestRuntime.cs

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,26 @@ namespace Python.EmbeddingTest
66
{
77
publicclassTestRuntime
88
{
9+
/// <summary>
10+
/// Test the cache of the information from the platform module.
11+
///
12+
/// Test fails on platforms we haven't implemented yet.
13+
/// </summary>
14+
[Test]
15+
publicstaticvoidPlatformCache()
16+
{
17+
Runtime.Runtime.Initialize();
18+
19+
Assert.That(Runtime.Runtime.Machine,Is.Not.EqualTo(Runtime.Runtime.MachineType.Other));
20+
Assert.That(!string.IsNullOrEmpty(Runtime.Runtime.MachineName));
21+
22+
Assert.That(Runtime.Runtime.OperatingSystem,Is.Not.EqualTo(Runtime.Runtime.OperatingSystemType.Other));
23+
Assert.That(!string.IsNullOrEmpty(Runtime.Runtime.OperatingSystemName));
24+
25+
// Don't shut down the runtime: if the python engine was initialized
26+
// but not shut down by another test, we'd end up in a bad state.
27+
}
28+
929
[Test]
1030
publicstaticvoidPy_IsInitializedValue()
1131
{

‎src/embed_tests/TestTypeManager.cs

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
usingNUnit.Framework;
2+
usingPython.Runtime;
3+
usingSystem.Runtime.InteropServices;
4+
5+
namespacePython.EmbeddingTest
6+
{
7+
classTestTypeManager
8+
{
9+
[SetUp]
10+
publicstaticvoidInit()
11+
{
12+
Runtime.Runtime.Initialize();
13+
}
14+
15+
[TearDown]
16+
publicstaticvoidFini()
17+
{
18+
// Don't shut down the runtime: if the python engine was initialized
19+
// but not shut down by another test, we'd end up in a bad state.
20+
}
21+
22+
[Test]
23+
publicstaticvoidTestNativeCode()
24+
{
25+
Assert.That(()=>{var_=TypeManager.NativeCode.Active;},Throws.Nothing);
26+
Assert.That(TypeManager.NativeCode.Active.Code.Length,Is.GreaterThan(0));
27+
}
28+
29+
[Test]
30+
publicstaticvoidTestMemoryMapping()
31+
{
32+
Assert.That(()=>{var_=TypeManager.CreateMemoryMapper();},Throws.Nothing);
33+
varmapper=TypeManager.CreateMemoryMapper();
34+
35+
// Allocate a read-write page.
36+
intlen=12;
37+
varpage=mapper.MapWriteable(len);
38+
Assert.That(()=>{Marshal.WriteInt64(page,17);},Throws.Nothing);
39+
Assert.That(Marshal.ReadInt64(page),Is.EqualTo(17));
40+
41+
// Mark it read-execute. We can still read, haven't changed any values.
42+
mapper.SetReadExec(page,len);
43+
Assert.That(Marshal.ReadInt64(page),Is.EqualTo(17));
44+
45+
// Test that we can't write to the protected page.
46+
//
47+
// We can't actually test access protection under Microsoft
48+
// versions of .NET, because AccessViolationException is assumed to
49+
// mean we're in a corrupted state:
50+
// https://stackoverflow.com/questions/3469368/how-to-handle-accessviolationexception
51+
//
52+
// We can test under Mono but it throws NRE instead of AccessViolationException.
53+
//
54+
// We can't use compiler flags because we compile with MONO_LINUX
55+
// while running on the Microsoft .NET Core during continuous
56+
// integration tests.
57+
if(System.Type.GetType("Mono.Runtime")!=null)
58+
{
59+
// Mono throws NRE instead of AccessViolationException for some reason.
60+
Assert.That(()=>{Marshal.WriteInt64(page,73);},Throws.TypeOf<System.NullReferenceException>());
61+
Assert.That(Marshal.ReadInt64(page),Is.EqualTo(17));
62+
}
63+
}
64+
}
65+
}

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp