Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

PyObject finalizer#692

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
filmor merged 29 commits intopythonnet:masterfromamos402:pyobject-finalizer
Apr 1, 2019
Merged

Conversation

amos402
Copy link
Member

What does this implement/fix? Explain your changes.

  • Added PyObject finalizer support, Python objects referred by C# can be auto collect now.
  • PythonException included C# call stack (well..., this added by debug passingly)

Does this close any currently open issues?

...

Any other comments?

...

Checklist

Check all those that are applicable and complete.

  • Make sure to include one or more tests for your change
  • If an enhancement PR, please create docs and at best an example
  • Add yourself toAUTHORS
  • Updated theCHANGELOG

@codecov
Copy link

codecovbot commentedJun 24, 2018
edited
Loading

Codecov Report

Merging#692 intomaster willdecrease coverage by0.21%.
The diff coverage is65%.

Impacted file tree graph

@@            Coverage Diff             @@##           master     #692      +/-   ##==========================================- Coverage    76.9%   76.69%   -0.22%==========================================  Files          63       64       +1       Lines        5798     5925     +127       Branches      950      977      +27     ==========================================+ Hits         4459     4544      +85- Misses       1025     1052      +27- Partials      314      329      +15
FlagCoverage Δ
#setup_linux68.55% <ø> (ø)⬆️
#setup_windows75.91% <65%> (-0.2%)⬇️
Impacted FilesCoverage Δ
src/runtime/runtime.cs86.84% <100%> (+0.11%)⬆️
src/runtime/pyscope.cs57.28% <20%> (-1.14%)⬇️
src/runtime/delegatemanager.cs81.96% <38.46%> (-6.22%)⬇️
src/runtime/finalizer.cs66.33% <66.33%> (ø)
src/runtime/pythonexception.cs76.66% <81.81%> (+10.56%)⬆️
src/runtime/pyobject.cs41.06% <85.71%> (+0.72%)⬆️

Continue to review full report at Codecov.

Legend -Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing data
Powered byCodecov. Last update5ee234a...4eff81e. Read thecomment docs.

@dmitriyse
Copy link
Contributor

This is duplicate for the#532. There are several PR that are exists more than half of a year, and still not approved.

@den-run-ai
Copy link
Contributor

den-run-ai commentedJun 24, 2018
edited
Loading

@amos402 please review#532 and let me know your feedback.@dmitriyse that code in your PR is really hard to review, it is beyond my understanding of Python/CLR internals. I still give preference to@dmitriyse PR but@amos402 if there is anything extra beyond#532 in your PR, please let us know.
I'm planning to review your PRs within 2 weeks and then merge if no issues are found.

@dmitriyse
Copy link
Contributor

@denfromufa, ok. I will update those PRs in this week.

@amos402
Copy link
MemberAuthor

@denfromufa@dmitriyse I reviewed the#532 roughly, it create a thread for decreasing the reference of python objects, it would slow down Python if you get GIL oftenly.
But my solusion it's no need for a new python thread or .net thread. You can event use it that Python didn't compile withWITH_THREAD macro.
Because of Python's GIL, I suggest we should avoid use multithread on it. btw. I'm going to remove the PyEval_InitThreads call on Initialize for my next version.

Note from python document

Note
When only the main thread exists, no GIL operations are needed. This is a common situation (most Python programs do not use threads), and the lock operations slow the interpreter down a bit. Therefore, the lock is not created initially. This situation is equivalent to having acquired the lock: when there is only a single thread, all object accesses are safe. Therefore, when this function initializes the global interpreter lock, it also acquires it. Before the Python _thread module creates a new thread, knowing that either it has the lock or the lock hasn’t been created yet, it calls PyEval_InitThreads(). When this call returns, it is guaranteed that the lock has been created and that the calling thread has acquired it.

It is not safe to call this function when it is unknown which thread (if any) currently has the global interpreter lock.

This function is not available when thread support is disabled at compile time.

@dmitriyse
Copy link
Contributor

dmitriyse commentedJun 25, 2018
edited
Loading

I agree with@amos402 Py_AddPendingCall can give some benefits and my approach can be improved for Python 3.1+.
But I cannot agree right now with WeakReferences and GC.ReRegisterForFinalize approach. I need some research.

We are looking for different directions: single and multi threaded environment. Dedicated thread was the only approach that succesfully solve memory leak in the my multithreaded application.

@amos402 Please check this#541 bug on your branch, ensure that your finalizers approach also elliminates memory leak.

@amos402
Copy link
MemberAuthor

@dmitriyse I try two cases, it seems no problems exceptPythonException was leak by original code. I fixed it on my branch.

can be improved for Python 3.1+
Why 3.1 specificed?
WeakReferences only use for query garbages, it's no need for running time.
If you want to avoid callGC.ReRegisterForFinalize, you can just make a interface for getting all python object pointer. But I think it's not necessary, garbages will put into garbage queue, with the queue's reference, they can be resurrected.

private ConcurrentQueue<IDisposable> _objQueue = new ConcurrentQueue<IDisposable>();

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
private delegate int PedingCall(IntPtr arg);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

PendingCall?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The calling type for Py_AddPendingCall

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I know, there's just a typo in the name ;)


public event EventHandler<CollectArgs> CollectOnce;

private ConcurrentQueue<IDisposable> _objQueue = new ConcurrentQueue<IDisposable>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

We already have a few places that fail in presence of multiple appdomains, this would too, I guess. Can you add a comment on this?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

What failures will have occurred, can you give me some examples?

private bool _pending = false;
private readonly object _collectingLock = new object();
public int Threshold { get; set; }
public bool Enable { get; set; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why are these public?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Make users can adjust the finalizer's configuration.

}
Instance.DisposeAll();
Instance.CallPendingFinalizers();
Runtime.PyErr_Clear();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why are you calling this?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Make sure no garbage left and clear the pending calls.
Pending calls wouldn't clear after calling Py_Finalize, if reset the .net environment, the function pointers would point to unknown memory and cause crash on calling them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why are you callingPyErr_Clear?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

emmm...It seems I should expose the exceptions in DisposeAll after eachobj.Dispose()


private void AddPendingCollect()
{
lock (_collectingLock)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This lock should probably encompass the whole function.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why it needs to be lock whole function, the flag already set.
I can't image any situations that Py_AddPendingCall will be called in multiple threads on same time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I don't follow, either you expect multiple concurrent executions, then the lock needs to surround the whole function, or you don't, then you don't need any locking.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Oh, yes, thank you for you reminder, I neglect that there is another store at line 114.

@@ -198,6 +199,8 @@ public class Runtime
internal static bool IsPython2 = pyversionnumber < 30;
internal static bool IsPython3 = pyversionnumber >= 30;

public static int MainManagedThreadId { get; internal set; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Isn't this rather aprivate set?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yes, it should be. I will modify it later.

@@ -350,6 +354,7 @@ internal static void Initialize()

internal static void Shutdown()
{
Finalizer.Shutdown();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is there a particular reason for doing theFinalizer shutdown first? I would have expected it to come last.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It seems there is no particular reason for that. I can't remember why I put it on first, maybe just put it casually....

@filmorfilmor added this to the2.4.0 milestoneJul 23, 2018
if (_pending)
{
return;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is this optimisation really needed? If not, I'd rather like the code to be obviously safe.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Just a simple double-check, make it check without lock at first.

@den-run-ai
Copy link
Contributor

@amos402 some git conflicts with master branch now.

@den-run-ai
Copy link
Contributor

This PR causes appveyor CI to get stuck for one hour on each build.

@amos402
Copy link
MemberAuthor

amos402 commentedOct 24, 2018
edited
Loading

This PR causes appveyor CI to get stuck for one hour on each build.

😥I still can't figure out why it just crash on 3.7 on Linux

@amos402
Copy link
MemberAuthor

Already fixed it. It crash on 3.7 because I changed PYTHONMALLOC in test case.

@amos402
Copy link
MemberAuthor

I added ref count check on debug mode, it should be helpful for finding the bugs like this

PyStrings1=newPyString("test_string");// s2 steal a reference from s1PyStrings2=newPyString(s1.Handle);

@filmorfilmor mentioned this pull requestJan 18, 2019
@filmorfilmor removed this from the2.4.0 milestoneFeb 5, 2019
@filmorfilmor merged commitf83c884 intopythonnet:masterApr 1, 2019
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@filmorfilmorfilmor left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

4 participants
@amos402@dmitriyse@den-run-ai@filmor

[8]ページ先頭

©2009-2025 Movatter.jp