Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Weakref support#1267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Closed
amos402 wants to merge13 commits intopythonnet:masterfromamos402:weakref-support
Closed

Conversation

amos402
Copy link
Member

@amos402amos402 commentedOct 22, 2020
edited
Loading

What does this implement/fix? Explain your changes.

  1. A new memory model for holding the CLR handle
  2. Unified the meaning ofpyHandle andtpHandle inManagedType, the usage ofobj.tpHandle == obj.tpHandle represents theobj as aClassBase itself is no need anymore
  3. weakref can be used for CLR object
weakref.ref(clr_obj)

Solved the problem that the code like as above would occur a crash because the slot of weakref and the slot of clr handle was overlapping.
4. Minor refactor ofSlotsHolder

Does this close any currently open issues?

#1219 (comment)

Any other comments?

...

Checklist

Check all those that are applicable and complete.

  • Make sure to include one or more tests for your change
  • If an enhancement PR, please create docs and at best an example
  • Add yourself toAUTHORS
  • Updated theCHANGELOG

@codecov-io
Copy link

codecov-io commentedOct 22, 2020
edited
Loading

Codecov Report

Merging#1267 intomaster willnot change coverage.
The diff coverage isn/a.

Impacted file tree graph

@@           Coverage Diff           @@##           master    #1267   +/-   ##=======================================  Coverage   86.25%   86.25%           =======================================  Files           1        1             Lines         291      291           =======================================  Hits          251      251             Misses         40       40
FlagCoverage Δ
#setup_linux64.94% <ø> (ø)
#setup_windows72.50% <ø> (ø)

Flags with carried forward coverage won't be shown.Click here to find out more.


Continue to review full report at Codecov.

Legend -Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing data
Powered byCodecov. Last updatede7c7c2...f08813c. Read thecomment docs.


public static int tp_init(IntPtr ob, IntPtr args, IntPtr kwds)
{
Exceptions.SetArgsAndCause(ob);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Please, move the original comment here.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

What comment?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

NIT: You seem to have moved this fromCLRObject, and removed

// Fix the BaseException args (and __cause__ in case of Python 3)// slot if wrapping a CLR exception

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The comment onSetArgsAndCause won't be enough?

@@ -2123,6 +2131,16 @@ internal static void Py_CLEAR(ref IntPtr ob)
ob = IntPtr.Zero;
}

internal static unsafe void Py_SETREF(IntPtr ob, int offset, IntPtr target)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This seem to be silently stealing a reference totarget. I'd prefer it to explicitly takeNewReference orref NewReference (to clear the original variable), or takeBorrowedReference andIncRef here.

Ideally, I would also checkob andoffset for sanity.

Also, this really should also be outside theRuntime class, but I am not sure where is best to put it.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The original version is the C macro, the implementation used pointer offset, maybe is not suited for such a high-level type to be involved. I want to keep such things simple, If you need that, maybe create a high-level wrapper.😓

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

NewReference is not high-level. It is just a struct.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Then you have to pull the pointer and add an offset, what a tortuous path.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It is better, than accidentally forgetting to incref thetarget in the caller.

Comment on lines 151 to 158
//TypeManager.CopySlot(base_type, type, TypeOffset.tp_dealloc);

// Hmm - the standard subtype_traverse, clear look at ob_size to
// do things, so to allow gc to work correctly we need to move
// our hidden handle out of ob_size. Then, in theory we can
// comment this out and still not crash.
TypeManager.CopySlot(base_type, type, TypeOffset.tp_traverse);
TypeManager.CopySlot(base_type, type, TypeOffset.tp_clear);

//TypeManager.CopySlot(base_type, type, TypeOffset.tp_traverse);
//TypeManager.CopySlot(base_type, type, TypeOffset.tp_clear);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Please, remove dead code entirely.

Can you also comment why is this no longer necessary?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

These slots can inherit from the base type automatically. Emmm, I can add some comments here, but it still needs the dead code comment as an explanation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The comment above also seems dangerously wrong.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hmmm, that was an old comment, that time before we unify the shutdown progresses, it can't and won't use tp_traverse and tp_clear. Seems it can be removed at one time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Any dead code should be removed immediately.

@@ -1991,6 +1996,9 @@ internal static IntPtr PyType_GenericAlloc(IntPtr type, long n)
[DllImport(_PythonDll, CallingConvention = CallingConvention.Cdecl)]
internal static extern void PyObject_GC_UnTrack(IntPtr tp);

[DllImport(_PythonDll, CallingConvention = CallingConvention.Cdecl)]
internal static extern void PyObject_ClearWeakRefs(IntPtr obj);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

We should try to use*Reference types for clarity for all new PInvoke functions.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

That makes sense, but can we just replace them all at once.😅

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@amos402 are you prepared to make this happen? )

@@ -1982,6 +1984,9 @@ internal static IntPtr PyType_GenericAlloc(IntPtr type, long n)
[DllImport(_PythonDll, CallingConvention = CallingConvention.Cdecl)]
internal static extern IntPtr _PyObject_GetDictPtr(IntPtr obj);

[DllImport(_PythonDll, CallingConvention = CallingConvention.Cdecl)]
internal static extern IntPtr _PyObject_GC_Calloc(IntPtr basicsize);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It is a very bad idea to use private function. Why is it required? Can the same be done with only public functions?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I'm afraid there's no such public function now, it's required to allocate a custom size Python GC object.


Marshal.WriteIntPtr(type, TypeOffset.tp_base, py_type);
IntPtrtype = Runtime._PyObject_GC_Calloc(new IntPtr(metaSize));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Specifically, can we usePyType_FromSpec here?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Maybe not,PyType_FromSpec still usePyType_Type to allocate memory, which means it would occur the same problem as before.

Copy link
Member

@lostmsulostmsuDec 21, 2020
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Actually, I think I solved this one. Please see9682dc6
So you should not need_PyObject_GC_Calloc anymore.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It's not about the type size which the MetaType can generate. It's about the size of MetaType itself. You can't create the MetaType by usingPyType_GenericAlloc(PyTypeType, 0), unless you

old_size=PyTypeType.tp_basicsize;PyTypeType.tp_basicsize+=x;PyType_GenericAlloc(PyTypeType,0);PyTypeType.tp_basicsize=old_size;

As long as it claimed itself as a managed type, it should have a slot indicate the place of its instances where the related GCHandle would be stored. As for now, the MetaType doesn't attach any C# object, so it remains 0. Look back to the master, if you put the PyCLRMetaType toGetManagedObject, actually it would read the memory it should not read(the part of the reservedtp_itemsize). The code doesn't explicitly call something like that, but something may trigger it does, like domain reload may visit it through PyVisit. Even it won't crash right now, but it's hard to say that's a kind of sanity code.
Of course, you can just remove the Manged flag from PyCLRMetaType to avoid that. But I suggest following the old rule: the type of a CLR type is a managed type.
Another approach is to allocate with PyObject_Malloc, but that means it has to be initialized manually, e.g. call_Py_NewReference manually on debug version, add it to GCTrack.
Nevertheless, the point is, the type must clarify itself what's itself and what it can do, that commit avoided some problems, but didn't solve completely.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

As far as I can see, CLR Metatype does not containGCHandle, so it does not needGetManagedObject to work on its instances. So I think my implementation linked above is correct. PerhapsPyCLRMetaType should not be a managed object then?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It indeed has a bit of struggle with that, the PyCLRMetaType it's the ancestor of all managed types, if it claimed it's not a managed type, it against the commonsense. Currently, it doesn't need to be bound with a clr instance even it can. Well, just an identity problem.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think it makes sense, that CLR metatype isnot a managed type. In my mind managed type == type, whose implementation is managed == has a managed object, that implements functionality == hasGCHandle pointing to that object. CLR metatype does not have aGCHandle.

Comment on lines 10 to 12
// XXX: Use field to trick the public property finding
public readonly int Size;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why do you think this is better, than the single line you removed from ABI.cs? This is hacky and uses reflection.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It's not related to that line, the reason for removing that line just because was because that's not the fixed slot anymore.
Well, it was just need something to store the size, tp_basicsize might capable do that too, this part can be improved.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This field has the same purpose and value as the one you removed from ABI -sizeof(PyTypeObject)

@amos402amos402 mentioned this pull requestFeb 19, 2021
4 tasks
public void WeakRefForClrObject()
{
var obj = new MyClass();
using (var scope = Py.CreateScope())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

NIT: use C# 9 using syntax

Comment on lines 151 to 158
//TypeManager.CopySlot(base_type, type, TypeOffset.tp_dealloc);

// Hmm - the standard subtype_traverse, clear look at ob_size to
// do things, so to allow gc to work correctly we need to move
// our hidden handle out of ob_size. Then, in theory we can
// comment this out and still not crash.
TypeManager.CopySlot(base_type, type, TypeOffset.tp_traverse);
TypeManager.CopySlot(base_type, type, TypeOffset.tp_clear);

//TypeManager.CopySlot(base_type, type, TypeOffset.tp_traverse);
//TypeManager.CopySlot(base_type, type, TypeOffset.tp_clear);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Any dead code should be removed immediately.

Comment on lines 364 to 365
public IntPtr ClrHandleOffset;
public IntPtr ClrHandle;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

These two refer to different handles, yet are named similarly, which is confusing.

Besides, in C# 9 we can start usingnint instead ofIntPtr for non-pointers.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

What name do you prefer?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is just a NIT. MaybeTypeClrHandle andInstanceClrHandleOffset?

Comment on lines +122 to +124
IntPtr op = typeof(Exception).IsAssignableFrom(ob.GetType()) ?
Exceptions.GetExceptHandle((Exception)ob)
: CLRObject.GetInstHandle(ob);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I feel likeExceptions.GetExceptionHandle should be built intoCLRObject.GetInstHandle.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The exception object is a special case, we should keep it outside from the generic caseGetInstHandle, for example, if it has multiple special cases instead of one, you won't want to contaminate the CLRObject methods, right? Just like I removed the special case code from the CLRObject's constructor.

Comment on lines +538 to 542
if (obj.RefCount > 1)
{
obj.FreeGCHandle();
Marshal.WriteIntPtr(obj.pyHandle, ObjectOffset.magic(obj.tpHandle), IntPtr.Zero);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why can't you do this unconditionally? This condition makes reasoning about the state much harder.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Ifobj.RefCount == 1, the nextXDecref would dealloc the obj, thus it's no need to call these things.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is this similar totp_clear in any way? I would suggest to extract these two updates (FreeGCHandle and zeroing out) into a method.

The point beingtp_clear has certain semantics, so it is easier to understand code containing a call to it, rather than has same actions inline.

Also, ifRefCount will correctly return 0, I'd move thisif below.

Copy link
MemberAuthor

@amos402amos402Feb 22, 2021
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Not exactly,tp_clear is used to clear the Python object reference in order to break reference cycles. These codes are used to disconnect the relation between Python and clr, if the RefCount reached 0, their relation would naturally be gone.
Their relation needs to be kept if RefCount > 1 because intp_dealloc, it needs to obtain the bound object.
tp_dealloc didn't zero out the handle because this chunk of memory would be freed soon aftertp_dealloc, extract these into a method is not really necessary.
Since after dealloc the memory may be released, theif scope cannot move to below.


Marshal.WriteIntPtr(type, TypeOffset.tp_base, py_type);
IntPtrtype = Runtime._PyObject_GC_Calloc(new IntPtr(metaSize));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

As far as I can see, CLR Metatype does not containGCHandle, so it does not needGetManagedObject to work on its instances. So I think my implementation linked above is correct. PerhapsPyCLRMetaType should not be a managed object then?

Copy link
Member

@lostmsulostmsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think it is good, except the_PyObject_GC_Calloc, which should be replaced as suggested.

Comment on lines +538 to 542
if (obj.RefCount > 1)
{
obj.FreeGCHandle();
Marshal.WriteIntPtr(obj.pyHandle, ObjectOffset.magic(obj.tpHandle), IntPtr.Zero);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is this similar totp_clear in any way? I would suggest to extract these two updates (FreeGCHandle and zeroing out) into a method.

The point beingtp_clear has certain semantics, so it is easier to understand code containing a call to it, rather than has same actions inline.

Also, ifRefCount will correctly return 0, I'd move thisif below.


Marshal.WriteIntPtr(type, TypeOffset.tp_base, py_type);
IntPtrtype = Runtime._PyObject_GC_Calloc(new IntPtr(metaSize));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think it makes sense, that CLR metatype isnot a managed type. In my mind managed type == type, whose implementation is managed == has a managed object, that implements functionality == hasGCHandle pointing to that object. CLR metatype does not have aGCHandle.

@lostmsu
Copy link
Member

I believe this has been superseded by one of the major changes in 3.0.

This works on the latest master:

>>>importclr>>>fromSystemimportUri>>>u=Uri('http://ddg.com')>>>importweakref>>>weak=weakref.ref(u)>>>u<System.Uriobjectat0x000001ED729D8240>>>>u.ToString()'http://ddg.com/'>>>weak().ToString()'http://ddg.com/'

lostmsu added a commit to losttech/pythonnet that referenced this pull requestApr 8, 2022
filmor pushed a commit that referenced this pull requestApr 9, 2022
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@lostmsulostmsulostmsu requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

4 participants
@amos402@codecov-io@lostmsu@filmor

[8]ページ先頭

©2009-2025 Movatter.jp