As far as I can see,PyHandle does not add much overIntPtr, exceptToString overload, which is helpful for debugging and conversions to and from the new*Reference types. In every other way it is the sameIntPtr type.

However, implicit conversions destroy the whole point of*Reference types, which is to track lifetime of objects received from and given to Python interpreter. Automatic conversion toPyHandle, which can later be stored elsewhere or passed to Python allows it to potentially outlive the object it is pointing to.

There are two reasons I did not add more explicit type toint PyList_Append(BorrowedReference pointer, IntPtr value);:

1. documentation does not explicitly say if it does or does notIncRef it internally, e.g. it is unlcear if aBorrowedReference will suffice forvalue, or we have to pass aNewReference, and somehow have itconsumed.
2. If it would have to be aNewReference, how can we ensure it isconsumed? E.g. how we can ensure that afterextern Py_SomeFunc(NewReference value) thevalue is set tonull. It is a problem you can help on. Ideally we could find a way to make PInvoke do it for us.

Overall the way I see correct refcount handling could work is:

• keep usingBorrowedReference for something that was borrowed from Python. It can not outlive the current .NET function (which is to say generally is insufficient, but better than no guarantees at all).
• useNewReference to distinguish references we have toDispose after use
• usePyObject for anything else, as it allows to safely store objects indefinitely (e.g. outside of the current function scope), including in collections
• we can make specialized collection(s) for borrowed references when needed (this is only when having a collection ofPyObject would have too much impact on performance). Those collections would have to have the same lifetime restrictions asBorrowedReference.

Also, I started with little conversions to*Reference so we could see pain points and address them before jumping on using*Reference types for all PInvoke. If you face one, open an issue, describe your scenario, and give a code example where*Reference types do not work well for you, and we can discuss how to fix it.

As far as I can see, PyHandle does not add much over IntPtr, except ToString overload, which is helpful for debugging and conversions to and from the new *Reference types. In every other way it is the same IntPtr type.

That's what I wanted, we need a equivalent type forIntPtr to distinguish the Python object and raw pointer for memory or other usage(e.g. size_t) and provides some basic method for Python object for OOP.

I don't like current usages of theNewReference andBorrowedReference, it only increase the verbose code. I barely don't want to use them at all.
Just as I said, people who use these API should know what value their returned, a<summary/> comment should be enough.

varobj=PyList_New();// do something// return and didn't call Decref or Dispose

As a result, the same as you useIntPtr, didn't help much at all.
On my opinion, I didn't need these useless things, they only increase my use-cost, I can do well as well as use PyHandle even IntPtr.

@amos402 actually, I recentlyupstreamed a change to FxCop, that will make your example fail at compile time because you did not callDispose. Once the new version of the analyzers package is released, we can add it to our build.

But even without it, the new types already prevent accidental copying without corresponding increfs.

That means I even can just return the returned value. I have to incref it first, and store its pointer fromDangerousGetAddress and callDispose to make it decref, then I can return it. One simple step become four steps.
That could be more worse.

@amos402 no, it does not. Static escape analyzer is smart enough to extend scope when the value is returned.

You can actually see how this works right now with[NonCopyable] onNewReference.

I don't like thatNonCopyable thing cause I just want to put the object into collection directly.

cause I just want to put the object into collection directly

Can you show a scenario in Python.NET code?

@amos402 I meant a full method, that does something useful.

BTW, the first X should not fail. I thinkNewReference needs aBorrow() method, that would returnBorrowedReference, or, perhaps, even an implicit conversion (I am still torn about this one).

The second X fails because we use C# 7.3. It would work in C# 8.0, which we could switch to after 2.4 is released.

I thought the usages above are just some general usages, but it can't do it.

If you need the concrete usages, like these:
Chains call: X
https://github.com/amos402/pythonnet/blob/66ab7192222e382faf6cff808a23964eb846fc34/src/runtime/runtime_data.cs#L248-L269

Generator call: X
https://github.com/amos402/pythonnet/blob/66ab7192222e382faf6cff808a23964eb846fc34/src/runtime/runtime_state.cs#L146-L159

@amos402 the first example will have only oneDangerousGetAddress after we add an implicit conversion fromNewReference toBorrowedReference, and update PInvoke signatures.

The second scenario is indeed dangerous, and I feel like it is good to have it indicated as such. You are serializing pointers, that would live across domain boundary and Python interpreter instantiations.

BTW, the first X should not fail. I think NewReference needs a Borrow() method, that would return BorrowedReference, or, perhaps, even an implicit conversion (I am still torn about this one).

Why you have to specific they're New or Borrowed precisely when they as parameters, wouldn't it make confuse to others? Is not easy to treat the parameters as PyHandle and returnNewReference orBorrowedReference? Of course, for better usage, I tend to make them can be convert a PyHandle automatically.

The second X fails because we use C# 7.3. It would work in C# 8.0, which we could switch to after 2.4 is released.

Please don't rely on the language version much.

@amos402 most parameters areBorrowedReference, but, for example,PyList_SetItemsteals the reference to theitem. So we can mark it asNewReference, and ensure, that PInvoke will not allow us to useitem after it is passed toPyList_SetItem.

That means it may confuse the concept of new reference and borrowed reference, steals the reference means it just would not add reference count, now you say the parameter is a new reference object, things become jumble.

@amos402 I think it is pretty clear. If a parameter isBorrowedReference, you can pass a borrowed reference to it. If it is aNewReference, you have to "create" a new reference to the object you want to pass, internally method will steal it. Usually we would do it by incref'ing the object.

BTW, opened a PR for allowing to borrow from aNewReference#1088, which should removeDangerousGetAddress from your first example, except 1 instance when you put it into a collection.

varxxx=PySysGetObject("xxx");// borrowedPyTuple_SetItem(t,xxx);// no no no, you're passing a BorrowedReference, you need to make it to NewReferencevarxxx=PyObject_GetString(obj,"list");// newPyList_Append(xxx,obj);// no no no, you're passing a NewReference, you need to make it to BorrowedReference. Oh, seems it can be upgrade automatically, fine.PyList_SetItem(xxx,xxx);// no no no, you're passing a NewReference, you need to make it to BorrowedReference. Can you see that, it can be upgrade automatically, OK... Wait, `xxx` is what, it's a new or borrowed, or something else..... Now then, let's check the manual and ignore what the signature said.....

You are increasing the use-cost.

For reference stealing we can have another non-copyableref struct StolenReference, and add

publicStolenReferenceSteal()=> ...

Methods to bothBorrowedReference andNewReference.
Then you'd use it like this:

PyList_SetItem(listReference,index,itemReference.Steal());

Alternatively, we could find a PInvoke trick, that would allows us to useNewReference directly, and somehow automaticallynull it afterwards.

Oh no, another type, that's what am saying, things become jumble. Do we really need this? I though just enough by using PyHandle, no tricks.

varxxx=PySysGetObject("xxx");// borrowedPyTuple_SetItem(t,xxx);// no no no, you're passing a BorrowedReference, you need to make it to NewReference

And that's exactly the point! You have to construct a suitable new reference forxxx here. You missed anIncRef!

varxxx=PyObject_GetString(obj,"list");// newPyList_Append(xxx,obj);// no no no, you're passing a NewReference, you need to make it to BorrowedReference. Oh, seems it can be upgrade automatically, fine.

As you say, can be fixed by#1080.

PyList_SetItem(xxx,xxx);// no no no, you're passing a NewReference, you need to make it to BorrowedReference. Can you see that, it can be upgrade automatically, OK... Wait, `xxx` is what, it's a new or borrowed, or something else..... Now then, let's check the manual and ignore what the signature said.....

That scenario I am still thinking about. With theStolenReference type above this would look like:

PyList_SetItem(xxx,xxx.Steal());// a person reading code can see reference is stolen without referring to Python docsPyList_SetItem(xxx,xxx);// <- won't compile! excellent: you forgot to IncRef xxx!

And that's exactly the point! You have to construct a suitable new reference for xxx here. You missed an IncRef!

I mean, I wouldn't say that's a NewReference, why it named New? It's not NEW.

All I just want to use the Python API just as write the C/C++ extension. People who use these API should always care about the reference. Since there's no smart pointer here, the operates handle the reference count does necessity, you can control them automatically either. But these types make code become verbose and ambiguous.

@amos402 , again, this part (where parameter is to be stolen) is not settled yet.
Ideally, I'd like it to look either like this:

PyTuple_SetItem(t,xxx.NewReference());

or like this:

PyTuple_SetItem(t,xxx.Steal());

But I am afraid we'd have to resort to something likePyTuple_SetItem(t, ref xxx), becausexxx here has to be nulled after use, but only if call does not fail.

Return value is just fine, but introduce them into parameters, I refuse.
BorrowedReference PyList_GetItem(PyHandle pointer, IntPtr index)
int PyList_SetItem(PyHandle pointer, IntPtr index, PyHandle value)
This's enough.

@amos402 any specific reasons?

I have one for the distinct types: when I read a pull request, I'd like to avoid looking up every single Python function call I encounter in docs to ensure the pull request author countedIncRefs andDecRefs correctly.

Yes, the type checking may be a little help for that. But it should not stuck the general usage. So I tend to use implicit operator to make them become PyHandle, user should know that they're just using the raw pointer when they use it, because you have to declare it asPyHandle op = PyList_New(10), notvar, notNewReference, no need to call that long verbose functionPyHandle op = PyList_New(10).DangerousGetAddress().
And if you mustDispose before leave the scope, when I want to put a IntPtr into collection, I may want tosteal it, so I must incref it first then callDispose to decrease it, these +- operates become meaningless.
Be ware the reference count does necessary even you use these wrapped types. Generally, I don't need to check the manual everytime, you can find there's a priciple for making the return value as NEW or BORROWED.
Steal reference only appeared by fewXXX_Set container APIs, it means it transfer the ownership to the container, not mean it should a New, also if you introduce a StoleReference, hard to explain why this can represent as a type. You can think all parameters are borrowed. If you do too much things for the signature, that may interfere the general usage.

All I just want to use the Python API just as write the C/C++ extension. People who use these API should always care about the reference. Since there's no smart pointer here, the operates handle the reference count does necessity, you can control them automatically either. But these types make code become verbose and ambiguous.

I'm sorry, but no. This mindset got us exactly all of those reference count issues we (and in particular you) have been working on for years now to resolve. There is no shame in getting the compiler to help out on this. Naming can be discussed, but with this the code becomes /less/ ambiguous because we don't (ab)useIntPtr for semantically different things. Withvar you won't see much of a difference in verbosity either.

I have to declare that I didn't critical the Rererence types.

• Return values for New/Borrowed references?
  I agree with that. I never said no.

• Why adding this PyHandle?
  I have no intend to replace New/Borrowed reference with it for the return values. This is a equivalent type for IntPtr, it's use for replace IntPtr for OOP and prevent ambiguous from other pointer types.

• Why PyHandle can be implicit convert from IntPtr?
  They're equivalent.

• Why replace all IntPtr(PyObject* in C) into PyHandle?
  I don't want to interfere current code too much. Old code can be compatible without any modifies. Besides, it's easy to batch replace IntPtr types for the P/Invoke signatures.

• Why PyHandle can be implicit convert from New/Borrowed reference types?
  I don't want to interfere current code too much. Even you change the signatures of return values to reference types, it don't need any changes.
  Besides, it's for the easy usages. I said, PyHandle is equivalent for IntPtr. To trigger the implicit converter, you have to declare the variable type explicit, you won't get a PyHandle when you just usingvar. If you did that, that means you knew it became a raw pointer, and you have to take care of its lifecycle.
  At another point, you can say IntPtr/PyHandle/NewReference/BorrowedReference are all equivalent.
  And I can easy put them into clr container. No matter what they were, you still need to care about the reference count.
  Of course, if you really really hate the implicit operator, you can add a move segment instead it latter.

• Why PyHandle implemented IDisposable?
  So it can be use by using statement replace those verbosetry {} finnaly{ decref }.

• Why refactor the XIncref/XDecref?
  Just write that incidentally(not sure I'm using the write word). You can see it's faster than before.

• Why add Incref/XDecref methods for PyHandle?
  So it can easy to use them when you want to assert the pointer isn't null. For reducing a custom assert checking orif (is null) throw.

• Parameters for New/Borrowed references?
  Not agree. I suggest limit them only in caller's scope except return it. Use them as return value for C# method is fit, I think.

• Introduce a new type for those function of steal reference?
  I object insistently.

• Other suggests?
  DangerousGetAddress is too long for type and read, make aHandle property instead.

p.s. If I used any wrong English grammar or words before, please point out them if it won't bother you too much😝.

We are not taking this change.