Is there a reason that enablingPyDataType_REFCHK doesn't work for you?

Perhaps renamingREFCHK to beNONTRIVIAL_COPY would be useful?

Hmm... My understanding, from having looked at some of the NumPy v1.11.0 source (what I'm using on Ubuntu), is thatREFCHK implied thatPy_INCREF /Py_DECREF could be called on the given object; in this case, the underlying memory of anAutoDiffXd dtype is not a Python object, but rather than the direct C++ memory, so it could cause issues.

Looking atmaster now, it seems thatPyArray_INCREF andPyArray_XDECREF should work sincetypenum != NPY_OBJECT; however, routines likePyArray_FillObjectArray, used fromPyArray_Zeros (_zerofill) andPyArray_Empty (used bynp.full) appear to default to assuming that the memory is based onPyObject* (viaNPY_COPY_PYOBJECT_PTR), which would cause segfaults (TTBOMK). Overall, it seems the policy forREFCHK is applied inconsistently?

Will still try it out on my test case to see if it works out, and post the results here. Thanks!

EDIT: Checked against v1.11.0 (unpatched) from system, and it works fornp.copy, but segfaults when I usenp.zeros. Will see how it works withmaster.

EDIT 2: Confirmed thatnp.zeros still segfaults againstmaster (1.15.0.dev0+3ec8875) :(

If these inconsistencies were fixed, thenNONTRIVIAL_COPY would definitely be useful! However, perhaps it could be namedNONTRIVIAL_CTOR_DTOR_COPY to also be general enough to address#10721, as reference counting is just a general case of construction / destruction? (assuming there is space in user dtype API to add hooks for construction / destruction, or just some capsules that can store those methods in thedtype or class?)

That also being said, if there is a hook to causenp.zeros to fail fast with an exception and not a segfault, then it'd be easier to direct users to use casting (e.g. rather thannp.zeros(..., dtype=AutoDiffXd), usenp.zeros(...).astype(AutoDiffXd)). Is there a way to do this?
(At present, on thepybind11 branch, I have to do this fornp.ones, as v1.11.0 uses some esoteric casting fromnp.int64 or something to implementnp.ones...)

EDIT 3: I may risk the segfaults withnp.zero, as long as that's the main thing that fails with this approach. Will test out a few more things and see if this is a viable route to avoid the need to fork to achieve the desired functionality.
EDIT 3.5: Seems thatnp.empty also suffers from the same behavior - added an edit above. So things likenp.full will also cause segfaults. It could be mitigated by replacingnp.full(..., AutoDiffXd(10)) something likex = np.zeros(...).astype(AutoDiffXd); x[:] = AutoDiffXd(10), but that seems to be stretching it a bit.

Looking again - seems you're right,REFCHK isn't appropriate at all.

Unfortunately, thedtype->flags field is achar, and all 8 flags are currently used.

So maybe this is the best call. I worry that there will be a lot of places that just assume they can memcpy a user dtype though, so this will be the first of many changes you need to make.

Why is it that a memcpy doesn't work for you?

Ah, gotcha. For the most part, though, it seems that the main algorithms do well in using the correct dispatches to copy arrays. Though I'm sure I'll run into that issue at some point. It'll be an interesting adventure, then :P

Why is it that a memcpy doesn't work for you?

In this case,AutoDiffXd (Eigen::AutoDiffScalar<Eigen::VectorXd>) has itsvalue() stored immediately in the structure (which is trivially copyable), while itsderivatives() are stored usingEigen::VectorXd, which is not trivially copyable since it has a pointer for its allocated data, and thus can't bememcpyd as the memory will be incorrectly aliased (and will later cause a double-free if the original and the copy have their destructors called or are assigned over).

EDIT: This may be jumping the design gun, but in the future, couldPyArray_Descr::typeobj be used to store the additional metadata (e.g. ctor / dtor flags, ctor / dtor functions (capsules), etc.)? Or perhaps thedtype.metadata field?

Just checking back in on this PR - may I ask if there's anything I may do to increase confidence in its change to see if it can be merged intomaster? (testing, etc.?)

Perhaps instrument some user-defined dtype test to check which user-defined methods have been called?

I think this is probably fine as it. The change is pretty straightforward, and I assume your code works with it.

There's a slight performance penalty here for custom dtypes, but I imagine it's pretty minor - those users can presumably just callPyArray_GetStridedCopyFn internally, at which the cost becomes just a few extra layers of indirection, which doesn't scale with array size.

If no one else has any opinions on this, I'll merge it in a few days. Tagging with 1.15 so I don't forget.

Awesome, thanks!

There's a slight performance penalty here for custom dtypes, but I imagine it's pretty minor - those users can presumably just call PyArray_GetStridedCopyFn internally [...]

One concern is it looks likePyArray_GetStridedCopyFn is decorated withNPY_NO_EXPORT /NPY_VISIBILITY_HIDDEN - will users be able to call this method?
Is it something that should be part of the exported API, or perhaps offered as a separate, internal capsule? (likenp._low_level["PyArray_GetStridedCopyFn"]?)

EDIT: Er, just to clarify, I don't think this PR should be blocked on account of a convenience method being user-accessible; for the time being, it seems like it's more-or-less trivial to write a non-swap version ofcopyswapn; as a simple example:
https://github.com/EricCousineau-TRI/pybind11/blob/e712b18acc86bd8c61df7c92433f1299d127236e/include/pybind11/numpy_dtypes_user.h#L700

@eric-wieser Just a small ping, if you're able to merge it - thanks!

(Would love to employ this onpydrake to parallel the C++ convention of scalar types for converting systems fromdouble toAutoDiffXd andsymbolic::Expression!)

it seems like it's more-or-less trivial to write a non-swap version of copyswapn

My point is thatPyArray_GetStridedCopyFn has gone to great lengths to try and be efficient, and a user implementation is unlikely to do so. Not a big deal though.

Can you perhaps add something to the release notes about this change?

Yup, can do!

Should I file an issue to consider exposingPyArray_GetStridedCopyFn publicly, so that user-defined dtypes can leverage it?

Done. Added an initial pass at release notes; please let me know what you'd like me to change.

@eric-wieser Ping.

LGTM.@eric-wieser I'll leave this to you.

Thanks@EricCousineau-TRI