@greglucas - theastropy.units error I think we would need to work around - as is, we don't really treatMaskedArray correctly and it shows here; we will just have to deal with it...
But theastropy.table error is more perhaps more serious:MaskedColumn is a subclass ofMaskedArray and it assumes it does not have to worry about__array_ufunc__. While we could likely define it and make things work again (indeed, that might be good), we may just be the canary in the coalmine: any otherMaskedArray subclasses may well break too.

Actually, I quickly tried to implement__array_ufunc__ myself onMaskedColumn, and it exposed a bug in the implementation here even fornp.ma.MaskedArray:

In [2]: np.signbit(np.ma.MaskedArray([1., 2.], mask=[True, False]))---------------------------------------------------------------------------TypeError                                 Traceback (most recent call last)Cell In[2], line 1----> 1 np.signbit(np.ma.MaskedArray([1., 2.], mask=[True, False]))TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(<ufunc 'signbit'>, '__call__', masked_array(data=[--, 2.0],             mask=[ True, False],       fill_value=1e+20)): 'MaskedArray'

The reason is that if the function is not defined innp.ma, you just hand off tosuper(), i.e.,np.ndarray.__array_ufunc__. But that one returnsNotImplemented because it does not know how to deal withMaskedArray. I think you'd have to extract the data and copy the mask over. Though perhaps easier is to definenp.ma.signbit!

p.s. Might want to check test completeness by checking all functions innp.testing.overrides.get_overridable_numpy_ufuncs() are covered...

Maybe worth pulling@seberg's comment in the main thread: a question is whether the regular numpy functions should just call thenp.ma ones, including the, to me, rather strange use of domains (seegh-22347), or rather do a simpler wrapping where masks of input arguments simply or-d together, without worry about the values of the inputs. Arguing against this of course is that one then is basically moving to a new class (which I did for astropy; seehttps://docs.astropy.org/en/latest/utils/masked/index.html and in particular the discussion of differences withnp.ma.MaskedArray).

But probably it is easier to just add the few missing ufunc tonp.ma and work with what you have!

p.s. Might want to check test completeness by checking all functions in np.testing.overrides.get_overridable_numpy_ufuncs() are covered...

But probably it is easier to just add the few missing ufunc to np.ma and work with what you have!

Thanks for pointing me to that function! Somewhat surprisingly, there are actually 30 ufuncs that have no implementation innp.ma currently! Most of them are easy, but I don't think there is any wrapper class that handles multi-output functions yet (ex:https://numpy.org/doc/stable/reference/generated/numpy.frexp.html)

I went digging through some of the old mailing list archives that I wasn't aware of and came across this really nice discussion:
https://mail.python.org/archives/list/numpy-discussion@python.org/thread/OSGGDTJX6XZK3LXLNQYGZP766FD3YFQQ/#QNVWET3O7KXFGAB7NVZNWUGHE766TGNL
and if I'm parsing it all correctly it seems like people are basically on the same page and working out the final details is all that is left. I had also forgotten that@ahaldane had pointed me at their implementation of a new class in one of the previous PRs:
https://github.com/ahaldane/ndarray_ducktypes/blob/master/ndarray_ducktypes/MaskedArray.py

My goal for this was to have a bit of a stepping stone before going all-in on a new class. However, I am now possibly seeing that this might just be playing a game of whack-a-mole that I'm sure will change some behavior and upset someone, so maybe we should just do a hard-breaking-change all at once instead of introduce these small papercuts. (i.e. work on the better implementation for a 2.0 release rather than adding these now)

I think one of the things missing from many of the discussions is that many downstream libraries make use ofnp.sum(),np.stack(), etc... without thinking about what instance is being passed into them (various arrays from numpy, pandas, dask, masked), so I do think it is important to try and start dispatching thenp.func tonp.ma.func specific methods for the Masked class since that seems to be the odd one out at the moment...

My desires from a new class:

1. No auto-masking of "bad" computation locations, let the user do that apriori if they want

I haven't seen anyone argue for this yet, and this looks like it goes way back in git blame.

2. No warnings when doing a computation of a bad value under a mask

It would be annoying to get divide by zero warnings on your computations when you've already pre-masked the data (see my example at the top of the PR for the motivation here!)

3. np.func gets forwarded tonp.ma.func whenever possible (implement__array_ufunc__ and__array_function__)

It is quite surprising right now to donp.stack(ma, ma) and get a Masked instance back, but with an incorrect mask! (it would be better to just get a plain ndarray IMO)

4. Proper subclass wrapping/heirarchy preservation

Should fall out of (3) if implemented properly with NotImplemented being returned.

I'll try and take a better look at@ahaldane's implementation in nd_duck_array and@mhvk's implementation in Astropy. I think right now we are missing having a clear path between where we are now and where we want to go, so something a little more concrete with actions of what is needed might be nice. I'll try and give it some thought.

Thanks, that's a great summary! There is only one item I'd add: think carefully how to deal with underlyingdata other thanndarray. For instance, part of the reason I made a separate implementation in astropy was that the currentMaskedArray simply cannot be used to createMaskedQuantity -- withMaskedArray holding aQuantity attributes like the unit become hidden (and I think@ahaldane's version did not make that easy either). I found it worked much better by havingQuantity on top, effectively holding a masked array as data (in our case, with corresponding class hierarchyMaskedQuantity(Quantity, MaskedNDArray)). But for something likedask, it might be better to have both the data and the mask be dask arrays, while for Xarray, that would not make sense.

Overall, it certainly is not crazy to move in small steps! I think a bigger question here is whether when wrapping regular numpyufunc, one wants to just get thenp.ma.<ufunc> equivalent, or something that moves closer to a final goal (i.e., as you note, no masking of bad elements but also no warnings for masked ones [note that the latter may be quite tricky...]).

Any news here?