I think the helpful first step may to be convert the file to C++ to continue in steps from there. These custom macros seem a step in the wrong direction, since if we are to replace custom templating with C++ templating, we should use that.

I'm going to do that in another PR because apparently is not as trivial as I thought. converting fromscalartypes.c.src toscalartypes.cpp.src

edit: I pushed the changes to review, and because this will have a dependency on it. Just confirm me if you want this in separate PR for easy review and to avoid noise on this

edit 2: well apparently for the migration to C++ in windows it requires C++ 20 :/. do you have a suggestion? I think it is not suitable to do it without designated initializers{ .tp_name = val }

error C7555: use of designated initializers requires at least '/std:c++20'

edit 3: learning stuff: struct factories at top level and compile time. introducemake_pyarr_type

edit 4: removed all designated initializers just trying to make it compile in MSVC so that I can mark as "completed" the migration to C++.

nicee. finally C++ compiles jaja!

Just confirm me if you want the migration to C++ in a separate PR for easy review and to avoid noise on the removal of.src x2

edit 5: cannot make factories for array of methods because a) trying to use static variables inside a constexpr is not allowed and b) returning a stack address associated with a local variable is not something we want.

a)definition of a static variable in a constexpr function is a C++23 extension
b)address of stack memory associated with local variable 'methods' returned

because of that, I will just repeat the definition of those

edit 6: cannot useconstexpr std::map until C++26 to avoid using a chain ofif-else constexpr

edit 7: cannot usetemplate <const char* mystr> to create function factories. nor I can make a function return another function (lambda). will need to make definition of each function that wraps the generic one

I think it is not suitable to do it without designated initializers { .tp_name = val }

I think they are allowed to be written as long as they are in the exactly correct order? I am not sure how many holes they have, this is one annoyance with C++. If there are few we might also just fill them in explicitly, otherwise, I guess this tricky is fine.

Maybe it makes sense to keep going here for now and then see if we want to split out just the initial conversion for example. (A bit conflicting, I would prefer to split it out, but if we don't follow through with the full thing there is no point.)

For what it's worth, I think the changes you did beyond the main change are not far reaching enough to be nice, we need to go all-in on templates for things to make sense. (Almost) anything repeated should be a template instead, the bufferprocs are a good example, that should just be something likebufferprocs<typename T>.
(The gentype version is then the overload that kicks in if a more specialized version is not defined.)

EDIT: CC@athurdekoos for awareness (and maybe if you want to even review).

the bufferprocs are a good example, that should just be something likebufferprocs<typename T>

you want

static auto unicode_arrtype_as_buffer = generic_pybufferprocs<unicode_getbuffer>;static auto bool_arrtype_as_buffer = generic_pybufferprocs<dunder_buffer_impl1<npy_bool, PyBoolScalarObject>>;

instead of

static auto unicode_arrtype_as_buffer = make_pybufferprocs(unicode_getbuffer);static auto bool_arrtype_as_buffer = make_pybufferprocs(dunder_buffer_impl1<npy_bool, PyBoolScalarObject>);

?

No,unicode_arrtype_as_buffer need not exisst at all.

unicode_arrtype_as_buffer needs to live in the stack at the top level scope because later its pointer is passed to their type definition like this

// that's why I dostaticauto unicode_arrtype_as_buffer = generic_pybufferprocs<unicode_getbuffer>;// << top levelPyUnicodeArrType_Type.tp_as_buffer = &unicode_arrtype_as_buffer;// instead ofPyUnicodeArrType_Type.tp_as_buffer = &make_pybufferprocs(unicode_getbuffer);

and doing something like

PyUnicodeArrType_Type.tp_as_buffer = &arrtype_as_buffer<unicode_getbuffer>

I wasn't able to do it yet. Can you give an example for that?

I think the PR can be taken a first look. Remarking the following points

1. The most difficult part of the migration was the__new__ impls block, I would like a review on that
2. I want to get rid of the macros but I don't find a better way to do it right now
3. seberg comment about assigning a pointer to a templated struct. I don't know how to do it

ccing the top 3 commiters ofscalartypes.c.src@seberg@ngoldbaum@mtsokol

ping@athurdekoos as well

seberg comment about assigning a pointer to a templated struct. I don't know how to do it

Maybe check out whatjax/ml_dtypes is doing and see if you get inspired from a pattern. I don't really want to try around right now, but I think this has to happen in some form.
I would hope all explicit instantiation with a name is unnecessary with the exception a single global type probably.
(In part, focusing on this type of discussion is why it might be helpful to do just the conversion first.)