I'm a really impressed by all the work done. It gives a good idea of all the different parts of the code that have to change to port the first set of functionalities of MFEM to GPU. From the MFEM core side, I think most of the changes are not too disruptive. I really like that the device/backend specific functions are exposed and a developer doesn't have to guess what has to be written, the level of abstraction is at a granularity where it is easy for a developer to understand what the backend functions have to do. I think l would feel confortable writing a backend in this code. However, I think the design should be pushed a little bit more in the direction of the separation of what is device specific and what is backend specific.
I would give the following definitions of device specific and backend specific:

• Device functions are meant to be uniquely implemented for a specific device. If a better implementation is found for a device function, it should replace the current implementation. For instance, all backends should use the same implementation ofdot on a GPU (I think). Device functions assume that there exists a best implementation on a device usable for all the different use cases.
• Backend functions are meant to have multiple implementations for a specific device. Backend specific functions are selected by the user and are expected to have different performance on different problem configurations.

Instead of having everything inkernels files, maybe having some indevice files and other inbackend files would help clarifying this. And showing how the differentbackend functions could be selected would be really interesting (amap<Backend, FunctionPointer>?).
Havingbackend files would also allow to directly see what needs to implemented by a backend.
Going even further in this design, I think it is important that abackend can only propose the implementation of a subset of all thebackend functions, and either select whichbackend to use when a backend function is not implemented, or use a default backend.

Regarding theArray,Vector classes, I would like to see if there is no impact on performance of the double pointer when some computations are achieved on the CPU. If there is a negative impact on performance, we might want to consider keeping themfem::Array andmfem::Vector classes as they are and make them compatible with the proposedArray andVector classes, they would have to be renamed toDeviceArray andDeviceVector (or something else). These classes would have a more limited set of functions (that could grow when needed) that all work both on gpu and on cpu.

I personally find thememoryManager class a bit scary, and the different macros not so naturals to use. I have difficulties seeing the consequences of such a class as it is operating at a very low level. I think it could be worth introducing this class in a way that we could backtrack in the future, or replace by something else.

I think that thePABilinearForm is going in the right direction but a lot still needs to be done if we want to be able to propose the different levels of assembly: matrix free, partial assembly, local matrices, global matrix. This could have a non negligible impact on the rest of the design though.

Congratulations@camierjs , impressive work done in such a small time frame!