@jorisvandenbossche - I've merged main and pushed a commit here. If you have any objections, I can pull it off.

• Adds whatsnew to 2.3.
• Simplifies conditionals in a test.
• Fixes behavior of ArrowExtensionArray and adds tests for it.

For the last one, previously ArrowExtensionArray vs Nan-Python was giving back NumPy bool. This was the only case where ArrowExtensionArray was not resulting in ArrowExtensionArray.

This does not yet handle the case of comparison to object dtype.

object dtype looks correct to me here.

object dtype looks correct to me here.

Hmm, not entirely sure anymore what I meant with that object dtype was not yet covered. I thought maybe the case where the object dtype does not contain just strings, but also that seems to work fine

One case related to object dtype that is still failing is comparing with an object series that has mixed types:

In [3]:ser1=pd.Series(["a",None,"b"],dtype=pd.StringDtype("pyarrow",na_value=np.nan))In [4]:ser2=pd.Series(["a",None,2],dtype=object)In [5]:ser1==ser2...File~/scipy/repos/pandas/pandas/core/arrays/arrow/array.py:517,inArrowExtensionArray._box_pa_array(cls,value,pa_type,copy)514pa_array=pa.array(value,type=pa_type,from_pandas=True)515except (pa.ArrowInvalid,pa.ArrowTypeError):516# GH50430: let pyarrow infer type, then cast-->517pa_array=pa.array(value,from_pandas=True)519ifpa_typeisNoneandpa.types.is_duration(pa_array.type):520# Workaround https://github.com/apache/arrow/issues/37291521frompandas.core.tools.timedeltasimportto_timedelta...ArrowTypeError:Expectedbytes,gota'int'objectIn [6]:ser1=pd.Series(["a",None,"b"],dtype=pd.StringDtype("python",na_value=np.nan))In [7]:ser1==ser2Out[7]:0True1False2False

So with just object dtype, such a comparison works. And it also works with the python-backed string dtype. But fails with the pyarrow-backed string dtype, because in this case the comparison defers to the ArrowExtensionArray implementation, which tries to convert the other side to a pyarrow array, which is not supported for mixed types. While we generally (although in many cases definitely not best practice) mixed-types object dtype in pandas.

(but let's consider this for a separate issue/PR)

Thank@jorisvandenbossche and@rhshadrach

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 2.3.xgit pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 6177e2233974643d17d1560a4d61804800534900

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #61138: API (string dtype): implement hierarchy (NA > NaN, pyarrow > python) for consistent comparisons between different string dtypes'

4. Push to a named branch:

git push YOURFORK 2.3.x:auto-backport-of-pr-61138-on-2.3.x

5. Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR#61138 on branch 2.3.x (API (string dtype): implement hierarchy (NA > NaN, pyarrow > python) for consistent comparisons between different string dtypes)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove theStill Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free tosuggest an improvement.