I think we should lead withWhy _not_ User-Defined Functions. While performance is called out down below, I think the poor behavior of UDFs should be mentioned as well. Namely that pandas has no information on what a UDF is doing, and so has to infer (guess) at how to handle the result.

In particular, I think it should be mentioned that none of the examples on this page should be UDFs in practice.

Can you also make a mention of resample, rolling, expanding, and ewm. Perhaps link to each section in the User Guide.

Can we add the other objects to this note, it seems to me they all belong together.

nit: "change the data differently" sounds very close to mutating in a UDF, which we explicitly do not support. What do you think of "behave differently".

“Behave differently” sounds clearer and avoids implying mutation. I'll update it!

I'm thinking we should removegroups of data here.DataFrame.apply that you're referencing doesn't operate on groups, and you mention groupby below.

Can we add the other objects to this note, it seems to me they all belong together.

Things like.agg(["sum", "mean"]) aren't UDFs, so I don't think they should be mentioned here, and it could be make users think these types of usages are slow (they are not).

This isn't an example of a UDF. I really like your example of using linear regression - can we do that here? It's a bit unfortunate that groupby.transform does not allow operating on the entire group (only works column-by-column) here.

fromsklearn.linear_modelimportLinearRegressiondf=pd.DataFrame({'group': ['A','A','A','B','B','B'],'x': [1,2,3,1,2,3],'y': [2,4,6,1,2,1.5]}).set_index("x")# Function to fit a model to each groupdeffit_model(group):x=group.index.to_frame()y=groupmodel=LinearRegression()model.fit(x,y)pred=model.predict(x)returnpredresult=df.groupby('group').transform(fit_model)

Not sure if Sphinx is more flexible now, but this had to be the same exact length as the title before.

Title marker needs to be at least as long as the text, but can be longer.

Maybe just personal opinion, but to me it makes more sense to explain what UDFs are in pandas before explaining when not to use them. This order seems reasonable assuming users already know what pandas udfs are in practice, but I'd personally prefer not to assume it in the user guide for UDFs.

In my opinion, after the previous introduction which is great, I'd show a very simple example so we make sure users reading this understand the very basics.

Something like:

defadd_one(x):returnx+1my_series=pd.Series([1,2,3])my_series.map(add_one)

Building on top of this, like then showing the same with aDataFrame, at some point showing UDFs that receive the whole column with.apply... should help make sure users are following and understanding all the information provided here.

I am a bit negative here. This is duplicating a lot of other documentation that we already have. I think we should instead link to that documentation.

Do you mind pointing out to an specific example@rhshadrach? I found documentation for the aggregate functions, but not much for themap,apply... onSeries andDataFrame other than in the API docs. I agree with not having much duplication. Personally, if there is few here and there like in the FAQs, Performance page... I'd rather have the docs related to these methods in this page, as it feels like the natural place, and link to the sections here in the FAQs, performance hints, groupby user guide... Of course there can be cases where it makes more sense the opposite, but maybe we can discuss the specific cases where there is duplication.

apply:https://pandas.pydata.org/docs/user_guide/basics.html#row-or-column-wise-function-application
map:https://pandas.pydata.org/docs/user_guide/basics.html#applying-elementwise-functions

I'd rather have the docs related to these methods in this page, as it feels like the natural place

If we are going to move the docs on e.g.DataFrame.agg here, then this no longer is a page just about UDFs asDataFrame.agg does more than just use UDFs. In addition, that seems like a large reworking of the docs for little (in my opinion, actually negative) benefit.

I totally missed the Essential basic functionality page, thanks for pointing that out. Fully agree with you that what I proposed here is repeating again the wholehttps://pandas.pydata.org/docs/user_guide/basics.html#function-application section . And I agree that's not a good idea.

Personally, I'd rather not have that section, and have that content here. At least in my experience, map and apply are common, but not essential as other parts described in that page. And also, I think the structure of the user guide will be clearer and easier to find things with the changes.

For theDataFrame.agg, there is already a groupby page, and I think just having the methods in the lists of methods that support udfs would be good, and then just a mention that points out to the group by page where all the detail explanation regarding groupping is presented with examples.

There may be other structures, but what I'd like is that we can give users structure to the related methods. I thinkSeries has around 200 methods and attributes. Users having to navigate that whole API to find out themselves that map, apply and pipe are kind of the same just changing the input of the udf, doesn't seem ideal. I think this page here can really help in that.

What do you think?

Personally, I'd rather not have that section, and have that content here.

If we move the main document ofapply here, then I am quite opposed to calling this a page on UDFs as apply does more than just take UDFs. By documentingapply("sum") et al here, it seems to me we make this page far less clear than leaving it as solely UDFs.

In any case, is that something you think should be tackled in this PR? This PR started as

A dedicated page in the users guide that guides users on when to use udf, a general idea of the API, the differences between the different methods, the options available... seems a better idea.

I do not think we should morph it into moving around documentation from other places, especially when there are disagreements.

Users having to navigate that whole API to find out themselves that map, apply and pipe are kind of the same just changing the input of the udf, doesn't seem ideal.

Which is why I think this page should be a comparison of UDF methods (as it mostly is now), while pointing to more thorough documentation elsewhere in the User Guide.

Fair enough, I think I understand your point better now. Maybe I'd like to improve a bit the apply/maps docs in essential, but that's unrelated to this PR. And happy to move forward here focussing on the UDFs and not on the methods, as you describe.

What do you think about having this as a table? Personally I think it should make it easier to understand the differences about the methods. As a general idea:

Not sure if it makes sense to combine with the table below.

Yeah I agree with you, thanks for the suggestion! I will keep the two tables separate for now

I'm unsure on havingfilter here for now. I think it's very good that you added it, as it doesn't support udfs, but it probably should. So, it opens a discussion we probably want to have about adding them.@rhshadrach thoughts?

I suspect the reason this was added is thatDataFrameGroupBy.filter does accept UDFs. Perhaps that should be mentioned instead?

I actually thinkDataFrame.filter should accept Boolean masks, similar to PySpark and Polars. But agreed that discussion is not for here!

Maybe just personal preference, but these last 3 sections seem to be talking about the same (performance), I'd have just a section about performance.

I'd keep it short for now, and we can iterate over it later. The reason is that each time we review this before merging it we need to re-read the whole document. So, if we can finish the main part above first, and have this as a placeholder, then in a second PR we can focus more on performance without having to keep reviewing the first part again.

I think this comes from my suggestion, but checking the descriptions now, feels like it'd be helpful to use the use the same terminology, to show how similar both functions are. Meaning that if we useApply a function to each column, I think it'd be helpful to useApply a function to each element. Feel free to disagree, and I see the point on using the name of the method in the description. But personally, I think it'd be good to highlight how similar these methods are, and let users understand the difference easily and quickly, and I think what I'm proposing should help with that.

Transform is actually very similar to apply. The function input is a column or a row depending on axis. And the output is also a column or a row (exactly same as apply). The only difference is thatapply allows to return a shorter or longer dataframe, while transform will raise. See this example:

# apply works fine when the output still have 3 samples:>>>pandas.DataFrame({"points": [100,30,50]}).apply(lambdax:pandas.Series([1,2,3]))points011223# transform also works fine>>>pandas.DataFrame({"points": [100,30,50]}).transform(lambdax:pandas.Series([1,2,3]))points011223# apply is still happy now that we removed one of the samples:>>>pandas.DataFrame({"points": [100,30,50]}).apply(lambdax:pandas.Series([1,2]))points0112# transform is not happy:>>>pandas.DataFrame({"points": [100,30,50]}).transform(lambdax:pandas.Series([1,2]))Traceback (mostrecentcalllast):File"<stdin>",line1,in<module>File"/home/mgarcia/src/pandas/pandas/core/frame.py",line10269,intransformresult=op.transform()^^^^^^^^^^^^^^File"/home/mgarcia/src/pandas/pandas/core/apply.py",line356,intransformraiseValueError("Function did not transform")ValueError:Functiondidnottransform

This is more useful with groupby, and I'm not even sure ifDataFrame.transform is that useful, it probably mostly exist for consistency.

For now I'd remove the function input/output fields, as there is no function.

Maybe worth mentioning and comparing also.pipe, which is both vectorized and a udf?

apply andtransform are almost the same, I'd also have it twice for axis=0 and axis=1 like apply.

Personally I'd only keep the first table. I think the description can already explain anything not clear present in the second table.

And I'd make this sectionchoosing the right method part of theMethods that support User-Defined Functions. To me it feels like we're having two sections for mostly the same.

vectorized?

I'd remove this for all methods, as the title section and every mention ofapply will already be a link.

I don't think map is more performant than apply, in general apply will be more performant. I'd remove this as a reason, as it depends a lot on the use case, and I don't think this is a general rule.

Got it. Just curious: Why is applying generally more performant than mapping in practice? I thought map was faster for simple element-wise UDFs.

In map, without anything special, pandas should necessarily loop in Python over the elements, casting each to a PyObject. In apply the same can be true, but since a whole column is given to the user, the user could also use vectorization. Like, a simple function doing ax + 1 would be fast when using apply and slow when using map. That's why I wouldn't say map is faster.