Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork26k
MNT Refactor_average_weighted_percentile to avoid double sort#31775
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Conversation
github-actionsbot commentedJul 17, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
| result = xp.where( | ||
| is_fraction_above, | ||
| array[percentile_in_sorted, col_indices], |
lucyleeowJul 17, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I initially thought this should bepercentile_plus_one_in_sorted as from the paper, when g>0,but butsearchsorted defaults to left (equals is on the right), whereas the paper definedj <= pn < j+1searchsorted effectively givesi-1 < pn <= i whereas the paper hadj <= pn < j+1. This means that whenpn is greater than the LHS,searchsorted'si equalsj+1, from the paper.
When the quantile exactly matches an index,searchsorted'si equalsj, from the paper (as the equals is on opposite sides in paper vssearchsorted).
Reference Issues/PRs
Supercedes#30945
What does this implement/fix? Explain your changes.
Refactor
_average_weighted_percentileso we are not just performing_weighted_percentiletwice, thus avoids sorting and computing cumulative sum twice.#30945 essentially uses the sorted indicies and calculates
_weighted_percentile(-array, 100-percentile_rank)- this was verbose and required computing cumulative sum again on the negative (you could have used symmetry to avoid computing cumulative sum in cases when fraction above is greater than 0 - i.e.,g>0from Hyndman and Fan)I've followed the Hyndman and Fan computation more closely and calculate
gand just usej+1(since we already knowj). This did make handling the case wherej+1had a sample weight of 0 (or when you have sample weight of 0 at the end of the array) more complex.Any other comments?