I tested your code in two strategies:

• single branch benchmark: create new branch, download your code, run the following command.

  python runtests.py --cpu-baseline="sse2" --bench bench_linalg

  • TheNPY_SIMDis 128.

  python runtests.py --cpu-baseline="avx2" --bench bench_linalg

  • TheNPY_SIMDis 256.

• two branch benchmark: create two new branch, download your code in both branch, run the following command.

  python runtests.py --cpu-baseline="sse2" --bench-compare test_asv bench_linalg

  • TheNPY_SIMDis 128.

  python runtests.py --cpu-baseline="avx2" --bench-compare test_asv bench_linalg

  • TheNPY_SIMDis 256.

To sum it up, the code here works fine in my computer. Thanks@seiko2plus .

@Qiyu8, thank you for reporting, you can also count on#17737 to verify the supported features. it provides the same format that used within pytest.

I think this PR is good to merge and will unlock other performance optimizations.

While I do not debate the value in passing optimization flags to asv, I think this PR still is not ready. There should be a simpler and more direct way to get interaction between asv and code building.

Here are some ideas, it may be worth discussing them more widely:

• overhaul runtests.py and refactor it into proper subcommands rather than using home-grown argparse enhancements. This would allow refactoring the code, which might clarify the interactions between building NumPy and running benchmarks, and reveal a better path for this kind of code.
• enhance ASV and/or the interaction between the benchmarks, ASV, and NumPy. PRsMAINT, Benchmark: print the supported CPU features during the run of performance tests #17737,ENH: Standalone benchmark script for the inner loops of ufunc #15987 andENH: Provides a deep benchmark for universal functions #15992 are also related to this.

@mattip,

There should be a simpler and more direct way to get interaction between asv and code building.

unfortunatelyasv continuous doesn't offer any other way, checkairspeed-velocity/asv#803

Here are some ideas, it may be worth discussing them more widely

First, we need to take a decision to either accepting this pull request as a workaround or close it and revert#17284

overhaul runtests.py and refactor it into proper subcommands rather than using home-grown argparse enhancements.

as a start, we can create an independent scriptbenchmark.py that mapping all important args to ASV, it should
provides similar functionality togh-15987

enhance ASV and/or the interaction between the benchmarks, ASV, and NumPy. PRsgh-17737,gh-15987 andgh-15992 are also related to this.

Here is the issue, ASV is based on command arguments that kill the flexibility.
I vote to create our own micro-benchmarking tool and drop ASV.

EDIT: I mentioned a wrong pr

I think It's hard time to dropasvsupport, because this benchmark toolkit is working well among normal modules, The currentargbased simd mechanism exposed the drawbacks ofasv, which should improved throughcommunity.

@Qiyu8, with all respect, we can't count on another project or community in every single new feature. I seegoogle benchmark as a good alternative, we can build our new tool on top of it. It already provides decent flexibility and been used by many projects.

@mattip, there's no other way to get ride of it without this workaround. ugly but necessary and the same thing for#17737.
please I need to hear your final call.

I marked this for triage-review so we can get some more opinions.

@mattip, thank you Matti, I really appreciate your help.

The SIMD optimizations is stretching the current workflows:

• a need to benchmark ufunc inner loops
• a new templating engine for C code
• a new compiler option extension for setting machine options to C flags (merged)

Maybe we should think a bit about separating the loops out into a separate library that would be easier to test and benchmark, along the lines of issuegh-17698.

In the triage meeting we decided to merge this, but would like to rethink benchmarking in general. That is a different discussion though, so in this goes.

Thanks@seiko2plus

@mattip,

Infrastructure, SIMD, and GPU optimizations must be part of the core of any modern computing library, branching the core or counting on other projects will badly affect workflow, especially for big projects.

We have something new here, let's give it the full chance before thinking of separating it into a separate repo. I think NumPy should find a foot in highly computing processing instead of playing the mediator role, in order to maintain its popularity.

need to benchmark ufunc inner loops

We need also a tool for sanity data check too. OpenCV has 2 in 1 tool benchmarking, and sanity checks at the same time. seeHowToUsePerfTests very effective in detecting any deviations.

a new templating engine for C code

Why do we need a new engine and we have python? we just need a better way to using it in our C sources. I think#17952 can be a good solution.

In the triage meeting we decided to merge this,

Thank you a lot.

but would like to rethink benchmarking in general.

I'm not sure what we should do. To create our own minimal tool that serving our only needs or contributing to ASV. What do you prefer?

In general, I do not like re-inventing the wheel. If a tool or paradigm can be adapted to our needs, then that is preferable over creating our own, unless our needs are very different.

I think NumPy should find a foot in highly computing processing instead of playing the mediator role, in order to maintain its popularity.

I disagree. If by splitting out the great work you have done with SIMD we enable others to reuse it, that would be progress.

What do you prefer?

It seems like right now ASV and pyperf are very similar, but neither serves our needs very well. I think we should explore ways to make ASV better before creating our own tool. Making those tools better would help others who are also struggling with benchmarking.

It seems like right now ASV and pyperf are very similar, but neither serves our needs very well. I think we should explore ways to make ASV better before creating our own tool. Making those tools better would help others who are also struggling with benchmarking.

Well said, I very much agree 👍

@mattip, I will give ASV a try I guess we have enough burden and no need to open a new front.