- Notifications
You must be signed in to change notification settings - Fork74
Yet Another Python Profiler, but this time multithreading, asyncio and gevent aware.
License
sumerc/yappi
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
A tracing profiler that ismultithreading, asyncio and gevent aware.
- Fast: Yappi is fast. It is completely written in C and lots of love and care went into making it fast.
- Unique: Yappi supports multithreaded,asyncio andgevent profiling. Tagging/filtering multiple profiler results has interestinguse cases.
- Intuitive: Profiler can be started/stopped and results can be obtained from any time and any thread.
- Standards Compliant: Profiler results can be saved in callgrind or pstat formats.
- Rich in Feature set: Profiler results can show eitherWall Time or actualCPU Time and can be aggregated from different sessions. Various flags are defined for filtering and sorting profiler results.
- Robust: Yappi has been around for years.
CPython standard distribution comes with three deterministic profilers.cProfile,Profile andhotshot.cProfile is implemented as a C module based onlsprof,Profile is in pure Python andhotshot can be seen as a small subset of a cProfile. The major issue is that all of these profilers lack support for multi-threaded programs and CPU time.
If you want to profile a multi-threaded application, you must give an entry point to these profilers and then maybe merge the outputs. None of these profilers are designed to work on long-running multi-threaded applications. It is also not possible to profile an application that start/stop/retrieve traces on the fly with these profilers.
Now fast forwarding to 2019: With the latest improvements onasyncio library and asynchronous frameworks, most of the current profilers lacks the ability to show correct wall/cpu time or even call count information per-coroutine. Thus we need a different kind of approach to profile asynchronous code. Yappi, with v1.2 introduces the concept ofcoroutine profiling. Withcoroutine-profiling, you should be able to profile correct wall/cpu time and call count of your coroutine. (including the time spent in context switches, too). You can see detailshere.
Can be installed via PyPI
$ pip install yappiOR from the source directly.
$ pip install git+https://github.com/sumerc/yappi#egg=yappiimportyappidefa():for_inrange(10000000):# do something CPU heavypassyappi.set_clock_type("cpu")# Use set_clock_type("wall") for wall timeyappi.start()a()yappi.get_func_stats().print_all()yappi.get_thread_stats().print_all()'''Clock type: CPUOrdered by: totaltime, descname ncall tsub ttot tavgdoc.py:5 a 1 0.117907 0.117907 0.117907name id tid ttot scnt_MainThread 0 139867147315008 0.118297 1'''
You can profile a multithreaded application via Yappi and can easily retrieveper-thread profile information by filtering onctx_id withget_func_stats API.
importyappiimporttimeimportthreading_NTHREAD=3def_work(n):time.sleep(n*0.1)yappi.start()threads= []# generate _NTHREAD threadsforiinrange(_NTHREAD):t=threading.Thread(target=_work,args=(i+1, ))t.start()threads.append(t)# wait all threads to finishfortinthreads:t.join()yappi.stop()# retrieve thread stats by their thread id (given by yappi)threads=yappi.get_thread_stats()forthreadinthreads:print("Function stats for (%s) (%d)"% (thread.name,thread.id) )# it is the Thread.__class__.__name__yappi.get_func_stats(ctx_id=thread.id).print_all()'''Function stats for (Thread) (3)name ncall tsub ttot tavg..hon3.7/threading.py:859 Thread.run 1 0.000017 0.000062 0.000062doc3.py:8 _work 1 0.000012 0.000045 0.000045Function stats for (Thread) (2)name ncall tsub ttot tavg..hon3.7/threading.py:859 Thread.run 1 0.000017 0.000065 0.000065doc3.py:8 _work 1 0.000010 0.000048 0.000048Function stats for (Thread) (1)name ncall tsub ttot tavg..hon3.7/threading.py:859 Thread.run 1 0.000010 0.000043 0.000043doc3.py:8 _work 1 0.000006 0.000033 0.000033'''
You can usefilter_callback onget_func_stats API to filter on functions, modulesor whatever available inYFuncStat object.
importpackage_aimportyappiimportsysdefa():passdefb():passyappi.start()a()b()package_a.a()yappi.stop()# filter by module objectcurrent_module=sys.modules[__name__]stats=yappi.get_func_stats(filter_callback=lambdax:yappi.module_matches(x, [current_module]))# x is a yappi.YFuncStat objectstats.sort("name","desc").print_all()'''Clock type: CPUOrdered by: name, descname ncall tsub ttot tavgdoc2.py:10 b 1 0.000001 0.000001 0.000001doc2.py:6 a 1 0.000001 0.000001 0.000001'''# filter by function objectstats=yappi.get_func_stats(filter_callback=lambdax:yappi.func_matches(x, [a,b])).print_all()'''name ncall tsub ttot tavgdoc2.py:6 a 1 0.000001 0.000001 0.000001doc2.py:10 b 1 0.000001 0.000001 0.000001'''# filter by module namestats=yappi.get_func_stats(filter_callback=lambdax:'package_a'inx.module ).print_all()'''name ncall tsub ttot tavgpackage_a/__init__.py:1 a 1 0.000001 0.000001 0.000001'''# filter by function namestats=yappi.get_func_stats(filter_callback=lambdax:'a'inx.name ).print_all()'''name ncall tsub ttot tavgdoc2.py:6 a 1 0.000001 0.000001 0.000001package_a/__init__.py:1 a 1 0.000001 0.000001 0.000001'''
You can see that coroutine wall-time's are correctly profiled.
importasyncioimportyappiasyncdeffoo():awaitasyncio.sleep(1.0)awaitbaz()awaitasyncio.sleep(0.5)asyncdefbar():awaitasyncio.sleep(2.0)asyncdefbaz():awaitasyncio.sleep(1.0)yappi.set_clock_type("WALL")withyappi.run():asyncio.run(foo())asyncio.run(bar())yappi.get_func_stats().print_all()'''Clock type: WALLOrdered by: totaltime, descname ncall tsub ttot tavgdoc4.py:5 foo 1 0.000030 2.503808 2.503808doc4.py:11 bar 1 0.000012 2.002492 2.002492doc4.py:15 baz 1 0.000013 1.001397 1.001397'''
You can use yappi to profile greenlet applications now!
importyappifromgreenletimportgreenletimporttimeclassGreenletA(greenlet):defrun(self):time.sleep(1)yappi.set_context_backend("greenlet")yappi.set_clock_type("wall")yappi.start(builtins=True)a=GreenletA()a.switch()yappi.stop()yappi.get_func_stats().print_all()'''name ncall tsub ttot tavgtests/test_random.py:6 GreenletA.run 1 0.000007 1.000494 1.000494time.sleep 1 1.000487 1.000487 1.000487'''
Coroutine Profiling(new in 1.2)
Greenlet Profiling(new in 1.3)
Note: Yes. I know I should be moving docs to readthedocs.io. Stay tuned!
Special thanks to A.Jesse Jiryu Davis:
Yappi is the default profiler inPyCharm. If you have Yappi installed,PyCharm will use it. Seethe official documentation for more details.
About
Yet Another Python Profiler, but this time multithreading, asyncio and gevent aware.