Repository files navigation

Profilers help to identify performance problems. These are tools designed to give the metrics to find the slowest parts of the code so that we can optimize what really matters. Profilers can gather a wide variety of metrics: wall time, CPU time, network or memory consumption, I/O operations, etc.
Profilers can answer questions like,

• How many times is each method in my code called?
• How long does each of these methods take?
• How much memory does the method consume?

• Deterministic Profiling: Deterministic profilers execute trace functions at various points of interest (function call, function return) and record precise timings of these events. It means the code runs slower under profiling. Its use in production systems is often impractical.

• Statistical profiling:  Instead of tracking every event (call to every function), statistical profilers interrupt applications periodically and collect samples of the execution state (call stack snapshots). The call stacks are then analyzed to determine the execution time of different parts of the application. This method is less accurate, but it also reduces the overhead.

All the profilers we are going to discuss here areDeterministic Profilers because they capture precise timings of events. Please note that theMemory Profiler package also has themprof module that doesstatistical profiling. It is discussed briefly in Memory Profiler notebook.

This GitHub aims to show different profilers for Python and explain in detail the procedure to profile different workloads with different profilers. Below is the list of all the profilers we will be discussing. Each profiler has a separate folder with a Jupyter Notebook to guide you.

We will also use the following Intel AI Reference Kit in our profiling examples:

• Scikit-Learn Intelligent Indexing for Incoming Correspondence –Ref Kit

Follow the steps mentioned in theintelligent-Indexing Ref Kit GitHub ReadMe to setup the environments accordingly.
The process involves

• Setting up a virtual environment for both stock and Intel®-accelerated machine learning packages
• Preprocessing data using Pandas*/Intel® Distribution of Modin and NLTK
• Training an NLP model for text classification using Scikit-Learn*/Intel® Extension for Scikit-Learn*