Repository files navigation

FlowGrid is a Python library designed to improve parallelization across multiple machines with a powerful and user-friendly interface, allowing you to focus on your application logic without worrying about the complexities of task distribution and management.

The primary goal ofFlowGrid is to provide an easy-to-use interface for parallelizing Python tasks across multiple workers.

In Python, a common design pattern is to launch asynchronous tasks while instantly responding to the user—FastAPI, for instance, addresses this withbackground_tasks, though it acknowledges this method may not be the most reliable, especially for long-running tasks.

FlowGrid abstracts the complexity of task management, progress tracking, and task cancellation, making it easier to build scalable and robust distributed systems. This library is also designed to seamlessly integrate with popular frameworks likeFastAPI,Flask, andDjango, solving common concurrency challenges effortlessly.

While other solutions likeCelery (which FlowGrid uses under the hood) are available, they often require more configuration and are more challenging to use. FlowGrid simplifies the process while maintaining the full power of Celery, with additional enhancements such as native task state checking and improved handling of task cancellations, even for tasks in progress.

Note: FlowGrid will soon be available on PyPI. You will be able to install it using pip:

pip install py-flowgrid

FlowGrid requires Python 3.7 or higher.

FlowGrid simplifies task management in Celery by allowing you to define tasks using decorators and manage them seamlessly. Here's an example:

importtimefromflowgridimportFlowGridfg=FlowGrid()@fg.taskdefadd(x:float,y:float)->float:# Simulate a long running tasktime.sleep(10)returnx+ydefmain():task=add(1,2)# Task id is none because it is not launchedprint('TASK:',task.task_id)# You can explicitly launch the task or let the# wait function do it for you# task = fg.launch(task)  # Can be uncommented# At this point the task id is available# print('TASK:', task.task_id)  # Can be uncommentedresponse=fg.wait(task)print('RESPONSE:',response)if__name__=='__main__':main()

FlowGrid supports real-time progress tracking for your tasks. You can update and monitor the progress easily:

importtimefromflowgridimportFlowGridfg=FlowGrid()@fg.taskdefadd_multiple(x:float,y:float,times:int=10)->float:response=xforiinrange(times):time.sleep(1)fg.update(progress=i,total=times,percent=100*i/times)response+=yreturnresponsedefmain():task=add_multiple(10,5,times=5)response=fg.wait(task)# Expected: 10 + 5*5 = 35print('RESPONSE:',response)if__name__=='__main__':main()

FlowGrid allows you to cancel tasks, either forcefully or gracefully:

importtimefromflowgridimportFlowGridfg=FlowGrid()@fg.taskdefadd_multiple(x:float,y:float,times:int=10)->float:response=xforiinrange(times):time.sleep(1)fg.update(progress=i,total=times,percent=100*i/times)response+=yreturnresponsedefmain():task=fg.launch(add_multiple(10,5,times=5))time.sleep(3)# Simulating a user cancelling the taskprint('CANCELLING TASK')# force=True will terminate the task immediatelyfg.revoke(task,force=True)if__name__=='__main__':main()

importtimefromflowgridimportFlowGridfg=FlowGrid()@fg.taskdefadd_multiple(x:float,y:float,times:int=10)->float:response=xforiinrange(times):# Check for revocation and stop if needediffg.is_revoked():print('CANCELLED')returntime.sleep(1)fg.update(progress=i,total=times,percent=100*i/times)response+=yreturnresponsedefmain():task=fg.launch(add_multiple(10,5,times=5))print('TASK:',task.task_id)time.sleep(3)# Simulating a user cancelling the taskprint('CANCELLING TASK')# Graceful cancellationfg.revoke(task)if__name__=='__main__':main()

FlowGrid workers can be launched with the command:

flowgrid worker --app<path-to-your-flowgrid-class>

You can also use the-A shorthand for--app:

flowgrid worker -A<path-to-your-flowgrid-class>

For example in order to prepare workers to launch the initial example from this repository, you can run:

flowgrid worker -A examples.01-base.fg

The.fg must be included because it is the variable name of my FlowGrid instance in the fileexamples/01-base.py.

To view all available options, use:

flowgrid worker -h

Some useful options include:

• Concurrency: Control the number of worker processes with--concurrency or-c.
• Log Level: Set the logging level with--loglevel or-l.

Here's a basic example that demonstrates how to define and launch tasks using FlowGrid:

importtimefromflowgridimportFlowGridfg=FlowGrid()@fg.taskdefadd(x:float,y:float)->float:time.sleep(10)returnx+ydefmain():task=add(1,2)print('TASK:',task.task_id)response=fg.wait(task)print('RESPONSE:',response)if__name__=='__main__':main()

• PyPI Release: Soon, FlowGrid will be available for installation via pip.
• Extended Documentation: More detailed documentation and examples will be added as the project evolves.
• Support chaining tasks without waiting: Currently, you have to wait for a task to finish before chaining another task. You can check the exampleexamples/06-chaining.py to see how to chain tasks. With the next solution it will look in the exact same way but if you usefg.launch the response will be instant and you can return the task to the user from the beginning.
• Support task sequence without relationship: Right now you can only chain tasks just by including them as paramteres, but what if the response of a task is not needed for the next one. We will create an interface called:fg.sequence that will allow you to define a sequence of tasks that will be executed in order.

FlowGrid is licensed under the MIT License. SeeLICENSE for more information.

Contributions are welcome! Please fork the repository and submit a pull request to contribute to FlowGrid.

For any questions or inquiries, please contact the maintainers via [your email/contact information].