Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork42
Perform update in parallel#110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
The update command implementation runs over files that are independentfrom each other. As such, the overall update operation can be triviallyparallelised to speed things up.This change introduces The list of files that need to becompared/updated is collected in a first past. This list is then givento a multiprocessing pool to farm out the actual update of eachindividual file. The amount of parallelism is controlled through a new"jobs" parameter, command line option and environment variable. If novalue is given for this option, all CPUs are used.I noticed this chance for improvement when doing a test run of theupdate of .po files for the Spanish translation of the CPythondocumentation. Local numbers in my 8-core, hyper-threaded AMD Ryzen 75825U:-j 1 (same as old behaviour)real 12m5.402suser 12m4.942ssys 0m0.273s-j 8real 2m23.609suser 17m45.201ssys 0m0.460s<no value given>real 1m57.398suser 26m22.654ssys 0m0.989sSigned-off-by: Rodrigo Tobar <rtobar@icrar.org>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thank you for your wonderful improvement suggestions.
Unfortunately, I wasn't able to check the effect of the sphinx-doc I used to check the speed, as there were not many documents. However, it seems that there is no bad effect on the operation.
I will merge it.
ee2537a intosphinx-doc:masterUh oh!
There was an error while loading.Please reload this page.
Thanks@shimizukawa for reviewing and merging. I wonder whether you'd be inclined to make a release to PyPI with these changes at some point? No rush though, we can always install the package from this git repository in the meanwhile. |
I'm going to drop py38 and release as soon as possible. |
2.3.0 has been shipped ;) |
@shimizukawa thank you very much, this is very helpful 😄 |
Éstos son pequeños cambios que mejoran ligeramente el proceso de laconstrucción de la documentación, y hacen más mantenible el código afuturo.Primero, la lista de rutas relativas que hay que arreglar en los .rst decpython ha sido simplificada, removiendo entradas innecesarias, yactualizando sólo los archivos que haga falta (en vez de ejecturas cadaactualización sobre todos los archivos cada vez).Segundo, el target `build` del Makefile fue separado en sus sub-partesconstituyentes, de tal modo que ahora en el step de CI donde antesteníamos una copia de los comandos `sed` ahora hay sólo una invocación a`make fix_relative_paths`.Finalmente, el PR que envié a `sphinx-intl` para realizar updates enparalelo [ya estáaceptado](sphinx-doc/sphinx-intl#110) y unanueva versión ya fue publicada, por lo que la lista de requisitos ahoraestá actualizada para usar esa última versión (y así hacer más rápido elproceso de actualización a 3.13).---------Signed-off-by: Rodrigo Tobar <rtobar@icrar.org>
The update command implementation runs over files that are independent from each other. As such, the overall update operation can be trivially parallelised to speed things up.
This change introduces The list of files that need to be compared/updated is collected in a first past. This list is then given to a multiprocessing pool to farm out the actual update of each individual file. The amount of parallelism is controlled through a new "jobs" parameter, command line option and environment variable. If no value is given for this option, all CPUs are used.
I noticed this chance for improvement when doing a test run of the update of .po files for the Spanish translation of the CPython documentation. Local numbers in my 8-core, hyper-threaded AMD Ryzen 7 5825U: