Overview

Hi, this is a draft pull request for improving thetfq.convert_to_tensor function by adding parallelization (#336). I'm confused about how the benchmarking suite works, so I am creating a draft pull request. For parallelization, I made sure to follow thePython documentation forconcurrent.futures and usedProcessPoolExecutor.

Since theconvert_item function is mostly CPU-bound and doesn't require heavy communication between processes, usingProcessPoolExecutor should provide a significant speed-up for large inputs.

I have simply added parallelization to thetfq.convert_to_tensor function and need help with benchmarking and testing.

Changes

Parallelization: usedconcurrent.futures.ProcessPoolExecutor to parallelize the conversion of items. Each item in the list is processed independently, which allows for significant speedup for converting large lists of items.

Docstrings: the original function had informative docstrings that described thetfq.convert_to_tensor function. I made sure to keep most of it and only added the necessary changes and fixed typos.

Concerns

I wasn't able to benchmark my changes or run unit tests. I tried my best to follow theContribution guidelines but ran into a lot of errors with Bazel. I am still working on the issue and wanted to inform you that the code has not been benchmarked using theBenchmark guidelines.

If the function needs to handle extremely deep nested lists, theconvert_item() function may need to be iterative rather than recursive. However, the original author already made a comment to dismiss this concern.