Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

ThreadPoolexecutor doesnt release memory #131448

Open
Labels
pendingThe issue will be closed if no feedback is providedtype-bugAn unexpected behavior, bug, or error
@ansultan1

Description

@ansultan1

Bug report

Bug description:

# Add a code block here, if required
    with ThreadPoolExecutor(max_workers=2) as thread_executor:        pending_thread_tasks.add(            thread_executor.submit(                get_bucket_objects, source_bucket_name, "source", source_api_key,                source_ibm_service_instance_id, source_location_constraint, source_cos_credentials            )        )        pending_thread_tasks.add(            thread_executor.submit(                get_bucket_objects, target_bucket_name, "target", target_api_key,                target_ibm_service_instance_id, target_location_constraint, target_cos_credentials            )        )        completed_process_tasks, _ = wait(pending_thread_tasks, return_when=ALL_COMPLETED)        for completed_process_task in completed_process_tasks:            bucket_type, objects = completed_process_task.result()            if bucket_type == "source":                LOGGER.info(f"Successfully fetched objects from source bucket '{source_bucket_name}' having "                            f"id {source_bucket_id}")                source_objects = objects            else:                LOGGER.info(f"Successfully fetched objects from target bucket '{target_bucket_name}' having "                            f"id {target_bucket_id}")                target_objects = objects

when celery task completes the woker should release memory. it dosnt release Full memory and the container contains 1 gb of memory when task completes because of using this threadpoolexecutor. when using processpool executor this issue doesnt come but as objects is a large list so when using multiprocessing the copy and pickling give memory spike thats why used multithreading. how to avoid the memory leak in this multithreading

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    pendingThe issue will be closed if no feedback is providedtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions


      [8]ページ先頭

      ©2009-2025 Movatter.jp