Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

fix CelerySignalProcessor delete handling#473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
Bragegs wants to merge2 commits intodjango-es:master
base:master
Choose a base branch
Loading
fromBragegs:patch-1

Conversation

@Bragegs
Copy link

@BragegsBragegs commentedFeb 2, 2024•
edited
Loading

  1. Removed unecessary comma from setter ofbulk_data variable.
  2. Renameregistry_delete_task function parameter namedata intobulk_data
  3. Adddoc_module toregistry_delete_task function parameters so that we can init the registered document correctly.
  4. On deletion (1): a list of related instances are now prepared and then we only send the ids to those to the task that should update the related instances. The reason for this change is that if we serialize these instances in the preparation period the instance we were deleting have not been deleted yet (pre-save) and thus if the related instances had data connected to this will-be-deleted instance it would stick instead of being deleted. By serializing the related instances after the instance actually were deleted, the related instances docs would be correctly serialized and updated in registry_delete_related_task.
  5. On deletion(2): Now it prepares the instance being deleted and deletes it in registry_delete_task. It does not handle related instances here as it did before, which is covered by (1).

Fixes472

duffn, lucasmoeskops, HengJunXi, and lombord reacted with thumbs up emoji
1. Removed unecessary comma from setter of `bulk_data` variable.2. Add `doc_module` to `registry_delete_task` function parameters so that we can init the registered document correctly.
@Teachmetech
Copy link

Teachmetech commentedFeb 7, 2024•
edited
Loading

@Bragegs After trying your changes I get the following error in DigitalOcean, is there a way to avoid this?:

  File "/workspace/.heroku/python/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task    R = retval = fun(*args, **kwargs)                 ^^^^^^^^^^^^^^^^^^^^  File "/workspace/.heroku/python/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__    return self.run(*args, **kwargs)           ^^^^^^^^^^^^^^^^^^^^^^^^^  File "/workspace/api/signals/django_elasticsearch_dsl_custom_celery_signal_handler.py", line 83, in registry_delete_task    doc_instance._bulk(bulk_data, parallel=parallel)  File "/workspace/.heroku/python/lib/python3.11/site-packages/django_elasticsearch_dsl/documents.py", line 234, in _bulk    return self.parallel_bulk(*args, **kwargs)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/workspace/.heroku/python/lib/python3.11/site-packages/django_elasticsearch_dsl/documents.py", line 194, in parallel_bulk    deque(bulk_actions, maxlen=0)  File "/workspace/.heroku/python/lib/python3.11/site-packages/elasticsearch/helpers/actions.py", line 592, in parallel_bulk    pool = BlockingPool(thread_count)           ^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/workspace/.heroku/python/lib/python3.11/multiprocessing/pool.py", line 930, in __init__    Pool.__init__(self, processes, initializer, initargs)  File "/workspace/.heroku/python/lib/python3.11/multiprocessing/pool.py", line 196, in __init__    self._change_notifier = self._ctx.SimpleQueue()                            ^^^^^^^^^^^^^^^^^^^^^^^  File "/workspace/.heroku/python/lib/python3.11/multiprocessing/context.py", line 113, in SimpleQueue    return SimpleQueue(ctx=self.get_context())           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/workspace/.heroku/python/lib/python3.11/multiprocessing/queues.py", line 341, in __init__    self._rlock = ctx.Lock()                  ^^^^^^^^^^  File "/workspace/.heroku/python/lib/python3.11/multiprocessing/context.py", line 68, in Lock    return Lock(ctx=self.get_context())           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/workspace/.heroku/python/lib/python3.11/multiprocessing/synchronize.py", line 162, in __init__    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)  File "/workspace/.heroku/python/lib/python3.11/multiprocessing/synchronize.py", line 57, in __init__    sl = self._semlock = _multiprocessing.SemLock(                         ^^^^^^^^^^^^^^^^^^^^^^^^^OSError: [Errno 38] Function not implemented```

@Bragegs
Copy link
Author

@Teachmetech
Copy link

@Teachmetech could this be a DigitalOcean issue?

https://stackoverflow.com/a/67347498

https://www.digitalocean.com/community/tutorials/python-multiprocessing-example

Interesting, thanks for that info! Is it possible to workaround this issue by having the deletes go through
the RealTimeSignalProcessor and the rest of the signals go through the celery processor? I have a task that runs nightly that schedules about a few million objects to be updated and the signals slow it down quite a bit. Trying to minimize the time updates take by creating a separate queue/workers specifically to handle the ES signals. Everything is succeeding except for the delete signals. Another option would to be to run ES sync after the updates are completed, however, not sure how to do that without wiping the data, I'm not sure if any of the management commands support that.

Screenshot 2024-02-09 at 2 11 26 AM

This version seems to actually delete all indexes and related docs instances correctly
@HengJunXi
Copy link

@safwanrahman sorry for disturbing, may I know what's the status for this PR?

@parth-paradkar
Copy link

@safwanrahman this has a fix for a critical bug inCelerySignalProcessor. Can you please take a look?

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ModuleNotFoundError: No module named 'MyIndex'

4 participants

@Bragegs@Teachmetech@HengJunXi@parth-paradkar
[8]ページ先頭
©2009-2025 Movatter.jp