- Notifications
You must be signed in to change notification settings - Fork28
Added support for searching large amount of indices#412
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation
…ices from the request url to the body of the request when size is larger than 4096 bytes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Nice work, I think this will be a useful fix for large catalogs, especially now that people are looking for ways to limit searches to certain collections based on authorizations (#409).
stac_fastapi/opensearch/stac_fastapi/opensearch/database_logic.py OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
…ort for large amount of queries to ElasticSearch database logic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Nice work!
90fd9ca
intostac-utils:mainUh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Description:
When searching the catalog with the
/search
-endpoint, aGET /<indices>/_search
request is done with all indices listed in the URL path. However when doing such a search on a large amount of indices, it is possible that the size of the endpoint exceeds Elasticsearch’s maximum allowed HTTP line length (4096 bytes), resulting in the following error:{"code":"RequestError","description":"RequestError(400, 'too_long_http_line_exception', 'An HTTP line is larger than 4096 bytes.')"}
The solution in this commit moves the indices from the endpoint to the body of the request once the amount of indices passes a certain threshold. The indices of the endpoint will be replaced by
ITEM_INDICES
. Since the query still filters on the correct indices, this change preserves the behavior while avoiding the URL length limitation.PR Checklist:
pre-commit run --all-files
)make test
)