- Notifications
You must be signed in to change notification settings - Fork28
Zero-config dynamically-generated queryables, Performance fixes#351
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Hi@Zaczero. Looks really good, just a few linting errors. |
It would be nice to have@rhysrevans3 look at this as he is doing work on the queryables. |
@jonhealy1 This looks good to me and is probably a neater solution for @Zaczero am I right in saying this doesn't change the filter extension query? So doesn't effect thequeryables mapping that is used into_es |
jonhealy1 commentedApr 8, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@rhysrevans3 I sent you an invite to be a maintainer of this project so you can do an official pr review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
These look good!
@jonhealy1 I think my queryables pull request could still be useful. It takes a different approach, extracting the fields from the extensions where possible and does something similar to this pull request for the other properties. I'd be happy to attempt to merge the two methods in my pull request once this one is accepted if we want both?
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
This looks amazing. I'd like to follow this same approach to generate aggregations as well |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thanks@Zaczero. Looks great!
6b25e56 intostac-utils:mainUh oh!
There was an error while loading.Please reload this page.
**Description:****Changes from 3.2.5:**#### Added- Added support for dynamically-generated queryables based onElasticsearch/OpenSearch mappings, with extensible metadata augmentation[#351](#351)- Included default queryables configuration for seamless integration.[#351](#351)- Added support for high-performance direct response mode for bothElasticsearch and Opensearch backends, controlled by the`ENABLE_DIRECT_RESPONSE` environment variable. When enabled(`ENABLE_DIRECT_RESPONSE=true`), endpoints return Starlette Responseobjects directly, bypassing FastAPI's jsonable_encoder and Pydanticserialization for significantly improved performance on large searchresponses. **Note:** In this mode, all FastAPI dependencies (includingauthentication, custom status codes, and validation) are disabled forall routes. Default is `false` for safety. A warning is logged atstartup if enabled. See [issue#347](#347)and [PR#359](#359).- Added robust tests for the `ENABLE_DIRECT_RESPONSE` environmentvariable, covering both Elasticsearch and OpenSearch backends. Testsgracefully handle missing backends by attempting to import both configsand skipping if neither is available.[#359](#359)#### Changed- Refactored database logic to reduce duplication[#351](#351)- Replaced `fastapi-slim` with `fastapi` dependency[#351](#351)- Changed minimum Python version to 3.9[#354](#354)- Updated stac-fastapi api, types, and extensions libraries to 5.1.1from 3.0.0 and made various associated changes[#354](#354)- Changed makefile commands from 'docker-compose' to 'docker compose'[#354](#354)- Updated package names in setup.py files to use underscores instead ofperiods for PEP 625 compliance[#358](#358) - Changed `stac_fastapi.opensearch` to `stac_fastapi_opensearch` - Changed `stac_fastapi.elasticsearch` to `stac_fastapi_elasticsearch` - Changed `stac_fastapi.core` to `stac_fastapi_core` - Updated all related dependencies to use the new naming convention- Renamed `docker-compose.yml` to `compose.yml` to align with DockerCompose V2 conventions[#358](#358)- Removed deprecated `version` field from all compose files[#358](#358)- Updated `STAC_FASTAPI_VERSION` environment variables to 4.0.0 in allcompose files[#362](#362)- Bumped version from 4.0.0a2 to 4.0.0 for the PEP 625 compliant release[#362](#362)- Updated dependency requirements to use compatible release specifiers(~=) for more controlled updates while allowing for bug fixes andsecurity patches[#358](#358)- Removed elasticsearch-dsl dependency as it's now part of theelasticsearch package since version 8.18.0[#358](#358)- Updated test suite to use `httpx.ASGITransport(app=...)` for FastAPIapp testing (removes deprecation warning).[#359](#359)- Updated stac-fastapi parent libraries to 5.2.0.[#359](#359)- Migrated Elasticsearch index template creation from legacy`put_template` to composable `put_index_template` API in`database_logic.py`. This resolves deprecation warnings and ensurescompatibility with Elasticsearch 7.x and 8.x.[#359](#359)- Updated all Pydantic models to use `ConfigDict` instead of class-based`Config` for Pydantic v2 compatibility. This resolves deprecationwarnings and prepares for Pydantic v3.[#359](#359)- Migrated all Pydantic `@root_validator` validators to`@model_validator` for Pydantic v2 compatibility.[#359](#359)- Migrated startup event handling from deprecated`@app.on_event("startup")` to FastAPI's recommended lifespan contextmanager. This removes deprecation warnings and ensures compatibilitywith future FastAPI versions.[#361](#361)- Refactored all boolean environment variable parsing in bothElasticsearch and OpenSearch backends to use the shared `get_bool_env`utility. This ensures robust and consistent handling of environmentvariables such as `ES_USE_SSL`, `ES_HTTP_COMPRESS`, and`ES_VERIFY_CERTS` across both backends.[#359](#359)#### Fixed- Improved performance of `mk_actions` and `filter-links` methods[#351](#351)- Fixed inheritance relating to BaseDatabaseSettings and ApiBaseSettings[#355](#355)- Fixed delete_item and delete_collection methods return types[#355](#355)- Fixed inheritance relating to DatabaseLogic and BaseDatabaseLogic, andApiBaseSettings[#355](#355)**PR Checklist:**- [x] Code is formatted and linted (run `pre-commit run --all-files`)- [x] Tests pass (run `make test`)- [x] Documentation has been updated to reflect changes, if applicable- [x] Changes are added to the changelog
Uh oh!
There was an error while loading.Please reload this page.
Related Issue(s):
Description:
This PR consists of self-contained commits (except the first commit that provides database_logic deduplication), making it easy to change or remove individual patches. It addresses several small issues, improves the performance of certain methods, and adds support for dynamically-generated queryables. This enhancement doesn't require any new configuration as queryables are generated on the fly based on the os/es mappings. The implementation is designed for extensibility, with built-in logic for augmenting fields metadata with additional information. Currently, it only includes the _DEFAULT_QUERYABLES configuration, which was simply copied from the pre-PR code.
Example queryables response:
{"$schema":"https://json-schema.org/draft/2019-09/schema","$id":"https://stac-api.example.com/queryables","type":"object","title":"Queryables for STAC API","description":"Queryable names for the STAC API Item Search filter.","properties":{"bbox":{"title":"Bbox","type":"number"},"collection":{"description":"Collection","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/then/properties/collection","title":"Collection","type":"string"},"geometry":{"description":"Geometry","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/1/oneOf/0/properties/geometry","title":"Geometry","type":"object"},"id":{"description":"ID","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id","title":"Id","type":"string"},"stac_extensions":{"title":"Stac Extensions","type":"string"},"stac_version":{"title":"Stac Version","type":"string"},"type":{"title":"Type","type":"string"},"constellation":{"title":"Constellation","type":"string"},"created":{"description":"Creation Timestamp","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/datetime.json#/properties/created","title":"Created","type":"string","format":"date-time"},"datetime":{"description":"Acquisition Timestamp","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/datetime.json#/properties/datetime","title":"Datetime","type":"string","format":"date-time"},"end_datetime":{"title":"End Datetime","type":"string","format":"date-time"},"eopf:datatake_id":{"title":"Eopf:Datatake Id","type":"string"},"eopf:instrument_configuration_id":{"title":"Eopf:Instrument Configuration Id","type":"number"},"instruments":{"title":"Instruments","type":"string"},"platform":{"title":"Platform","type":"string"},"processing:datetime":{"title":"Processing:Datetime","type":"string","format":"date-time"},"processing:facility":{"title":"Processing:Facility","type":"string"},"processing:level":{"title":"Processing:Level","type":"string"},"processing:version":{"title":"Processing:Version","type":"string"},"product:timeliness":{"title":"Product:Timeliness","type":"string"},"product:timeliness_category":{"title":"Product:Timeliness Category","type":"string"},"product:type":{"title":"Product:Type","type":"string"},"published":{"title":"Published","type":"string","format":"date-time"},"sar:center_frequency":{"title":"Sar:Center Frequency","type":"number"},"sar:frequency_band":{"title":"Sar:Frequency Band","type":"string"},"sar:instrument_mode":{"title":"Sar:Instrument Mode","type":"string"},"sar:observation_direction":{"title":"Sar:Observation Direction","type":"string"},"sar:pixel_spacing_azimuth":{"title":"Sar:Pixel Spacing Azimuth","type":"number"},"sar:pixel_spacing_range":{"title":"Sar:Pixel Spacing Range","type":"number"},"sar:polarizations":{"title":"Sar:Polarizations","type":"string"},"sar:resolution_azimuth":{"title":"Sar:Resolution Azimuth","type":"number"},"sar:resolution_range":{"title":"Sar:Resolution Range","type":"number"},"sat:absolute_orbit":{"title":"Sat:Absolute Orbit","type":"integer"},"sat:orbit_cycle":{"title":"Sat:Orbit Cycle","type":"number"},"sat:orbit_state":{"title":"Sat:Orbit State","type":"string"},"sat:platform_international_designator":{"title":"Sat:Platform International Designator","type":"string"},"sat:relative_orbit":{"title":"Sat:Relative Orbit","type":"integer"},"start_datetime":{"title":"Start Datetime","type":"string","format":"date-time"},"updated":{"description":"Creation Timestamp","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/datetime.json#/properties/updated","title":"Updated","type":"string","format":"date-time"},"view:azimuth":{"title":"View:Azimuth","type":"number"},"view:incidence_angle":{"title":"View:Incidence Angle","type":"number"},"auth:schemes.oidc.openIdConnectUrl":{"title":"Auth:Schemes.Oidc.Openidconnecturl","type":"string"},"auth:schemes.oidc.type":{"title":"Auth:Schemes.Oidc.Type","type":"string"},"auth:schemes.s3.type":{"title":"Auth:Schemes.S3.Type","type":"string"},"storage:schemes.cdse-s3.description":{"title":"Storage:Schemes.Cdse-S3.Description","type":"string"},"storage:schemes.cdse-s3.platform":{"title":"Storage:Schemes.Cdse-S3.Platform","type":"string"},"storage:schemes.cdse-s3.requester_pays":{"title":"Storage:Schemes.Cdse-S3.Requester Pays","type":"boolean"},"storage:schemes.cdse-s3.title":{"title":"Storage:Schemes.Cdse-S3.Title","type":"string"},"storage:schemes.cdse-s3.type":{"title":"Storage:Schemes.Cdse-S3.Type","type":"string"},"storage:schemes.creodias-s3.description":{"title":"Storage:Schemes.Creodias-S3.Description","type":"string"},"storage:schemes.creodias-s3.platform":{"title":"Storage:Schemes.Creodias-S3.Platform","type":"string"},"storage:schemes.creodias-s3.requester_pays":{"title":"Storage:Schemes.Creodias-S3.Requester Pays","type":"boolean"},"storage:schemes.creodias-s3.title":{"title":"Storage:Schemes.Creodias-S3.Title","type":"string"},"storage:schemes.creodias-s3.type":{"title":"Storage:Schemes.Creodias-S3.Type","type":"string"}},"additionalProperties":false}PS. I think the auto-generated "title" should be removed completely, but I included it because I found it to be common practice in some STAC projects. I'm not sure how you feel about it.
PR Checklist:
pre-commit run --all-files)make test)