- milli -> nano conversion in the querier- add deletes to store requests from the ingester

* Add some commentsSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Use deletes in tailSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Change error messageSigned-off-by: Michel Hollands <michel.hollands@grafana.com>

* fix(gateway): make DNS resolver configurableSigned-off-by: Ivan Rizzante <i.rizzante@gmail.com>* fix: lint

This fixes a bug in the index gateway when querying values for a label.Since the gRPC handler for LabelValuesForMetricName in the index gatewayallows empty matchers in the LabelValuesForMetricNameRequest, we need tocheck if the matchers string is an empty matcher (`{}`) before we parsethe string.This bug was introduced with the index gateway api refactoring in#5892Fixes:#5965Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

when index gateway does not implement the new API.This bug did not yield an error but only returned incorrect resultsbecause both functions `c.IndexStore.LabelNamesForMetricName` and`c.IndexStore.LabelValuesForMetricName` return `([]string, error)`.Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

Signed-off-by: irizzant <i.rizzante@gmail.com>

* Restrict gRPC calls to ingesters when requesting series if there is no overlap within the query_ingesters_within value* Add tests for isWithinIngesterMaxLookbackPeriod* Adding tests for calculateIngesterMaxLookbackPeriod* Only send query for range overlapping with ingesters* Restrict gRPC calls to ingesters when requesting series if there is no overlap within the query_ingesters_within value* Add tests for isWithinIngesterMaxLookbackPeriod* Adding tests for calculateIngesterMaxLookbackPeriod* Only send query for range overlapping with ingesters* Add test for when to query ingester and storage* Fix deadlock in Series when just queriying the ingesters* Refactored slightly for better readability* Added CHANGELOG entry* Fix typo* Rephrase test cases names* Move block to outer loopCo-authored-by: Salva Corts <salva.corts@grafana.com>

* Generate cache invalidation numbers in the delete store* Get cache generation numbers from the store on request* changlog* rename tombstones to something more meaningful* User invisible module* query frontend relies on a compactor to get the cache generation number* fix serialization* source -> name* lint errors* lint errors* log non-200 responses* add jsonnet changes* lint* review feedback* review feedback* client rename

This is part 1 of 2 PRs and updates the build image to version0.10.4 and adding the buf utility which will later be used forformatting *.proto files

…ed messages (#5992)

* Disable calling new index-gateway client's API.* Remove infinite recursive call* Correct mock setup for test

* Remove empty match within concat of regex* add back tests

* Revendor grafana/tail.- This revendoring brings the fix to relative symlinks, as described in#3374.* Prune package from `go mod tidy`.* noop

…ranteed query sort order when receiving entries for the same stream that have duplicate timestamps. (#6042)* Add a per tenant ability to fudge duplicate timestamps in the distributorSigned-off-by: Edward Welch <edward.welch@grafana.com>* update docsSigned-off-by: Edward Welch <edward.welch@grafana.com>* lintSigned-off-by: Edward Welch <edward.welch@grafana.com>* update changelogSigned-off-by: Edward Welch <edward.welch@grafana.com>

* Add two metrics to the IndexGateway.- Add a new `query_readiness_duration_seconds` metric, that reports  query readiness duration of a tablemanager/index gateway instance. We  should use it later to report performance against the ring mode- Add a new `usersToBeQueryReadyForTotal` metric, that reports number of  users involved in the query readiness operation. We should use it  later to correlate number of users with the query readiness duration.* Remove `usersToBeQueryReadyForTotal`.- It will report all users always for now, so it isn't too helpful the  way it is.* Rename metric help text to not mislead people.* Log queryReadiness duration.* Fix where log message and duration and triggered.* Join users list in a single string.- This is necessary since go-kit doesn't support array type.* Tweak queryReadiness log messages.- As suggested by Ed on#5972 (comment) and#5972 (comment)* Ensure queryReadinessDuration metric.- It is redundant with a recently added log line.* noop

* Revise the getting started section* Docs: revise getting started guide* Intermediate commit of getting started material.* Intermediate revision of the getting started guide.    Add instructions to download, configure, and deploy the test environment.* Add sample queries and instructions for breaking down the test env* Revise prose using reviewer comments.* update to use promtail and flog in the docker compose fileSigned-off-by: Edward Welch <edward.welch@grafana.com>* Revise guide to put all parts into Docker containers.* Add README files for examples directory. Revise getting started instructions.* Address DylanGuedes review comments* Getting started guide: incorporate suggestions from ssncferreiraCo-authored-by: Edward Welch <edward.welch@grafana.com>

This component has been around for a while but has not been added to theloki-mixin yet.Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

…tor (#5879)* Add filter parameter to reboundSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix linting issuesSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Add filter function to delete requestSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix linting issuesSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Enable api for filter and delete modeSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Add settings for retentionSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Use labels to check and add testSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Simplify filter functionSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Also enable filter modeSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Remove test settings in config file for dockerSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Add extra (unused) param for ProcessString in filterSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Empty commit to trigger CI againSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Update changelogSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix flapping testSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Remove commented out unit tests and add some moreSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Add extra test case for delete request without line filterSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Use chunk boundsSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* check if the log selector has a filter if the whole chunk is selectedSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* fix lint issue: use correct go-kit importSigned-off-by: Michel Hollands <michel.hollands@grafana.com>

)* Docs: Add changelog entry for configuration parameter name change* Update changelog message, per review request

…by the AST (#6346)* Panic with an actual error if clone (which calls range aggregationexpr's validate) errors, instead of swallowing the error and thenattempting to execute a nil function, which panics anyways.Signed-off-by: Callum Styan <callumstyan@gmail.com>* Call the validation function on a modified expression before attemptingto execute it. If the validation fails we should return the originalexpression.Signed-off-by: Callum Styan <callumstyan@gmail.com>

Signed-off-by: Christopher Norris <christopher.norris@grafana.com>

So that you can do quantile calculation on metadata queries.Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

This updates to a version that contains the fix toCVE-2022-28948.

#6310)* Add support for client-side TLS certs to loki-canary* fix lint errors, make the case of using TLS but not client certs more clear, check that -tls is set if client certs are used* move changelog to canary section instead of main* wrap httpClient errors and simplify dialer creation for websocket

* index sampling endpoint* fix bug in withTenantLabelMatcher* show zero values in stats* handles new errcase* fixes comments

Signed-off-by: Trevor Wood <Trevor.G.Wood@gmail.com>

…#6288)

* cmd/loki: minor consistency patch* cmd/loki: fixed a typo

* begins building queryrange compatible types for index sampling* adds MatcherGroups() method to SampleExpr* MatcherGroups test + ShardResolver interface* index stats codec wiring* millisecond precision on index stats codec* lint* cleanMatchers in tsdb.IndexClient for consistency + adds catchall matcher if nil

* Add metrics for journal targetThere are no metrics for the journal target yet, this adds two basicones:- Total number of lines processed- Total number of errors processing lines (with a reason)Because of the way the journal works, tailing one file with manypossible process writing to it, I chose note to expose counts per unit.* Add changelog entry* Fix compilation of (unsupported) journalmanager target on windows* Extract possible errors to constants & add tests

* Backport memberlist dependency version pin to k102Signed-off-by: JordanRushing <rushing.jordan@gmail.com>* Revert "Loki Prometheus mixin: templatize cluster label" (#6377)* Revert "Loki Prometheus mixin: templatize cluster label (#6353)"This reverts commit459036f.* Replace `loki-mixin` dependency to working commit.Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>(cherry picked from commit666bac2)Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>(cherry picked from commit372961d)Co-authored-by: JordanRushing <rushing.jordan@gmail.com>

) (#6493)(cherry picked from commitefda5b2)Co-authored-by: Travis Patterson <travis.patterson@grafana.com>(cherry picked from commitf79ff95)

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>(cherry picked from commit671afe6)Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>(cherry picked from commita7f8be2)

* Add more logging for delete requestsSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix typoSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Rephrase logging message and add chunk IDSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Rename userID field to userSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* rename more userIDSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix existing log messagesSigned-off-by: Michel Hollands <michel.hollands@grafana.com>(cherry picked from commit20db4df)Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>(cherry picked from commitc707faf)

…) (#6496)* Compactor: add per tenant compaction delete enabled flag (#6410)* Add per tenant compaction delete enabled flagSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Remove changes in wrong placeSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Add compactor deletion enabled fieldSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Use limit in compactorSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Use http middleware and add testSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix lint issueSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Add changelogSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Revert to default setting if no overrideSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Add default value command line optionSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Update the docsSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Enable access to deletion API for integration testSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Rename flag to allow_deletesSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Update per review commentsSigned-off-by: Michel Hollands <michel.hollands@grafana.com>(cherry picked from commitb4e6c59)* Fix changelogSigned-off-by: Michel Hollands <michel.hollands@grafana.com>(cherry picked from commitdf48f93)Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>

)* Remove whole stream deletion mode (#6435)* Remove whole-stream-deletion modeSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Remove whole-stream-deletion from docsSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Update the changelogSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Sort changelog entriesSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Remove link to wrong configurationSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix integration testSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Set default deletion mode to disabledSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Remove extra white line in documentationSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix default value in docsSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix changelogSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Add DeletionEnabled method on modeSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Rename testSigned-off-by: Michel Hollands <michel.hollands@grafana.com>(cherry picked from commitf80e487)* Fix changelogSigned-off-by: Michel Hollands <michel.hollands@grafana.com>* Fix changelog linkSigned-off-by: Michel Hollands <michel.hollands@grafana.com>(cherry picked from commitda3fbd9)Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>

This reverts commit36e0979.

* disable tests which are failing from hardcoded certs which are now expired.Signed-off-by: Edward Welch <edward.welch@grafana.com>* lintSigned-off-by: Edward Welch <edward.welch@grafana.com>

Signed-off-by: Danny Kopping <danny.kopping@grafana.com>