Repository files navigation

A fast text scanner/regex extractor and realtime summarizer. Quickly search, reformat and visualize text filessuch as logs, csv, json, etc.

Supports various CLI-based graphing and metric formats (filter (grep-like), histogram, table, bargraph, heatmap, reduce).

rare is a play on "more" and "less", but can also stand for "realtime aggregated regular expressions".

Seerare.zdyn.net or thedocs/ folder for the full documentation.

• Multiple summary formats including: filter (like grep), histogram, bar graphs, tables, heatmaps, reduce, and numerical analysis
• Parse using regex (-m) or dissect tokenizer (-d)
• File glob expansions (eg/var/log/* or/var/log/*/*.log) and-R
• Optional gzip decompression (with-z)
• Following-f or re-open following-F (use--poll to poll, and--tail to tail)
• Ignoring lines that match an expression (with-i)
• Aggregating and realtime summary (Don't have to wait for all data to be scanned)
• Multi-threaded reading, parsing, and aggregation (It's fast)
• Color-coded outputs (optionally)
• Pipe support (stdin for reading, stdout will disable realtime, and--csv formatting) eg.tail -f | rare ... > out

Take a look atexamples to see more of whatrare does.

Output formats include:

• filter is grep-like, in that each line will be processed and the extracted key will be output directly to stdout
• histogram will count instances of the extracted key
• table will count the key in 2 dimensions
• heatmap will display a color-coded version of the strength of a cell in a dense format
• sparkline draws rows of sparklines for the data
• bargraph will create either a stacked or non-stacked bargraph based on 2 dimensions
• analyze will use the key as a numeric value and compute mean/median/mode/stddev/percentiles
• reduce allows evaluating data using expressions, and grouping/sorting the output

More details on various output formats and aggregators (including examples) can be found inaggregators

This script downloads the latest version from github, and installs it to~/.local/bin (/usr/bin ifroot).

curl -sfL https://rare.zdyn.net/install.sh| bash

Download appropriate binary or package fromReleases

brew tap zix99/rarebrew install rare

The below install methods have been contributed by the community, and aren't maintained directly.

sudo port selfupdatesudo port install rare

Clone the repo, and:

Requires GO 1.23 or higher

go mod download# Build binarygo build.# OR, with experimental featuresgo build -tags experimental.

Available tags:

• experimental Enable experimental features (eg. fuzzy search)
• pcre2 Enables PCRE 2 (v10) where able. Currently linux only
• rare_no_pprof Disables profiling capabilities, which reduces binary size
• urfave_cli_no_docs Disables man and markdown documentation generation, which reduces binary size

A Note on PCRE (Perl Compatible Regex Library)

Besides your standard OS versions, there is an additionalpcre build which is ~4x faster than go'sre2 implementation in moderately complex cases. In order to use this, you must make sure thatlibpcre2 is installed (egapt install libpcre2-8-0). Right now, it is only bundled with the linux distribution.

PCRE2 also comes with pitfalls, two of the most important are:

1. Thatrare is now dynamically linked, meaning that you need to have libc and libpcre installed
2. That pcre is an exponential-time algorithm (re2 is linear). While it can be significantly faster than go'sre2, it can also be catastrophically slower in some situations. There is a good posthere that talks about regexp timings.

I will leave it up to the user as to which they find suitable to use for their situation. Generally, if you know whatrare is getting as an input, the pcre version is perfectly safe and can be much faster.

All documentation may be found here, in thedocs/ folder, by runningrare docs (embedded docs/ folder), or on the websiterare.zdyn.net

You can also see a dump of the CLI options atcli-help.md

$ cat input.txt121310$ rare histo input.txt1                   3         0                   1         2                   1         3                   1         Matched: 6 / 6 (Groups: 4)

$ rare filter -n 4 -m"(\d{3}) (\d+)" -e"{1} {2}" access.log404 169404 169404 571404 571Matched: 4 / 4

$ rare histo \    -m'"(\w{3,4}) ([A-Za-z0-9/.]+).*" (\d{3})'\# The regex that extracts match-groups    -e'{3} {1}'\# The expression will be the key, referencing the match-groups    access.log# One or more files (or -R for recursion)200 GET                          160663404 GET                          857304 GET                          53200 HEAD                         18403 GET                          14

For more examples, check out thedocs orthe website

I know there are different solutions, and rare accomplishes summarization in a waythat grep, awk, etc can't, however I think it's worth analyzing the performance of thistool vs standard tools to show that it's at least as good.

Seebenchmarks for comparisons between common tools likegrep | wc,silversearcher-ag, etc.

New additions torare should pass the following checks

• Documentation for any new functionality or expression changes
• Before and after CPU and memory benchmarking for core additions (Expressions, aggregation, benchmarking, and rendering)
• Limit memory allocations (preferably 0!) in the high-throughput functions
• Tests, and if it makes sense, benchmarks of a given function

go run.gotest ./...

New high-throughput changes should be performance benchmarked.

To Benchmark:

go run. --profile out<yourtest code>go tool pprof -http=:8080 out.cpu.prof# CPUgo tool pprof -http=:8080 out_num.prof# Memory

New functionality should be well-documented indocs/ folder. The docs areboth published to the website viamkdocs and embedded in the application.

Testing and publishing docs:

# Install mkdocs versionpip3 install -r requirements.txt# Test locallymkdocs serve# Test cli docsgo run. docs# If CLI arguments change, publish updated docsgo run. _gendoc> docs/cli-help.md

Copyright (C) 2019  Christopher LaPointeThis program is free software: you can redistribute it and/or modifyit under the terms of the GNU General Public License as published bythe Free Software Foundation, either version 3 of the License, or(at your option) any later version.This program is distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See theGNU General Public License for more details.You should have received a copy of the GNU General Public Licensealong with this program.  If not, see <https://www.gnu.org/licenses/>.