- Notifications
You must be signed in to change notification settings - Fork3
internet-innovation/netrics
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
The network measurements execution framework.
Netrics streamlines the development, debugging, distribution and periodic execution of tasks – specifically supporting the execution of computer network measurements.
This software is both a framework and a bundle: Netrics is distributed with built-in, research-backed measurements,and the installed software may be trivially configured and extended.
The project aims to support network researchers, engineers, providers and users.
Features:
Configuration for Humans: in either YAML or TOML formats
Measurements built in: Provide your own measurements or hit the ground running with built-in, pre-configured measurements from Ookla, Measurement Lab, iPerf, and more
Open, operating system-level interfaces: measurements are implemented as arbitrary executables reading structured standard input parameters, writing structured standard output results,etc.
Diverse installation targets: native support for*nix devices featuring Python version 3.8 or later, via pip or via all-in-one pre-built bundles – and, Docker images, too!
Debugging tools: command-line utilities to test and debug measurement executables – on your laptop or on your Raspberry Pi
Python SDK: simple tools to help in the implementation of measurements in Python (and serving as prototypes for their implementation in other languages)
💡 | Not interested in network measurements? Netrics is itself a distribution ofFate, a general-purpose, feature-rich scheduler for periodic tasks. |
Not quite ready to dive in? Check out ourguide in the netrics-demo repository.
Netrics publishes a number ofDebian-based Docker images, for the ease of users familiar with Docker, to support otherwise unsupported systems, and otherwise to provision Netrics in systems utilizing containers.
For example, to get started with the default Netrics Docker image:
docker run ghcr.io/internet-equity/netrics
The "one-line" installation script is the recommended method formost humans to get started with Netrics.
The script will download and unpack thepre-built bundle appropriate to your system, as well as perform the software’sinitialization.
The one-liner below will install Netrics for the current user:
curl -sL https://github.com/internet-equity/netrics/raw/0.0.1/install.py| python3To install Netrics system-wide you might usesudo:
curl -sL https://github.com/internet-equity/netrics/raw/0.0.1/install.py| sudo python3💡 | You may pass arguments to the installer – even in the above form – separated from the Python executable with a single dash: curl -sL https://github.com/internet-equity/netrics/raw/0.0.1/install.py| python3 - --helpAnd, for example, if the installer cannot find a suitable location for the Netrics executables – one which is both writeable by the current user and on the |
Alternatively, download the installer to interact with it:
curl -LO https://github.com/internet-equity/netrics/raw/0.0.1/install.pypython3 install.py --help
Netrics software is implemented inPython and requires Python version 3.8 or above.
Casual inspection of your system’s/usr/bin/ directory may indicate which versions of Python you have installed:
ls -1 /usr/bin/python3.*Python installed? Great. That’s all the pre-built bundle needs.
Find the appropriate build for your system on theNetrics release announcement page.
For example, the below package includes the Netrics software bundle version0.0.1, and all its CLI commands, pre-built for the ARM CPU architecture (aarch64) and Python version 3.8 (py38):
netrics-all-0.0.1-py38-aarch64.zip
Having downloaded the above ZIP file, installation is as simple as the following:
sudo unzip netrics-all-0.0.1-py38-aarch64.zip -d /usr/local/bin
Alternatively, install Netrics to any directory from which you’d like to execute it, or to any writeable directory on your PATH,e.g.:
unzip netrics-all-0.0.1-py38-aarch64.zip -d~/.local/bin💡 | If you’re into that kind of thing, a (multi)-line no-brainer – making use of the TAR file alternative – might look like the following: ARCH=$(arch)PYTHON=py$(python3 -c"import sys; print(*sys.version_info[:2], sep='')")NETRICS=$(curl -s https://api.github.com/repos/internet-equity/netrics/releases/latest| jq -r .tag_name)curl -sL https://github.com/internet-equity/netrics/releases/download/$NETRICS/netrics-all-$NETRICS-$PYTHON-$ARCH.tar| sudo tar -xf - -C /usr/local/bin/ …But, why not use theone-line installer, instead? |
With Python installed, the Package Installer for Python –pip – should be installed, as well. If not, it may be installed with the following:
python3 -m ensurepip --upgrade
Netrics may then be installed viapip install. Refer to the sections that follow for specific information on downloading and installing the Netrics package.
💡 | Commands of the form As appropriate to the target system, you might instead install Netrics under your user path only: pip install --user [URI] Or, to ensure successful installation, consider avirtual environment, with which to isolate the library’s dependencies from others on the system. Better yet, consider the additional utilitypipx: pipx install [URI] With Finally, consider one of the preceding installation methods, such as theone-line installer, which will attempt to install Netrics as apre-built, pre-packaged bundle, without risking the above concerns. |
Netrics may be installed from itssource repository viapip.
To make use of an SSH configuration,e.g.:
pip install git+ssh://git@github.com/chicago-cdac/netrics.git
Note that the above URI may also include a Git reference specification, such as a tag or a branch:
pip install git+ssh://git@github.com/chicago-cdac/netrics.git@main
Alternatively, you may supply HTTPS URIs to the above.
With HTTPS, it is also possible to request a code archive of a particular tag or branch, (which may be faster than the above):
pip install https://github.com/chicago-cdac/netrics/archive/REF.zip
Outside ofthe Docker container, andinstallation by a package manager or bythe one-line installer, initialization is suggested to set up your Netrics installation.
Pre-built bundle,PyPI andsource distributions feature thenetrics sub-commandinit:
netrics init
The above, (executed from a standard shell), will walk you through the process of initializing your system for Netrics, (executing all tasks which follow below).
To initialize configuration in particular,init features the sub-commandconf:
netrics init conf
The above will copy the built-in default configuration distributed with Netrics to the appropriate path on your system (or to a specified path). From there, this configuration may be customized.
ℹ️ | Netrics isreally a distribution ofFate, and as such shares its configuration and execution scheme. |
Netrics expects two configuration files:measurements anddefaults.
Should either file not be found on the sytem, Netrics will fall back to its built-in configuration. As necessary for your installation, to initialize these files for customization, seeInitialization.
💡 | Netrics supports both TOML and YAML configuration formats. |
ℹ️ | The commands In lieu of these, files |
The measurements file configures and schedules programs to be executed by Netrics. These configured programs are alternately called "measurements," "tasks" and "modules."
Only one setting is strictly required of a measurement: itsschedule. (Without this setting, a measurementmay be executed ad-hoc via thedebug command; however, it cannot be scheduled.)
Additionally, measurement configuration must indicate what is to be executed. This may be indicated either via the settingexec orcommand, or it will be inferred.
The example below demonstrates configuration options further.
| measurements.toml | measurements.yaml |
|---|---|
[ping]schedule ="0 */6 * *"[ping-slim]command ="ping"schedule ="*/30 * * *"param = {target = ["google.com"]}[cowsay]exec ="cowsay"schedule ="@hourly"param ="yo dawg"path = {result ="/root/cows/"}[cowsay-custom]exec = ["cowsay","-e","^^"]schedule ="@daily"param ="i heard you like cows"# no file extension for result files; do not attempt to detectformat = {result =""}path = {result ="/root/cows/"}[dump-db]exec = ["sh","/home/ubuntu/dump-db"]schedule ="@daily"format = {result ="csv"} | ping:schedule:"0 */6 * *"ping-slim:command:pingschedule:"*/30 * * *"param:{target: [google.com]}cowsay:exec:cowsayschedule:"@hourly"param:yo dawgpath:{result: /root/cows/}cowsay-custom:exec:[cowsay, -e, ^^]schedule:"@daily"param:i heard you like cows# no file extension for result files; do not attempt to detectformat:{result: null}path:{result: /root/cows/}dump-db:exec:[sh, /home/ubuntu/dump-db]schedule:"@daily"format:{result: csv} |
In the above example, the "measurements"cowsay,cowsay-custom anddump-db each specify theexec setting. With this setting, a measurement may executeany system command.
Note, however, that Netricswill not, by default, launch a shell to interpret the value of your measurement’sexec setting. This setting must be either a string or an array indicating an executable command available through the process environment’sPATH. Command arguments areonly accepted via array notation.
Netrics further features a plug-in system whereby programs abiding byits contract are granted greater functionality. Any programmay abide by this contract, (including those specified viaexec). Programs available through the process environment’sPATH under a name bearing thenetrics- prefix –e.g.,netrics-ping – enjoy the small privilege of becoming Netrics "commands."
In the above example, the measurementping-slim specifies the commandping. This simply instructs Netrics to execute a program under the namenetrics-ping.
The example measurementping neglects to specify a command at all. Theping command will be inferred for it as well – this is: the programnetrics-ping.
Under theframework contract, programs may be given configured parameters via their process’s standard input.
The example measurementping-slim is configured to input to theping command the parameters:
{"target": ["google.com"]}Thecowsay measurement, on the other hand, is configured with the scalar string input:"yo dawg".
Structured (non-scalar) parameters are serialized to JSON by default. (This default may be overidden either per-measurement or globally. See:format.)
Theformat setting, when specified,must be a mapping.
The defaults of settings nested underformat may be overidden per-measurement or globally.
param
The nested settingparam indicates the serialization format of structured parameters (given by top-level measurement settingparam). JSON (json), TOML (toml) and YAML (yaml) serialization formats are supported. The default format is JSON.
result
The nested settingresult indicates in what format results will be produced by the measurement’sstandard output.
The default for this setting is "auto" – Netrics will attempt to characterize themeasurement result format, so as to assign an appropriate extension to its generated file name.JSON (json), TOML (toml) and YAML (yaml) serializations support "auto" characterization.
Alternatively, the result format may be specified explicitly: in addition to the valuesjson,toml andyaml, this setting supportscsv.
Finally, result characterization may be disabled by any "false-y" value, such asnull (in YAML), or the empty string (generally).
Thepath setting, when specified,must be a mapping.
The defaults of settings nested underpath may be overidden per-measurement or globally.
result
The nested settingresult indicates the directory path to which measurement result files are written. The default path is installation-dependent (e.g.,/var/log/netrics/result/ when Netrics is installed system-wide).
Settingsformat andpath may be overidden globally via the defaults file, as in the example below.
| defaults.toml | defaults.yaml |
|---|---|
[format]param ="json"result ="auto"[path]result ="/var/log/netrics/result/" | format:param:jsonresult:autopath:result:/var/log/netrics/result/ |
Netrics includes a set of built-in measurement commands, such asnetrics-ping.
Any task configuration may specify thecommand setting with the valueping to make use of this built-in; (or, a task with the labelping may omit this setting to default to this command).
| command | executable | parameters (defaults) | description |
|---|---|---|---|
|
| … | … |
|
| … | … |
|
| … | … |
|
| … | Run a network diagnostic test using Measurement Lab’s ndt7-client. |
|
| … | Run a network diagnostic test using Ookla’s Speedtest. |
|
| {"count":10,"interval":0.25,"targets": ["facebook.com","google.com","nytimes.com" ],"timeout":5,"verbose":false} | Execute the |
|
| … | … |
The Netrics framework invokes executables available through the operating system. As such, built-in measurements enjoy next-to-nil privilege relative to any other installed executable. Measurements abiding by the framework’s expectations may be scheduled with a minimum of effort, provided that they follow the contract below.
The framework communicates with the programs it executes through the operating system, principally via processes' standard input, standard output, standard error and exit code.
Minima
An executed task must at minimum:
write its result to standard output (though this is ignored if reporting failure)
report its success or failure via exit code (only exit code
0indicates successful execution)
💡 | The examples below represent shell scripts; and, Netrics’s built-in measurements are implemented in Python. Tasks may executeany program. And "commands" named with thenetrics- prefix may themselves be implemented inany language. |
This may be accomplished as simply as the following example executable, which reports network status as indicated by sending an ICMP Echo request (ping) to host example.com:
#!/bin/sh# For this simple example we're not interested in detailed ping data# (and we don't want it echo'd as a "result") -- discard it.ping -c 1 -w 1 example.com> /dev/null (1)# Rather, determine our result according to ping's own exit codes.case$?in0)echo'{"example.com": "FAST ENOUGH"}' (2)exit 0 (3);;1) (4)echo'{"example.com": "TOO SLOW"}'exit 0;;*)exit 1 (5);;esac
As noted in the preceding comment, care must be taken with shell scripts which pass through sub-processes' standard output and error. Any standard output is treated as part of a measurement’s "result." And any standard error will be logged.
Results are reported via an executable’s standard output. Results may be inany plain text format (or none at all). (JSON is merely a handy one, and enjoys automatic detection.)
The default exit code of a program is of course
0. It doesn’t hurt to make this explicit: any non-zero exit code indicates to the framework a failed execution. Failures are logged as such. Any content written to standard output by a failed task is not recorded as a measurement result.The underlying
pingutility (from Linux package iputils) communicates state with its own exit codes: exit code1indicates packets were not received. This is an error state for iputils; but, for our measurement, this is a valid result. We detect this state, report it, and exit with the success code0.Any other case indicates an error with our measurement. We exit with a non-zero exit code to notify the framework of this failure. As this is a shell script, any standard error written by the
pingutility has been passed through and captured; (and, we could write our own).
Parameterization
Tasks' input may be configured in themeasurements file and is supplied to executables via their standard input. Structured input is serialized in JSON format by default. (See:param.)
We might extend our example to read and process JSON-encoded standard input via thejq utility:
#!/bin/bash# collect targets from standard input parameters## we expect input of the form:## {# "targets": ["host0", "host1", ..., "hostN"]# }#PARAM="$(jq -r'.targets | join(" ")'2> /dev/null)" (1)# default to just Wikipediaif [-z"$PARAM" ];then PARAM="wikipedia.org"fi (2)# run all measurements concurrently# (and collect their PIDs for inspection)PIDS=()fordestin$PARAM;do ping -c 1 -w 1$dest> /dev/null& pids+=($!)done (3)# collect measurements' exit codesCODES=()forpidin${PIDS[*]};dowait$pid CODES+=($?)done# convert exit code to a statusSTATUS=()forcodein${CODES[*]};docase"$code"in 0) STATUS+=("FAST ENOUGH") ;; 1) STATUS+=("TOO SLOW") ;;*)echo'FAILURE!!!'>&2exit 1 (4) ;;esacdone# generate reportjq' [ .targets, .statuses | split(" ") ] | transpose | map( {(.[0]): .[1]})'<<DOC { "targets": "$PARAM", "statuses": "${STATUS[@]}" }DOC (5)
It’s perfectly reasonable to log issues with parameterization to standard error. But theremight be no input at all. Rather than differentiate these cases in our shell script, we just silence any complaints from
jq.The user may elect not to configure any parameters, and so we fall back to a default.
Our underlying measurement is much the same as before; only now, we test each configured target in parallel.
Any of our measurements could still fail in a way we don’t know how to handle. In this case, this task elects to report the entire run as a failure. Additionally, a profoundly interesting message is logged via standard error.
Yikes!!! We elected to write our executable in Bash to show how simple itcan be. But there’s nothing simple about that. Admittedly, we might have serialized our result in any format – CSV is supported, for one; and, even space- or tab-separated values would suffice, here. But, now we’ve demonstrated the limits of this implementation, as well. Foryour executable, you might select another language….
For more robust examples, consultNetrics’s built-in measurement commands (implemented in Python).
Measurement executables may nominally associate themselves with the Netrics framework and become "commands" by simply being available on the process environmentPATH under a name beginning with the prefixnetrics-.
In this manner,built-in measurements such asnetrics-ping are distributed alongside thenetrics framework command, and may be referred to in configuration asping.
Any other discovered executable, such asnetrics-cowsay[1], will be treated the same way.
Any executable may be invoked (with optional arguments) by the Netricsexecute command:
netrics debug execute [options]command [arguments]The above generates an execution report for use in development and debugging.
Options such as-i|--stdin may be useful to supply measurment parameters to the executable according to theframework’s contract.
Once added to Netrics configuration, executables becometasks, (also known asmeasurements ormodules). These may be invoked ad-hoc by therun command:
netrics debug run [options] task
The options and output of therun command are similar to those ofexecute.
Unlike with scheduled tasks, the results of tasks performed byrun are not, by default, persisted to file. Either specify option--record to capture these as configured, or option--stdout to capture these at an arbitrary path.
❗ | Note that this – and further sections – arenot necessary to deploy your own Netrics measurements. Builtins are distributed withall installations forall users of Netrics: to demonstrate intended functionality, to documentone manner of designing robust measurements, and as a common core of network measurements. If you believe your measurement should be included in this manner and for these reasons, read on. If you merely want to add your measurement to your own deployments, consultNovel measurements andConfiguration. |
Havingtested your novel measurement, it might be added to the Netrics framework for availability acrossall installations of this software viapull request.
At this time, all Netrics builtins are implemented in Python, as simple submodules of the Netrics sub-packagenetrics.measurement. As such, built-in measurement module files neednot be marked with the "execute" bitnor need they include a "shebang" line (e.g.#!/usr/bin/env python3).
Name your module succinctly and appropriately for its functionality. Do not include any
netrics-prefix.E.g.:MOD.py.Place your module under the path:src/netrics/measurement/.
The functionality of your measurement should be invoked entirely by a module-level function:
main(). This function will be invoked without arguments.Optional: Enable invocation of your module through the package –
python -m netrics.measurement.MOD– with the final module-level block:if__name__=='__main__':main()
Configure the Netrics distribution to install your command executable by adding a line to thepyproject.toml file section
tool.poetry.scripts,e.g.:[tool.poetry.scripts]netrics-MOD ="netrics.measurement.MOD:main"
Add your command to this document’s table ofbuilt-in measurements.
❗ | Note that these steps are not necessary to the development ofNovel measurements. |
The Netrics framework is implemented inPython and the framework’s distribution is managed viaPoetry.
Python v3.8 may be supplied by an operating system package manager, bypython.org, or by a utility such aspyenv; pyenv isrecommended for development butnot required.
With Python installed,Poetry may be installedaccording to its instructions.
💡 | If you are managing your own virtual environment,e.g. viapyenv-virtualenv, then this step may be as simple aspip install poetry. However, this tooling is not required, and Poetry offers its own automated set-up, as well as management of virtual environments. |
Finally, from the root directory of a repository clone, the framework may be installed in development mode:
poetry install
ℹ️ | Poetry will use any existing, activated virtual environment, or create its own into which dependencies will be installed. |
Thenetrics command is now available for use in your development environment.
❗ | For simplicity, it is presumed that A virtual environment under management by Poetry may be activated via sub-shell with: poetry shell Alternatively, any command installed into Poetry’s virtual environment may be executed ad-hoc via the poetry run netrics ... |
netrics-cowsay … yet!About
The network measurements scheduling framework