Repository files navigation

Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed.

With Miller, you get to use named fields without needing to count positionalindices, using familiar formats such as CSV, TSV, JSON, JSON Lines, andpositionally-indexed. Then, on the fly, you can add new fields which arefunctions of existing fields, drop fields, sort, aggregate statistically,pretty-print, and more.

• Miller operates onkey-value-pair data while the familiarUnix tools operate on integer-indexed fields: if the natural data structure forthe latter is the array, then Miller's natural data structure is theinsertion-ordered hash map.

• Miller handles avariety of data formats,including but not limited to the familiarCSV,TSV, andJSON/JSON Lines.(Miller can handlepositionally-indexed data too!)

In the above image you can see how Miller embraces the common themes ofkey-value-pair data in a variety of data formats.

• Miller in 10 minutes
• A Guide To Command-Line Data Manipulation
• A quick tutorial on Miller
• Tools to manipulate CSV files from the Command Line
• www.togaware.com/linux/survivor/CSV_Files.html
• MLR for CSV manipulation
• Linux Magazine: Process structured text files with Miller
• Miller: Command Line CSV File Processing
• Miller - A Swiss Army Chainsaw for CSV Data, Data Science and Data Munging
• Pandas Killer: mlr, the Scientist

• Full documentation
• Miller's license is two-clause BSD
• Notes about issue-labeling in the Github repo
• Active issues

There's a good chance you can get Miller pre-built for your system:

See alsoREADME-versions.md for a full list of package versions. Note that long-term-support (LtS) releases will likely be on older versions.

See alsobuilding from source.

• Discussion forum:https://github.com/johnkerl/miller/discussions
• Feature requests / bug reports:https://github.com/johnkerl/miller/issues
• How to contribute:https://miller.readthedocs.io/en/latest/contributing/

• First:
  • cd /where/you/want/to/put/the/source
  • git clone https://github.com/johnkerl/miller
  • cd miller
• Withmake:
  • To build:make. This takes just a few seconds and produces the Miller executable, which is./mlr (or.\mlr.exe on Windows).
  • To run tests:make check.
  • To install:make install. This installs the executable/usr/local/bin/mlr and manual page/usr/local/share/man/man1/mlr.1 (so you can doman mlr).
  • You can do./configure --prefix=/some/install/path beforemake install if you want to install somewhere other than/usr/local.
• Withoutmake:
  • To build:go build github.com/johnkerl/miller/v6/cmd/mlr.
  • To run tests:go test github.com/johnkerl/miller/v6/pkg/... andmlr regtest.
  • To install:go install github.com/johnkerl/miller/v6/cmd/mlr will install toGOPATH/bin/mlr.
• See also the doc page onbuilding from source.
• For more developer information please seeREADME-dev.md.

• README-dev.md
• How to contribute

License: BSD2

• Miller ismulti-purpose: it's useful fordata cleaning,data reduction,statistical reporting,devops,systemadministration,log-file processing,format conversion, anddatabase-query post-processing.

• You can use Miller to snarf and mungelog-file data, including selectingout relevant substreams, then produce CSV format and load that intoall-in-memory/data-frame utilities for further statistical and/or graphicalprocessing.

• Miller complementsdata-analysis tools such asR,pandas, etc.:you can use Miller toclean andprepare your data. While you can dobasic statistics entirely in Miller, its streaming-data feature andsingle-pass algorithms enable you toreduce very large data sets.

• Miller complements SQLdatabases: you can slice, dice, and reformat dataon the client side on its way into or out of a database. You can also reap someof the benefits of databases for quick, setup-free one-off tasks when you justneed to query some data in disk files in a hurry.

• Miller also goes beyond the classic Unix tools by stepping fully into ourmodern,no-SQL world: its essential record-heterogeneity property allowsMiller to operate on data where records with different schema (field names) areinterleaved.

• Miller isstreaming: most operations need only a single record inmemory at a time, rather than ingesting all input before producing any output.For those operations which require deeper retention (sort,tac,stats1),Miller retains only as much data as needed. This means that wheneverfunctionally possible, you can operate on files which are larger than yoursystem’s available RAM, and you can use Miller intail -f contexts.

• Miller ispipe-friendly and interoperates with the Unix toolkit.

• Miller's I/O formats includetabular pretty-printing,positionallyindexed (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others.

• Miller doesconversion between formats.

• Miller'sprocessing is format-aware: e.g. CSVsort andtac keep header lines first.

• Miller has high-throughputperformance on par with the Unix toolkit.

• Miller is written in portable, modern Go, withzero runtime dependencies.You can download or compile a single binary,scp it to a faraway machine,and expect it to work.

Today I discovered Miller—it's like jq but for CSV:https://t.co/pn5Ni241KM

Also, "Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data."@GreatBlueC@nfmcclure

— Adrien Trouillaud (@adrienjt)September 24, 2020

Underappreciated swiss-army command-line chainsaw.

"Miller is like awk, sed, cut, join, and sort for [...] CSV, TSV, and [...] JSON."https://t.co/TrQqSUK3KK

— Dirk Eddelbuettel (@eddelbuettel)February 28, 2017

Miller looks like a great command line tool for working with CSV data. Sed, awk, cut, join all rolled into one:http://t.co/9BBb6VCZ6Y

— Mike Loukides (@mikeloukides)August 16, 2015

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV:http://t.co/1zPbfg6B2W - handy tool!

— Ilya Grigorik (@igrigorik)August 22, 2015

Btw, I think Miller is the best CLI tool to deal with CSV. I used to use this when I need to preprocess too big CSVs to load into R (now we have vroom, so such cases might be rare, though...)https://t.co/kUjrSSGJoT

— Hiroaki Yutani (@yutannihilat_en)April 21, 2020

Miller: a *format-aware* data munging tool By@__jo_ker__ to overcome limitations with *line-aware* workshorses like awk, sed et alhttps://t.co/LCyPkhYvt9

The project website is a fantastic example of good software documentation!!

— Donny Daniel (@dnnydnl)September 9, 2018

Holy holly data swiss army knife batman! How did no one suggest Millerhttps://t.co/JGQpmRAZLv for solving database cleaning / ETL issues to me before

Congrats to@__jo_ker__ for amazingly intuitive tool for critical data management tasks!#DataScienceandLaw#ComputationalLaw

— James Miller (@japanlawprof)June 12, 2018

🤯@__jo_ker__'s Miller easily reads, transforms, + writes all sorts of tabular data. It's standalone, fast, and built for streaming data (operating on one line at a time, so you can work on files larger than memory).

And the docs are dream. I've been reading them all morning!https://t.co/Be2pGPZK6t

— Benjamin Wolfe (he/him) (@BenjaminWolfe)September 9, 2021

Thanks to all the fine people who help make Miller better (emoji key):

Andrea Borruso 🤔🎨	Shaun Jackman 🤔	Fred Trotter 🤔🎨	komosa 🤔	jungle-boogie 🤔	Thomas Klausner 🚇	Stephen Kitt 📦
Leah Neukirchen 🤔	Luigi Baldoni 📦	Hiroaki Yutani 🤔	Daniel M. Drucker 🤔	Nikos Alexandris 🤔	kundeng 📦	Victor Sergienko 📦
Adrian Ho 🎨	zachp 📦	David Selassie 🤔	Joel Parker Henderson 🤔	Michel Ace 🤔	Matus Goljer 🤔	Richard Patel 📦
Jakub Podlaha 🎨	Miodrag Milić 📦	Derek Mahar 🤔	spmundi 🤔	Peter Körner 🛡️	rubyFeedback 🤔	rbolsius 📦
awildturtok 🤔	agguser 🤔	jganong 🤔	Fulvio Scapin 🤔	Jordan Torbiak 🤔	Andreas Weber 🤔	vapniks 📦
Zombo 📦	Brian Fulton-Howard 📦	ChCyrill 🤔	Jauder Ho 💻	Paweł Sacawa 🐛	schragge 📖	Jordi 📖🤔

This project follows theall-contributors specification. Contributions of any kind are welcome!