MILLER(1) MILLER(1) (NAME) mlr - awk sed cut join sort CSV JSON (SYNOPSIS) : mlr [flags] [verb] [verb-dependent options ...] { } verb "cat" : mlr --c2j < example.csv : mlr --csv sort -f shape example.csv verb "then" verb : mlr --csv stats1 -a min,mean,max -f quantity then sort -f color example.csv 'mlr help topics' . https://miller.readthedocs.io . (DESCRIPTION) Miller - (key-value-pair) : Miller (insertion-ordered hash map) . CSV TSV JSON ( ). (Miller .) 6.21.0 mlr . (EXAMPLES) mlr --icsv --opprint cat example.csv mlr --icsv --opprint sort -f shape example.csv mlr --icsv --opprint sort -f shape -nr index example.csv mlr --icsv --opprint cut -f flag,shape example.csv mlr --csv filter '$color == "red"' example.csv mlr --icsv --ojson put '$ratio = $quantity / $rate' example.csv mlr --icsv --opprint --from example.csv sort -nr index then cut -f shape,quantity (FILE FORMATS) CSV/CSV-lite: TSV: +---------------------+ | apple,bat,cog | | 1,2,3 | Record 1: "apple":"1", "bat":"2", "cog":"3" | 4,5,6 | Record 2: "apple":"4", "bat":"5", "cog":"6" +---------------------+ JSON ( ): +---------------------+ | [ | | { | | "apple": 1, | Record 1: "apple":"1", "bat":"2", "cog":"3" | "bat": 2, | | "cog": 3 | | }, | | { | | "dish": { | Record 2: "dish.egg":"7", | "egg": 7, | "dish.flint":"8", "garlic":"" | "flint": 8 | | }, | | "garlic": "" | | } | | ] | +---------------------+ JSON Lines ( ): +------------------------------------------------+ | {"apple": 1, "bat": 2, "cog": 3} | | {"dish": {"egg": 7, "flint": 8}, "garlic": ""} | +------------------------------------------------+ Record 1: "apple":"1", "bat":"2", "cog":"3" Record 2: "dish:egg":"7", "dish:flint":"8", "garlic":"" YAML ( = ---): +---------------------+ | apple: 1 | | bat: 2 | Record 1: "apple":"1", "bat":"2", "cog":"3" | cog: 3 | | --- | | dish: 7 | Record 2: "dish":"7", "egg":"8" | egg: 8 | +---------------------+ PPRINT: (pretty-printed) +---------------------+ | apple bat cog | | 1 2 3 | Record 1: "apple:"1", "bat":"2", "cog":"3" | 4 5 6 | Record 2: "apple":"4", "bat":"5", "cog":"6" +---------------------+ Markdown tabular: +-----------------------+ | | apple | bat | cog | | | | --- | --- | --- | | | | 1 | 2 | 3 | | Record 1: "apple:"1", "bat":"2", "cog":"3" | | 4 | 5 | 6 | | Record 2: "apple":"4", "bat":"5", "cog":"6" +-----------------------+ XTAB: +---------------------+ | apple 1 | Record 1: "apple":"1", "bat":"2", "cog":"3" | bat 2 | | cog 3 | | | | dish 7 | Record 2: "dish":"7", "egg":"8" | egg 8 | +---------------------+ DKVP: - ( Miller) +---------------------+ | apple=1,bat=2,cog=3 | Record 1: "apple":"1", "bat":"2", "cog":"3" | dish=7,egg=8,flint | Record 2: "dish":"7", "egg":"8", "3":"flint" +---------------------+ DKVPX: - CSV +----------------------------------+ | apple=1,bat=2,cog=3 | Record 1: "apple":"1", "bat":"2", "cog":"3" | "x,y"="a,b,c",z=3 | Record 2: "x,y":"a,b,c", "z":"3" +----------------------------------+ NIDX: ( ) +---------------------+ | the quick brown | Record 1: "1":"the", "2":"quick", "3":"brown" | fox jumped | Record 2: "1":"fox", "2":"jumped" +---------------------+ DCF: +------------+ | apple: 1 | | bat: 2 | | cog: 3 | | | | dish: 7 | | egg: 8 | | 3: flint | +------------+ 'mlr help {topic}' : : mlr help topics mlr help basic-examples mlr help file-formats : mlr help flags mlr help flag mlr help list-separator-aliases mlr help list-separator-regex-aliases mlr help comments-in-data-flags mlr help compressed-data-flags mlr help csv/tsv-only-flags mlr help dkvp-only-flags mlr help file-format-flags mlr help flatten-unflatten-flags mlr help format-conversion-keystroke-saver-flags mlr help json-only-flags mlr help legacy-flags mlr help markdown-only-flags mlr help miscellaneous-flags mlr help output-colorization-flags mlr help pprint-only-flags mlr help profiling-flags mlr help separator-flags : mlr help list-verbs mlr help usage-verbs mlr help verb : mlr help list-functions mlr help list-function-classes mlr help list-functions-in-class mlr help usage-functions mlr help usage-functions-by-class mlr help function : mlr help list-keywords mlr help usage-keywords mlr help keyword : mlr help auxents mlr help terminals mlr help mlrrc mlr help output-colorization mlr help type-arithmetic-info mlr help type-arithmetic-info-extended : mlr -g = mlr help flags mlr -l = mlr help list-verbs mlr -L = mlr help usage-verbs mlr -f = mlr help list-functions mlr -F = mlr help usage-functions mlr -k = mlr help list-keywords mlr -K = mlr help usage-keywords 'mlr help ...' '...' 'mlr help flag' 'mlr help verb' 'mlr help function' 'mlr help keyword' . () 'mlr help find ...' 'mlr help find map' "map" . (VERB LIST) altkv bar bootstrap bootstrap-ci case cat check clean-whitespace count-distinct count count-similar cut decimate describe fill-down fill-empty filter flatten format-values fraction gap grep group-by group-like gsub having-fields head histogram json-parse json-stringify join label latin1-to-utf8 least-frequent merge-fields most-frequent nest nothing put rank regularize remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle skip-trivial-records sort sort-within-records sparkline sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1 unflatten uniq unspace unsparsify (FUNCTION LIST) abs acos acosh antimode any append apply arrayify asin asinh asserting_absent asserting_array asserting_bool asserting_boolean asserting_bytes asserting_empty asserting_empty_map asserting_error asserting_float asserting_int asserting_map asserting_nonempty_map asserting_not_array asserting_not_empty asserting_not_map asserting_not_null asserting_null asserting_numeric asserting_present asserting_string atan atan2 atanh base64_decode base64_encode bitcount boolean bytes capitalize cbrt ceil clean_whitespace collapse_whitespace concat contains cos cosh count depth dhms2fsec dhms2sec distinct_count erf erfc every exec exp expm1 flatten float floor fmtifnum fmtnum fold format fsec2dhms fsec2hms get_keys get_values gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue hex_decode hex_encode hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent is_array is_bool is_boolean is_bytes is_empty is_empty_map is_error is_float is_int is_map is_nan is_nonempty_map is_not_array is_not_empty is_not_map is_not_null is_null is_numeric is_present is_string joink joinkv joinv json_parse json_stringify kurtosis latin1_to_utf8 leafcount leftpad length localtime2gmt localtime2nsec localtime2sec log log10 log1p logifit lstrip madd mapdiff mapexcept mapselect mapsum max maxlen md5 mean meaneb median mexp min minlen mmul mode msub next nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime null_count os percentile percentiles pow qnorm reduce regextract regextract_or_else rightpad round roundm rstrip sec2dhms sec2gmt sec2gmtdate sec2hms sec2localdate sec2localtime select sgn sha1 sha256 sha512 sin sinh skewness sort sort_collection sparkline splita splitax splitkv splitkvx splitnv splitnvx sqrt ssub stat stddev strfntime strfntime_local strftime strftime_local string strip strlen strmatch strmatchx strpntime strpntime_local strptime strptime_local sub substr substr0 substr1 sum sum2 sum3 sum4 sysntime system systime systimeint tan tanh tolower toupper truncate typeof unflatten unformat unformatx upntime uptime urand urand32 urandelement urandint urandrange utf8_to_latin1 variance version ! != !=~ % & && * ** + - . .* .+ .- ./ / // < << <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ | || ~ Miller : # This is a comment for a CSV file a,b,c 1,2,3 4,5,6 : * . * . ( (opt-in) .) * `--pass-comments` . . . --pass-comments ( `#`) . --pass-comments-with {string} . CSV . --skip-comments ( `#`) . --skip-comments-with {string} . CSV . (COMPRESSED-DATA FLAGS) Miller . * Miller : `--bz2in` `--gzin` `--zin``--zstdin` * Miller : `--prepipe` `--prepipex` `--prepipe` `--prepipex` . prepipe `{command} < {filename}` . prepipex `{command} {filename}` . : mlr --prepipe gunzip mlr --prepipe zcat -cf mlr --prepipe xz -cd mlr --prepipe cat . . ( ) : `mlr ... | {your compression command} > outputfilenamegoeshere` `--prepipe` `--prepipex` . `--prepipe` `--gzin`/`--bz2in`/`--zin``--zin` . --bz2in bzip2 Miller. `.bz2` . --gzin gzip Miller. `.gz` . --prepipe {decompression command} `gunzip < myfile.csv.gz | mlr ...`. `.mlrrc` . --prepipe-bz2 `--prepipe bz2` `.mlrrc` . --prepipe-gunzip `--prepipe gunzip` `.mlrrc` . --prepipe-zcat `--prepipe zcat` `.mlrrc` . --prepipe-zstdcat `--prepipe zstdcat` `.mlrrc` . --prepipex {decompression command} `--prepipe` : `<` . `unzip -qc` . `.mlrrc` . --zin zlib Miller. `.z` . --zstdin zstd Miller. `.zstd` . CSV/TSV (CSV/TSV-ONLY FLAGS) CSV . --allow-ragged-csv-input --ragged --allow-ragged-tsv-input . implicit-header . --csv-trim-leading-space CSV . '"foo", "bar' RFC-4180 . --headerless-csv-output --ho --headerless-tsv-output CSV/TSV CSV/TSV . --implicit-csv-header --headerless-csv-input --hi --implicit-tsv-header 1,2,3,... . : `label` . --lazy-quotes . --no-auto-unsparsify CSV/TSV: . RFC 4180 . --no-implicit-csv-header --no-implicit-tsv-header `--implicit-csv-header`. -- `mlr join` () CSV/TSV . : `mlr --csv --implicit-csv-header join --no-implicit-csv-header -l your-join-in-with-header.csv ... your-headerless.csv`. --quote-all (double-quoting) CSV. -N `--implicit-csv-header --headerless-csv-output`. DKVP . --incr-key DKVP . : `a=10,b=20,30,d=40,50` `$a=10,$b=20,$3=30,$d=40,$5=50` . . : `a=10,b=20,30,d=40,50` `$a=10,$b=20,$1=30,$d=40,$2=50` . (FILE-FORMAT FLAGS) Miller File formats `mlr help file-formats` . : `--csv` CSV `--icsv --opprint` CSV (pretty-printed). `--format1 --oformat2` `--iformat1 --oformat2` . `format1` `format2` . --asv or --asvlite ASV . --csv or -c or --c2c CSV . --csvlite CSV-lite . --dcf (DCF) . --dkvp or --d2d DKVP . --dkvpx DKVPX . --gen-field-name --igen. : "i". --gen-start --igen. : 1. --gen-step --igen. : 1. --gen-stop --igen. : 100. --iasv or --iasvlite ASV . --icsv CSV . --icsvlite CSV-lite . --idcf (DCF) . --idkvp DKVP . --igen --gen-field-name --gen-start --gen-step --gen-stop. seqgen / . --ijson JSON . --ijsonl JSON Lines . --imd or --imarkdown . --inidx NIDX . --io {format name} format name . : `--io csv` `--csv` . --ipprint PPRINT . --irecutils GNU recutils (.rec) . --itsv TSV . --itsvlite TSV-lite . --iusv or --iusvlite USV . --ixtab XTAB . --iyaml YAML . --json or -j or --j2j JSON . --jsonl or --l2l JSON Lines . --md or --markdown . --nidx or --n2n NIDX . --oasv or --oasvlite ASV . --ocsv CSV . --ocsvlite CSV-lite . --odcf (DCF) . --odkvp DKVP . --ojson JSON . --ojsonl JSON Lines . --omd or --omarkdown . --onidx NIDX . --opprint PPRINT . --orecutils GNU recutils (.rec) . --otsv TSV . --otsvlite TSV-lite . --ousv or --ousvlite USV . --oxtab XTAB . --oyaml YAML . --pprint or --p2p PPRINT . --recutils GNU recutils (.rec) . --tsv or -t or --t2t TSV . --tsvlite TSV-lite . --usv or --usvlite USV . --xtab or --x2x XTAB . --xvright XTAB. --yaml or --y2y YAML . -i {format name} format name . : `-i csv` `--icsv` . -o {format name} format name . : `-o csv` `--ocsv` . (FLATTEN-UNFLATTEN FLAGS) (map) (array) Miller JSON/YAML ( - flattening) JSON/YAML JSON/YAML ( - unflattening). flatten/unflatten https://miller.readthedocs.io/en/latest/flatten-unflatten . --flatsep or --jflatsep {string} JSON : `{"a":{"b":3}}` JSON `a:b => 3` . `.` . --no-auto-flatten JSON YAML . : `$y = [7,8,9]` `y.1=7,y.2=8,y.3=9` . `--no-auto-flatten` `$y=[1, 2, 3]` . --no-auto-unflatten JSON YAML JSON YAML . : `y.1=7,y.2=8,y.3=9` `$y=[7,8,9]` . `--no-auto-flatten` `${y.1}=7,${y.2}=8,${y.3}=9` . . c t j l d n x p m y CSV TSV JSON JSON Lines DKVP NIDX XTAB PPRINT markdown YAML . DCF ( --dcf DCF ). GNU recutils ( --recutils recutils ). | In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML | +----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+ | CSV | --c2c,-c | --c2t | --c2j | --c2l | --c2d | --c2n | --c2x | --c2p | --c2m | --c2y | | TSV | --t2c | --t2t,-t | --t2j | --t2l | --t2d | --t2n | --t2x | --t2p | --t2m | --t2y | | JSON | --j2c | --j2t | --j2j,-j | --j2l | --j2d | --j2n | --j2x | --j2p | --j2m | --j2y | | JSONL | --l2c | --l2t | --l2j | --l2l | --l2d | --l2n | --l2x | --l2p | --l2m | --l2y | | DKVP | --d2c | --d2t | --d2j | --d2l | --d2d | --d2n | --d2x | --d2p | --d2m | --d2y | | NIDX | --n2c | --n2t | --n2j | --n2l | --n2d | --n2n | --n2x | --n2p | --n2m | --n2y | | XTAB | --x2c | --x2t | --x2j | --x2l | --x2d | --x2n | --x2x | --x2p | --x2m | --x2y | | PPRINT | --p2c | --p2t | --p2j | --p2l | --p2d | --p2n | --p2x | -p2p | --p2m | --p2y | | Markdown | --m2c | --m2t | --m2j | --m2l | --m2d | --m2n | --m2x | --m2p | | --m2y | | YAML | --y2c | --y2t | --y2j | --y2l | --y2d | --y2n | --y2x | --y2p | --y2m | --y2y | -p `--nidx --fs space --repifs`. -T `--nidx --fs tab`. JSON (JSON-ONLY FLAGS) JSON . --jlistwrap or --jl JSON `[ ]`. JSON . --jvquoteall JSON -- -- (string). --jvstack - JSON ( ). JSON . --no-jlistwrap JSON `[ ]` . JSON Lines . --no-jvstack / JSON. JSON Lines . --no-yarray YAML YAML `---` . --yarray or --ya YAML (top-level). YAML . (LEGACY FLAGS) Miller . (no-op) . --jknquoteint JSON . --jquoteall JSON . --json-fatal-arrays-on-input Miller . --json-map-arrays-on-input Miller . --json-skip-arrays-on-input Miller . --jsonx `--jvstack` Miller 6 . --mmap Miller (memory-mapping) . --no-mmap Miller (memory-mapping) . --ojsonx `--jvstack` Miller 6 . --quote-minimal . . --quote-none . . --quote-numeric . . --quote-original . . --vflatsep . JSON . MARKDOWN (MARKDOWN-ONLY FLAGS) markdown . --md-aligned or --markdown-aligned markdown (padded). --md --md . --omd-aligned or --omarkdown-aligned markdown markdown . ( .) --omd --omd . (MISCELLANEOUS FLAGS) . --errors-json JSON stderr . . `MLR_ERRORS_JSON` (truthy). --fflush . (pipe) . . . --files {filename} . . --from {filename} () . . : `mlr --from a.dat --from b.dat cat` `mlr cat a.dat b.dat` . --hash-records . Miller . --no-hash-records . --hash-records . --infer-int-as-float or -A (float). --infer-none or -S 123 456.7 int/float . --infer-octal or -O 0123 . 00--07 int 08-09 float . --load {filename} DSL put/filter . `--load` `*.mlr` . `put -f` `filter -f` `alias mlr='mlr --load ~/myscripts'` . --mfrom {filenames} () . . `--` . `--from *.csv` `--mfrom *.csv --` . --mload {filenames} `--load` `--mload *.mlr --`. --no-dedupe-field-names `x` `x` `x_2` . `x` `x` . JSON . --no-fflush . . . . . --no-hash-records : --hash-records. --no-shell Miller : `system` `exec` DSL `tee | "command"` `--prepipe`/`--prepipex` (fail cleanly). `MLR_NO_SHELL` (truthy). ( `mlr mcp`) . . --norc .mlrrc. --nr-progress-mod {m} m : os.Stderr m . --ofmt {format} `%.18f` `%.0f` `%9.6e`. sprintf (https://pkg.go.dev/fmt) . . `fmtnum` `format-values` . --ofmte {n} --ofmte 6 --ofmt %.6e . --ofmtf {n} --ofmtf 6 --ofmt %.6f . --ofmtg {n} --ofmtg 6 --ofmt %.6g . --profile or -P {name} [name] .mlrrc ( ). .mlrrc . : https://miller.readthedocs.io/en/latest/customization --records-per-batch {n} (batch) . `tail -f` . : https://miller.readthedocs.io/en/latest/reference-main-flag-list --s-no-comment-strip {file name} -s . : https://miller.readthedocs.io/en/latest/scripting --seed {n} `n` `12345678` `0xcafefeed`. `urand` `urandint` `urand32` `put`/`filter`. --tz {timezone} `$TZ` ( ) . -I (in-place) . . : CSV CSV . -n . `mlr put` `begin`/`end` . ( `--from /dev/null`.) `mlr -n put -v '...'` ( ). -s {file name} . : https://miller.readthedocs.io/en/latest/scripting -x . `(error)` . Miller . . : Windows . : * CSV JSON * CSV JSON regression-test * (online-help) : * stdout stdout TTY . * : : `mlr --csv cat foo.csv` * : : `mlr --csv cat foo.csv > bar.csv` * : : `mlr --csv cat foo.csv | less` * . : * Miller ANSI . Windows Cygwin . * `TERM` . * ANSI ANSI . 0..15 ANSI . : * /: * `export MLR_NO_COLOR=true` `export NO_COLOR=true` stdout+TTY . * `export MLR_ALWAYS_COLOR=true` stdout+TTY . mlr (pipe) `less -r` . * : `--no-color` `-M` `--always-color` `-C`. * 0..255 : * `export MLR_KEY_COLOR=208` `MLR_VALUE_COLOR=33` : `MLR_KEY_COLOR` `MLR_VALUE_COLOR` `MLR_PASS_COLOR` `MLR_FAIL_COLOR` `MLR_REPL_PS1_COLOR` `MLR_REPL_PS2_COLOR` `MLR_HELP_COLOR` * `--key-color 208` `--value-color 33` : `--key-color` `--value-color` `--pass-color` `--fail-color` `--repl-ps1-color` `--repl-ps2-color` `--help-color` * . . "red" "orchid" : `mlr --list-color-names` . 0..255 170 : `mlr --list-color-codes` . "bold" "underline" / "reverse" . "-" "red-bold" "bold-170" "bold-underline" . --always-color -C Miller . `less -r` . --fail-color (: `--list-color-codes` `--list-color-names`) `mlr regtest`. --help-color (: `--list-color-codes` `--list-color-names`) `mlr help`. --key-color (: `--list-color-codes` `--list-color-names`) . --list-color-codes 0..255 170 . --list-color-names `orchid` . --no-color -M Miller . --pass-color (: `--list-color-codes` `--list-color-names`) `mlr regtest`. --value-color (: `--list-color-codes` `--list-color-names`) . PPRINT (PPRINT-ONLY FLAGS) PPRINT . --barred --barred-output (border) PPRINT. --barred-input --pprint . --barred-unicode . --fixed {string} . : 'widths:,,...' left-align left-align-multi-word right-align right-align-multi-word --fw {string} --fixed left-align-multi-word --right PPRINT. --right-align-numeric PPRINT . . markdown (`---:`) . (PROFILING FLAGS) Miller . --cpuprofile {CPU-profile file name} CPU-profile . stderr . 'mlr' . --time stderr . --traceprofile trace-profile . stderr . 'mlr' . Separators . File formats `mlr help file-formats` Miller . : * DKVP `x=1,y=2,z=3` - (newline) . * . * CSV : CSV CSV . * : JSON (colon) JSON . * Miller -- `--ifs` `--ofs` . : * (`--irs` `--ors`) (newline) carriage-return/newline ( ) . * CSV CSV-lite TSV TSV-lite ORS () carriage-return/newline : `--ors crlf` `--ors '\r\n'` RFC-4180 . : * IPS/OPS DKVP XTAB - . * IRS/ORS XTAB . IFS OFS XTAB IFS/OFS -- -- . `--irs/--ors/--rs auto` XTAB `--ifs/--ofs/--fs auto` . ( IFS/OFS XTAB "auto" .) * OFS PPRINT . . * OPS XTAB . * FS/PS markdown RS . * FS PS JSON JSON . * : - : `--fs '|' --ips :` - C `--rs '\r\n' --fs '\t'`. - : ascii_esc = "\x1b" ascii_etx = "\x03" ascii_fs = "\x1c" ascii_gs = "\x1d" ascii_null = "\x00" ascii_rs = "\x1e" ascii_soh = "\x01" ascii_stx = "\x02" ascii_us = "\x1f" asv_fs = "\x1f" asv_rs = "\x1e" colon = ":" comma = "," cr = "\r" crcr = "\r\r" crlf = "\r\n" crlfcrlf = "\r\n\r\n" equals = "=" lf = "\n" lflf = "\n\n" newline = "\n" pipe = "|" semicolon = ";" slash = "/" space = " " tab = "\t" usv_fs = "\xe2\x90\x9f" usv_rs = "\xe2\x90\x9e" - `--ifs-regex` `--ips-regex` : spaces = "( )+" tabs = "(\t)+" whitespace = "([ \t])+" * : Format FS PS RS csv "," N/A "\n" csvlite "," N/A "\n" dcf N/A N/A N/A dkvp "," "=" "\n" dkvpx "," "=" "\n" gen "," N/A "\n" json N/A N/A N/A markdown " " N/A "\n" nidx " " N/A "\n" pprint " " N/A "\n" recutils N/A N/A N/A tsv " " N/A "\n" xtab "\n" " " "\n\n" yaml N/A N/A N/A --fs {string} FS . --ifs {string} FS . --ifs-regex {string} FS . --ips {string} PS . --ips-regex {string} PS . --irs {string} RS . --ofs {string} FS . --ops {string} PS . --ors {string} RS . --ps {string} PS . --repifs IFS: . --rs {string} RS . : mlr aux-list mlr hex mlr lecat mlr termcvt mlr unhex mlr {subcommand} --help . MLRRC (MLRRC) $HOME/.mlrrc ./.mlrrc ( ) . CSV "--csv" .mlrrc / . .mlrrc "--flag" "--option value" "--" . (#) . .mlrrc: # CSV ( # mlr ): csv # CSV JSON # (pretty-printing) : jvstack jlistwrap .mlrrc: * $MLRRC : o "__none__" .mlrrc . o ( ) . mlr ( ). . o $MLRRC .mlrrc . * : o $HOME/.mlrrc . o ./.mlrrc . ( .mlrrc .mlrrc .) * "--norc" .mlrrc . : https://miller.readthedocs.io/en/latest/customization.html (REPL) : mlr repl [] { } -v (AST) stdout Miller . -d -v AST . -D -d . -w -q -s (prompts) --load {DSL script file} . --load "*.mlr" . --mload {DSL script files} -- --load '--mload *.mlr --'. -h|--help . / --icsv --ojson Miller. :open {filenames} Miller REPL . (VERBS) altkv : mlr altkv [] a,b,c,d,e,f a=b,c=d,e=f . : -h|--help . bar : mlr bar [] . --opprint --oxtab . : -f {a,b,c} . --lo {lo} : '0.000000'. --hi {hi} : '100.000000'. -w {n} : '40'. --auto --lo --hi. . -c {character} : '*'. -x {character} : '#'. -b {character} : '.'. -h|--help . . . : mlr bootstrap [] n- . : mlr sample mlr shuffle. : -n {n} . . . -h|--help . bootstrap-ci : mlr bootstrap-ci [] : . {field}_{stat} {field}_{stat}_lo {field}_{stat}_hi . mlr --seed . : mlr bootstrap mlr stats1. : -a {mean,...} : mlr stats1 -a. median ( p50) p{n} n 0..100 . mean . -f {a,b,c} . . -g {d,e,f} (group-by). -n {n} . . 1000 . -c {level} 0 1. 0.95 . -i type=7 R type=1. -h|--help . : mlr --seed 12345 bootstrap-ci -f x,y : mlr --seed 12345 bootstrap-ci -a mean,median -f x -g shape -n 5000 -c 0.99 case : mlr case [] / . : -k . -v . -f {a,b,c} (: ). -u (uppercase). -l (lowercase). -s ( ). -t ( ). -h|--help . cat : mlr cat [] . . : -n "n" . -N {name} {name} . -g {a,b,c} a,b,c. --filename . --filenum ( ) . -h|--help . check : mlr check [] stderr . . : * * : -h|--help . clean-whitespace : mlr clean-whitespace [] / . . DSL lstrip rstrip strip collapse_whitespace clean_whitespace . : -k|--keys-only . -v|--values-only . -h|--help . -k -v -- -k -v . : mlr count-distinct [] . uniq -c. : -f {a,b,c} ( -g). -g {a,b,c} . -x {a,b,c} . -n . -u . -o {name} . : "count". -u . -u (unlashed) . -f a,b -u a b . -f a,b -u a b . -h|--help . count : mlr count [] . : -g {a,b,c} (group-by) a,b,c -n . -g . -o {name} . : "count". -h|--help . count-similar : mlr count-similar [] . : -g {a,b,c} (group-by) a,b,c -o {name} . : "count". -h|--help . cut : mlr cut [] . : -f {a,b,c} a,b,c. -o -f . -x|--complement -f . -r (regular expressions). "ab" "a.*b" "ab" "a.*b" "^ab$" "^a.*b$" . -h|--help . : mlr cut -f hostname,status mlr cut -x -f hostname,status mlr cut -r -f '^status$,sda[0-9]' mlr cut -r -f '^status$,"sda[0-9]"' mlr cut -r -f '^status$,"sda[0-9]"i' ( ) decimate : mlr decimate [] n . : -b (decimate) n . -e (decimate) n (). -g {a,b,c} a,b,c. -n {n} ( 10). -h|--help . describe : mlr describe [] : . / ( ) . : field_name types (int string ) count null_count JSON null distinct_count min, max / ( ) values -- distinct_count -n : * -- 4.1 4.10 . * JSON `mlr --ojson describe` types values (flatten) . * summary ( ) . : -n|--max-values {n} {n} 0 values . : 20. -h|--help . : mlr fill-down [] . <<>> . -a <<>> . : --all . -a|--only-if-absent . <<>> . -a <<>> . -f {a,b,c} fill-down. -h|--help . fill-empty : mlr fill-empty [] fill-value . : -v {string} : "N/A" -S -- '-v 0' 0 0. -h|--help . filter : mlr filter [] {DSL expression} (DSL) . : https://miller.readthedocs.io/en/latest/reference-verbs : -f {file name} DSL ( ). *.mlr . -e {expression} DSL . -f . : / -f -e . -s {name=value} @name . mlr put -s foo=97 '$column += @foo' mlr put 'begin {@foo = 97} $column += @foo' . . -s name1=value1 -s name2=value2. : -s sequence=$SEQUENCE. -x {expression} true false . false . -q . begin / end . -S Miller 6 (no-op) filter/put . . -F Miller 6 (no-op) filter/put . . -w . -W -w . -p (AST) stdout Miller . -d -p AST . -D -d . -E (Echo) DSL . -v -E -p. -X . -v/-d/-D . --explain DSL . 0 --errors-json . -h|--help . -e -f . (;) . : -w -W -p -d -D -E -v -X. (bare-boolean) DSL . < == > . : mlr --csv --from example.csv filter '$color == "red"' mlr --csv --from example.csv filter '$color == "red" && flag == true' : : 'FNR == 1' : 'urand() < 0.001' : '$color != "blue" && $value > 4.2' '($x < 0.5 && $y < 0.5) || ($x > 0.5 && $y > 0.5)' '($name =~ "^sys.*east$") || ($name =~ "^dev.[0-9]+"i)' : '$ab = $a+$b; $cd = $c+$d; $ab != $cd' : 'if (NR < 100) { $x > 0.3; } else { $x > 0.002; } ' 'any' $index 10 20 30 : 'any([10,20,30], func(e) {return $index == e})' https://miller.readthedocs.io/reference-dsl . : mlr flatten [] . : 'a' '{"b": { "c": 4 }}' 'a.b.c' 4 . : -f ( ). -s {string} mlr --flatsep. -h|--help . format-values : mlr format-values [] . * (integer) . * (float) . * (string) . : . fmtnum DSL mlr put . : / . "man printf" . : -i { integer} "%d". : "%06lld" "%08llx". Miller long long long long ll . . -f { float} "%f". : "%8.3lf" "%.6le". Miller (double) double l[efg] . . -s { string} "%s". : "_%s" "%08s". s . . -n int float float. -h|--help . fraction : mlr fraction [] . x=1 x=2 x=3 x=4 : x=1,x_fraction=0.1 x=2,x_fraction=0.2 x=3,x_fraction=0.3 x=4,x_fraction=0.4 : : . . : -f {a,b,c} () / -g {d,e,f} () (group-by) -p [0..100] [0..1]. "_fraction" "_percent" -c : . x=1 x=2 x=3 x=4 : x=1,x_cumulative_fraction=0.1 x=2,x_cumulative_fraction=0.3 x=3,x_cumulative_fraction=0.6 x=4,x_cumulative_fraction=1.0 -h|--help . gap : mlr gap [] n . -n -g . : -g {a,b,c} ( a,b,c) . -n {n} n . -g . -h|--help . grep : mlr grep [] { } . : -i . -v : regex . -a grep . -h|--help . "mlr filter" . "mlr grep" regex . DKVP ( NIDX -a ) OFS "," OPS "=" regex . regex : CSV "x,y,z" "1,2,3" regex DKVP "x=1,y=2,z=3" . grep . grep : "mlr --odkvp ... | grep ... | mlr --idkvp ..." : mlr group-by [options] {comma-separated field names} . : -h|--help . group-like : mlr group-like [options] . : -h|--help . gsub : mlr gsub [options] () regex DSL `gsub`. C \n \t \x1f . \\ . `sub` `ssub` . : -f {a,b,c} . -r {regex} . -a . -h|--help . having-fields : mlr having-fields [options] . : --at-least {comma-separated names} . --which-are {comma-separated names} . --at-most {comma-separated names} . --all-matching {regular expression} regex . --any-matching {regular expression} regex . --none-matching {regular expression} regex . -h|--help . : mlr having-fields --which-are amount,status,owner mlr having-fields --any-matching 'sda[0-9]' mlr having-fields --any-matching '"sda[0-9]"' mlr having-fields --any-matching '"sda[0-9]"i' ( ) head : mlr head [options] n . -g n ( ). : -g {a,b,c} group-by a,b,c. -n {n} . 10. -n -2 n . -h|--help . histogram . lo hi . : mlr histogram [options] : -f {a,b,c} . --lo {lo} . --hi {hi} . --nbins {n} . 20 . --auto --lo --hi. . -o {prefix} . : . -s sparkline . -h|--help . -s sparkline . json-parse : mlr json-parse [options] JSON "[1,2,3]" -> [1,2,3]. : -f {a,b,c} json-parse ( ). -k (cell) ( ) . -h|--help . : mlr json-stringify [] - [1,2,3] -> "[1,2,3]". : -f {a,b,c} json-stringify ( ). --jvstack JSON . --no-jvstack JSON (). -h|--help . join : mlr join [] Miller. "join" . : -f { } (join). -j {a,b,c} . -l {a,b,c} -j . -r {a,b,c} () -j . --lk|--left-keep-field-names {a,b,c} . () . . : "" lk-- : . --lp {} . . --rp {} () . . --np . --ul . --ur () . --ignore-empty . - - np/--ul/--ur-- . -s|--sorted-input : . . -u . ( u-.) . --prepipe {} prepipe. mlr --help . prepipe . --prepipex {} prepipe ( ). mlr --help . -h|--help . Miller . <> : -i { } 'csv' 'json'. icsv-- ijson-- : 'icsv--' 'i csv-' . --irs { } --ifs { } --ips { } --repifs --implicit-csv-header --implicit-tsv-header --no-implicit-csv-header --no-implicit-tsv-header '... mlr --csv ... join -l foo' CSV '... mlr --csv ... join --ijson -l foo' . '... mlr --csv --implicit-csv-header' 'join' 'no-implicit-csv-header--' . <> . https://miller.readthedocs.io/en/latest/reference-verbs#join . label : mlr label [] {new1,new2,new3,...} n n . ( n .) inidx-- implicit-csv-header-- . : -h|--help . : mlr latin1-to-utf8 . Latin-1 UTF-8 . DSL latin1_to_utf8 . : -h|--help . least-frequent : mlr least-frequent [] . (anti-mode) . : -f {a,b,c} . . -n {count} . . -b . -o {name} . "count". -h|--help . : "mlr most-frequent". merge-fields : mlr merge-fields [] . : -a {sum,count,...} : . -f {a,b,c} . -o . -r {a,b,c} . -o . -c {a,b,c} : . . -i type=7 R type=1 . . -o {name} -f/-r. -k . -S (No-op) Miller 5. -F (No-op) Miller 5. -h|--help . -a: count null_count /JSON-null distinct_count mode antimode sum mean ( ) mad (MAD) var stddev meaneb ( ) skewness kurtosis min max minlen maxlen sum mean . . : "a_in_x=1,a_out_x=2,b_in_y=4,b_out_x=8". : mlr merge-fields -a sum,count -f a_in_x,a_out_x -o foo "b_in_y=4,b_out_x=8,foo_sum=3,foo_count=2" "a_in_x,a_out_x" . : mlr merge-fields -a sum,count -r in_,out_ -o bar "bar_sum=15,bar_count=4" . : mlr merge-fields -a sum,count -c in_,out_ "a_x_sum=3,a_x_count=2,b_y_sum=4,b_y_count=1,b_x_sum=8,b_x_count=1" "a_in_x" "a_out_x" "a_x" "b_in_y" "b_y" "b_out_x" "b_x" . most-frequent : mlr most-frequent [] . () . : -f {a,b,c} . . -n {count} . . -b . -o {name} . "count". -h|--help . : "mlr least-frequent". nest : mlr nest [] / . : --explode / . --explode --implode . --implode --explode. --explode --implode . --values . --values --pairs . --pairs - . --values --pairs . --across-records / . --across-records --across-fields . --across-fields / . --across-records --across-fields . -f {field name} . -r {field names} -f . . : -r '^[xy]$'. --nested-fs {string} . ";". --nested-ps {string} - . ":". --evar {string} --explode --values --across-records --nested-fs {string}. --ivar {string} --implode --values --across-records --nested-fs {string}. -h|--help . "mlr --usage-separator-options" . : mlr nest --explode --values --across-records -f x "x=a;b;c,y=d" : "x=a,y=d" "x=b,y=d" "x=c,y=d" --implode . mlr nest --explode --values --across-fields -f x "x=a;b;c,y=d" : "x_1=a,x_2=b,x_3=c,y=d" --implode . mlr nest --explode --pairs --across-records -f x "x=a:1;b:2;c:3,y=d" : "a=1,y=d" "b=2,y=d" "c=3,y=d" mlr nest --explode --pairs --across-fields -f x "x=a:1;b:2;c:3,y=d" : "a=1,b=2,c=3,y=d" : * --pairs --implode . * "--implode --values --across-records" (non-streaming) : . `tail -f` . ( `tail -f`) . `tail -f` `--records-per-batch 1` . * nested-fs IFS : . : mlr reshape. : mlr nothing [] . tee/print/ . : -h|--help . put : mlr put [] {DSL expression} (DSL) . : https://miller.readthedocs.io/en/latest/reference-verbs : -f {file name} DSL ( ). *.mlr . -e {expression} DSL . -f . : / -f -e . -s {name=value} @name . mlr put -s foo=97 '$column += @foo' mlr put 'begin {@foo = 97} $column += @foo' . (type-inferencing) . -s name1=value1 -s name2=value2. : -s sequence=$SEQUENCE. -x {expression} (false) (true) . false . -q . begin / end . -S Miller (No-op) filter/put record-reader . . -F Miller (No-op) filter/put record-reader . . -w . -W -w . -p (AST) stdout Miller . -d -p AST . -D -d . -E DSL (echo). -v -E -p. -X . -v/-d/-D . --explain DSL . --errors-json . -h|--help . -e -f . . : -w -W -p -d -D -E -v -X. : mlr --from example.csv put '$qr = $quantity * $rate' put: (If-statements): 'if ($flag == true) { $quantity *= 10}' 'if ($x > 0.0) { $y=log10($x); $z=sqrt($y) } else {$y = 0.0; $z = 0.0}' : '$new_field = $index**2; $qn = $quantity * $new_field' (Regex-replacement): '$name = sub($name, "http.*com"i, "")' (Regex-capture): 'if ($a =~ "([a-z]+)_([0-9]+)") { $b = "left_\1"; $c = "right_\2" }' (Built-in variables): '$filename = FILENAME' ( mlr put -q ): '@sum += $x; end {emit @sum}' '@sum[$shape] += $quantity; end {emit @sum, "shape"}' '@sum[$shape][$color] += $x; end {emit @sum, "shape", "color"}' ' @min = min(@min,$x); @max=max(@max,$x); end{emitf @min, @max} ' https://miller.readthedocs.io/reference-dsl . rank : mlr rank [] (1,2,2,4,...) . x=10 x=20 x=20 x=30 : x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 x=30,x_rank=4. : - (two-pass) : . . - --sorted . : -f {a,b,c} () . -g {d,e,f} () (group-by). --sorted () ( -g). O(1) . . -h|--help . : mlr rank -f x data/rank-example.csv : mlr rank -f x -g g data/rank-example.csv : mlr sort -f x then rank -f x --sorted data/rank-example.csv : mlr regularize [] . : -h|--help . remove-empty-columns : mlr remove-empty-columns [] . (Non-streaming). : -h|--help . rename : mlr rename [] {old1,new1,old2,new2,...} . : -r . "ab" "a.*b" "ab" "a.*b" "^ab$" "^a.*b$" . "\1" "\9" . 'i' . -g . -h|--help . : mlr rename old_name,new_name mlr rename old_name_1,new_name_1,old_name_2,new_name_2 mlr rename -r 'Date_[0-9]+,Date' "Date" mlr rename -r '"Date_[0-9]+",Date' mlr rename -r 'Date_([0-9]+).*,\1' 20151015 mlr rename -r '"name"i,Name' "name" "Name" "NAME" "Name" reorder : mlr reorder [] . : -e : . -f {a,b,c} . -r {a,b,c} . . : -r '^YYY,^XXX' YYY XXX . -b {x} -f {x} . {x} . -a {x} -f {x} . {x} . -h|--help . : mlr reorder -f a,b "d=4,b=2,a=1,c=3" "a=1,b=2,d=4,c=3" . mlr reorder -e -f a,b "d=4,b=2,a=1,c=3" "d=4,c=3,a=1,b=2" . mlr reorder -r '^YYY,^XXX' YYY XXX . repeat : mlr repeat [] . -n -f . : -n {repeat count} . -f {field name} -n . -h|--help . : echo x=0 | mlr repeat -n 4 then put '$x=urand()' : x=0.488189 x=0.484973 x=0.704983 x=0.147311 : echo a=1,b=2,c=3 | mlr repeat -f b : a=1,b=2,c=3 a=1,b=2,c=3 : echo a=1,b=2,c=3 | mlr repeat -f c : a=1,b=2,c=3 a=1,b=2,c=3 a=1,b=2,c=3 reshape : mlr reshape [] : -i {input field names} -o {key-field name,value-field name} -r {input field regex} -o {key-field name,value-field name} / / . : -r . : tail -f . `tail -f` `--records-per-batch 1` . : -s {key-field name,value-field name} / . : tail -f . : -i {input field names} . -o . -r {input field regex} . . -o . -r . -o {key-field name,value-field name} . -i -r . -s {key-field name,value-field name} . -h|--help . : "wide.txt": time X Y 2009-01-01 0.65473572 2.4520609 2009-01-02 -0.89248112 0.2154713 2009-01-03 0.98012375 1.3179287 mlr --pprint reshape -i X,Y -o item,value wide.txt time item value 2009-01-01 X 0.65473572 2009-01-01 Y 2.4520609 2009-01-02 X -0.89248112 2009-01-02 Y 0.2154713 2009-01-03 X 0.98012375 2009-01-03 Y 1.3179287 mlr --pprint reshape -r '[A-Z]' -o item,value wide.txt time item value 2009-01-01 X 0.65473572 2009-01-01 Y 2.4520609 2009-01-02 X -0.89248112 2009-01-02 Y 0.2154713 2009-01-03 X 0.98012375 2009-01-03 Y 1.3179287 "long.txt": time item value 2009-01-01 X 0.65473572 2009-01-01 Y 2.4520609 2009-01-02 X -0.89248112 2009-01-02 Y 0.2154713 2009-01-03 X 0.98012375 2009-01-03 Y 1.3179287 mlr --pprint reshape -s item,value long.txt time X Y 2009-01-01 0.65473572 2.4520609 2009-01-02 -0.89248112 0.2154713 2009-01-03 0.98012375 1.3179287 : mlr nest. : mlr sample [] ( ) . : mlr bootstrap mlr shuffle. : -g {a,b,c} : a,b,c. -k {k} : -g. -h|--help . sec2gmtdate : mlr sec2gmtdate { } (epoch) -- GMT . sec2gmtdate : mlr sec2gmtdate time1,time2 : mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)' : -h|--help . sec2gmt : mlr sec2gmt [] { } (epoch) GMT . sec2gmt : mlr sec2gmt time1,time2 : mlr put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)' : -1 . -2 . -3 . -4 . -5 . -6 . -7 . -8 . -9 . --millis (epoch) . --micros (epoch) . --nanos (epoch) . -h|--help . seqgen : mlr seqgen [] . . . : -f {name} . : "i". --start {value} ( ). : 1. --step {value} . : 1. ( start == stop ). --stop {value} ( ). : 100. -h|--help . start stop / step . start stop step . step . start == stop . shuffle : mlr shuffle [] . . : mlr bootstrap mlr sample. : -h|--help . skip-trivial-records : mlr skip-trivial-records [] . : -h|--help . sort : mlr sort {} . ( .) (stable) : . : -f {a,b,c} . -r {a,b,c} . -c {a,b,c} . -cr {a,b,c} . -n {a,b,c} null . -nf {a,b,c} -n. -nr {a,b,c} null . -t {a,b,c} (natural) . -b reorder -b. -tr|-rt {a,b,c} . -h|--help . : mlr sort -f a,b -nr x,y,z : mlr sort -f a -f b -nr x -nr y -nr z : mlr sort-within-records [] . : -f {names} . -r {regex} regex . : -r '^[xy]' x y . regex -r / ( JSON) -f regex . -n ( 2 12). -f/-r. -h|--help . sparkline : mlr sparkline [] sparkline -- -- . . . : -f {a,b,c} sparkline. -h|--help . sparsify : mlr sparsify [] ( ) . CSV TSV . : -s {filler string} . . -f {a,b,c} . . -h|--help . : a=1,b=,c=3 a=1,c=3 . split : mlr split [] {filename} : -n {n} N . -m {m} M (round-robin) . -g {a,b,c} . --prefix {p} . "split". --suffix {s} . "csv". --folder {f} . . -a . -v (downstream) . -e URL-escape . -j {J} . "_". -h|--help . -m -n -g . ( mlr -h ). : mlr --icsv --from myfile.csv split --ojson -n 1000 CSV JSON . : myfile.csv . . split_1.csv split_2.csv . mlr --csv --from myfile.csv split -n 10000 . split_1.csv split_2.csv . mlr --csv --from myfile.csv split -m 10 JSON. mlr --csv --from myfile.csv split -m 10 -o json split_1.csv split_2.csv test_1.dat test_2.dat . mlr --csv --from myfile.csv split -m 10 --prefix test --suffix dat /tmp/. mlr --csv --from myfile.csv split -m 10 --prefix /tmp/test --suffix dat --folder: mlr --csv --from myfile.csv split -m 10 --folder /tmp --prefix test --suffix dat shape triangle square split_triangle.csv split_square.csv . mlr --csv --from myfile.csv split -g shape color yellow green shape triangle square split_yellow_triangle.csv split_yellow_square.csv . mlr --csv --from myfile.csv split -g color,shape DSL "tee" . ssub : mlr ssub [] () regex DSL `ssub`. C \n \t \x1f . \\ . `gsub` `sub` . : -f {a,b,c} . -r {regex} . -a . -h|--help . : mlr stats1 [] . : -a {sum,count,...} : . median ( p50) p{n} n 0..100 p10 p25.2 p50 p98 p100. -f {a,b,c} . --fr {regex} ( regex). --fx {regex} ( regex). -g {d,e,f} (group-by). --gr {regex} ( regex). --gx {regex} ( regex). --grfx {regex} --gr {regex} --fx { regex}. -i type=7 R type=1 . . -s . tail -f pprint . `tail -f` `--records-per-batch 1` . -w {n} : n ( ) . -g . . -s . -S Miller 5. -F Miller 5. -h|--help . -a : median p50 p10 p25.2 p50 p98 p100 count null_count /JSON-null distinct_count mode antimode sum mean ( ) mad (MAD) var stddev meaneb ( ) skewness kurtosis min max minlen maxlen : mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape : mlr stats1 -a count,mode -f size : mlr stats1 -a count,mode -f size -g shape : mlr stats1 -a mean,min,max -f quantity -g name -w 7 name. : mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$' count mode a h k . : * p50 median . * min max p0 p100 . * sum mean . . * count mode . 1 1.0 count mode . * mode . stats2 : mlr stats2 [] . : -a {linreg-ols,corr,...} : . -f {a,b,c,d} . . -g {e,f,g} (group-by). -v linreg-pca. -s . tail -f pprint . `tail -f` `--records-per-batch 1` . --fit . . linreg-ols linreg-pca logireg . -S Miller 5. -F Miller 5. -h|--help . -a : linreg-ols (OLS) linreg-pca (PCA) r2 linreg-ols ( linreg-pca ) logireg corr cov covx -s --fit . : mlr stats2 -a linreg-pca -f x,y : mlr stats2 -a linreg-ols,r2 -f x,y -g size,shape : mlr stats2 -a corr -f x,y : mlr step [] / . : -a {delta,rsum,...} : . -f {a,b,c} . -g {d,e,f} (group-by). -F ( counter) . Miller 6 Miller 5 no-op . -d {x,y,z} EWMA. 1 ( ) 1 0 . : "mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". "-d 0.5" . -o {a,b,c} EWMA. -d . -o -d . -h|--help . -a : counter () delta () . delta delta_1 delta_{n} n . ewma from-first () ratio () . ratio ratio_1 ratio_{n} n . rprod () rsum () shift shift_lag. shift shift_1 shift_{n} n . shift_lag () () . shift_lag shift_lag_1 shift_lag_{n} n . shift_lead () () . shift_lead shift_lead_1 shift_lead_{n} n . slwin m n . slwin_7_2 . : mlr step -a rsum -f request_size mlr step -a delta -f request_size -g hostname mlr step -a ewma -d 0.1,0.9 -f x,y mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y -g group_name mlr step -a slwin_9_0,slwin_0_9 -f x mlr step -a shift_lag_12 -f sales shift shift_lag shift_lead delta ratio : shift_lag_{n} n shift_lead_{n} n . : shift_lag shift_lag_1 . EWMA https://miller.readthedocs.io/en/latest/reference-verbs.html#filter https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average . sub : mlr sub [] () (regex) DSL `sub`. C \n \t \x1f . \\ . `gsub` `ssub` . : -f {a,b,c} . -r {regex} . -a . -h|--help . summary : mlr summary [] . : field_type string int -- ( ) count +1 null_count JSON null distinct_count mode sum mean stddev var skewness minlen maxlen min p25 median p75 max iqr : p75 - p25 lof : p25 - 3.0 * iqr lif : p25 - 1.5 * iqr uif : p75 + 1.5 * iqr uof : p75 + 3.0 * iqr : field_type count mean min max null_count distinct_count : * min p25 median p75 max * -- 4.1 4.10 . * mode mode . * field_type "int-string" "empty-string" -- . : -a {mean,sum,etc.} . -x {mean,sum,etc.} . --all . --transpose . -h|--help . : mlr surv -d {duration-field} -s {status-field} - ( ). : -d {field} ( ). -s {field} (0= 1=). -h|--help . tac : mlr tac [] . : -h|--help . tail : mlr tail [] n . : -g {a,b,c} tail a,b,c. -n {n} . 10. '+' n- n : -n +3 . -h|--help . tee : mlr tee [] {filename} : -a . -p filename (pipe-to). -h|--help . ( mlr -h). : mlr --icsv --opprint put '...' then tee --ojson ./mytap.dat then stats1 ... CSV (pretty-print) tee-file JSON . template : mlr template [] . fill-with . . : -f {a,b,c} a,b,c. -t {filename} CSV . --fill-with {filler string} . . -h|--help . : * a,b,c. * c=3,a=1,f=6 . * a=1,b=,c=3 . top : mlr top [] : -f {a,b,c} . -g {d,e,f} . -n {count} 1. -a . . --max . . --min . -F ( Miller 6 ). -o {name} . "top_idx". -a . -h|--help . n / . -a . -a -f -g top-index . https://miller.readthedocs.io/en/latest/reference-verbs#top utf8-to-latin1 : mlr utf8-to-latin1 . UTF-8 Latin-1 . DSL utf8_to_latin1 . : -h|--help . : mlr unflatten [] flatten . : 'a.b.c' 4 'a' '{"b": { "c": 4 }}' . : -f {a,b,c} unflatten ( ). -s {string} mlr --flatsep. -h|--help . uniq : mlr uniq [] . -c count-distinct . uniq -f -g . -g -f . head : "mlr head -n 1 -g hash" hash . : -g {d,e,f} (Group-by) uniq. -f {d,e,f} -g. -x {a,b,c} uniq . -c . -n . -o {name} . "count". -a . -g. -c . -n . -c -n . -h|--help . unspace : mlr unspace [] / _ . PPRINT . : -f {x} . "_". -k . -v . -h|--help . unsparsify : mlr unsparsify [] . . . : --fill-with {filler string} . . -f {a,b,c} (streaming) . -h|--help . : 'a=1,b=2' 'a=,b=3,c=4' . FILTER/PUT (FUNCTIONS FOR FILTER/PUT) abs (class=math #args=1) . acos (class=math #args=1) . acosh (class=math #args=1) . antimode (class=stats #args=1) . / . "1" 1 . . : antimode([3,3,4,4,4]) 3 antimode([3,3,4,4]) 3 (class=higher-order-functions #args=2) (map) (array) / true true false . . . : : any([10,20,30], func(e) {return $index == e}) : any({"a": "foo", "b": "bar"}, func(k,v) {return $[k] == v}) append (class=collections #args=2) . apply (class=higher-order-functions #args=2) / . . - ( ) . : : apply([1,2,3,4,5], func(e) {return e ** 3}) [1, 8, 27, 64, 125] . : apply({"a":1, "b":3, "c":5}, func(k,v) {return {toupper(k): v ** 2}}) {"A": 1, "B":9, "C": 25}" . arrayify (class=collections #args=1) / "1" "2" ... . unflatten . asin (class=math #args=1) . asinh (class=math #args=1) . asserting_absent (class=typing #args=1) is_absent false . asserting_array (class=typing #args=1) is_array false . asserting_bool (class=typing #args=1) is_bool false . asserting_boolean (class=typing #args=1) is_boolean false . asserting_bytes (class=typing #args=1) is_bytes false . asserting_empty (class=typing #args=1) is_empty false . asserting_empty_map (class=typing #args=1) is_empty_map false . (class=typing #args=1) is_error false . asserting_float (class=typing #args=1) is_float false . asserting_int (class=typing #args=1) is_int false . asserting_map (class=typing #args=1) is_map false . asserting_nonempty_map (class=typing #args=1) is_nonempty_map false . asserting_not_array (class=typing #args=1) is_not_array false . asserting_not_empty (class=typing #args=1) is_not_empty false . asserting_not_map (class=typing #args=1) is_not_map false . asserting_not_null (class=typing #args=1) is_not_null false . asserting_null (class=typing #args=1) is_null false . asserting_numeric (class=typing #args=1) is_numeric false . asserting_present (class=typing #args=1) is_present false . asserting_string (class=typing #args=1) is_string false . atan (class=math #args=1) . (class=math #args=2) . atanh (class=math #args=1) . base64_decode (class=string #args=1) base64 (bytes) UTF-8 string() . base64 . : base64_decode("aGVsbG8=") 68656c6c6f string(base64_decode("aGVsbG8=")) "hello" base64_encode (class=string #args=1) (bytes) base64 . : base64_encode("hello") "aGVsbG8=" base64_encode(b"\xff") "/w==" bitcount (class=arithmetic #args=1) . boolean (class=conversion #args=1) int/float/bool/string (boolean). bytes (class=conversion #args=1) (bytes) . string() . : bytes("hi") 6869 string(bytes("hi")) "hi" capitalize (class=string #args=1) . cbrt (class=math #args=1) ( ). ceil (class=math #args=1) (Ceiling): . clean_whitespace (class=string #args=1) collapse_whitespace strip (type inference). collapse_whitespace (class=string #args=1) (whitespace) . concat (class=collections #args=variadic) . . : concat(1,2,3) [1,2,3] concat([1,2],3) [1,2,3] concat([1,2],[3]) [1,2,3] (class=string #args=2) true . `index(arg1, arg2) >= 0` . (map) . `any` `any([1,2,3], func(e) {return e == $foo})`. : contains("abcde", "e") true contains("abcde", "x") false contains(12345, 34) true contains("foret", "e") true contains([1,2,3], 2) (error) cos (class=math #args=1) . cosh (class=math #args=1) . count (class=stats #args=1) . / . : count([7,8,9]) 3 count({"a":7,"b":8,"c":9}) 3 depth (class=collections #args=1) / . 0 . dhms2fsec (class=time #args=1) dhms2fsec("5d18h53m20.250000s") = 500000.250000 dhms2sec (class=time #args=1) dhms2sec("5d18h53m20s") = 500000 distinct_count (class=stats #args=1) . / . "1" 1 . : distinct_count([7,8,9,7]) 3 distinct_count([1,"1"]) 1 distinct_count([1,1.0]) 2 erf (class=math #args=1) . erfc (class=math #args=1) . every (class=higher-order-functions #args=2) / true true false . ( ) ( ) . . : : every(["a", "b", "c"], func(e) {return $[e] >= 0}) : every({"a": "foo", "b": "bar"}, func(k,v) {return $[k] == v}) exec (class=system #args=variadic) '$output = exec( "command", ["arg1", "arg2"], {"env": ["ENV_VAR=ENV_VALUE", "ENV_VAR2=ENV_VALUE2"], "dir": "/tmp/run_command_here", "stdin_string": "this is input fed to program", "combined_output": true )' (stdout) carriage return . : exec("echo", ["I don't do", "$SHELL things"], {"env": "SHELL=sh"}) "I don't do $SHELL things" (class=math #args=1) e**x. expm1 (class=math #args=1) e**x - 1. flatten (class=collections #args=2,3) . JSON JSON CSV . ( $*) . flatten($*, ".") flatten("", ".", $*) . <> https://miller.readthedocs.io . : flatten({"a":[1,2],"b":3}, ".") {"a.1": 1, "a.2": 2, "b": 3}. flatten("a", ".", {"b": { "c": 4 }}) {"a.b.c" : 4}. flatten("", ".", {"a": { "b": 3 }}) {"a.b" : 3}. float (class=conversion #args=1) int/float/bool/string float. floor (class=math #args=1) : . fmtifnum (class=conversion #args=2) fmtnum . : fmtifnum(3.4, "%.6f") 3.400000" fmtifnum("abc", "%.6f") abc" $* = fmtifnum($*, "%.6f") fmtnum (class=conversion #args=2) int/float/bool printf (https://pkg.go.dev/fmt) '$s = fmtnum($n, "%08d")' '$t = fmtnum($n, "%.6e")'. Miller: "%_d" "%_f" . . : $y = fmtnum($x, "%.6f") $o = fmtnum($n, "%d") $o = fmtnum($n, "%12d") $y = fmtnum($x, "%.6_f") $o = fmtnum($n, "%_d") $o = fmtnum($n, "%12_d") fold (class=higher-order-functions #args=3) -- . : . : - ( ) . . : : fold([1,2,3,4,5], func(acc,e) {return acc + e**3}, 10000) 10225 . : fold({"a":1, "b":3, "c": 5}, func(acck,accv,ek,ev) {return {"sum": accv+ev**2}}, {"sum":10000}) 10035 . format (class=string #args=variadic) <<{}>> . <<{1}>> / . <<{}>> . . <<{0}>> . : format("{}:{}:{}", 1,2) "1:2:". format("{}:{}:{}", 1,2,3) "1:2:3". format("{}:{}:{}", 1,2,3,4) "1:2:3". format("{1}:{2}:{1}", "a","b") "a:b:a". format("{2}{}:{1}{}", 3,4) "43:34". fsec2dhms (class=time #args=1) fsec2dhms(500000.25) = "5d18h53m20.250000s" fsec2hms (class=time #args=1) fsec2hms(5000.25) = "01:23:20.250000" (class=collections #args=1) map get_values (class=collections #args=1) map -- gmt2localtime (class=time #args=1,2) GMT . $TZ . : gmt2localtime("1999-12-31T22:00:00Z") = "2000-01-01 00:00:00" TZ="Asia/Istanbul" gmt2localtime("1999-12-31T22:00:00Z", "Asia/Istanbul") = "2000-01-01 00:00:00" gmt2nsec (class=time #args=1) GMT (epoch) . : gmt2nsec("2001-02-03T04:05:06Z") = 981173106000000000 gmt2sec (class=time #args=1) GMT (epoch) . : gmt2sec("2001-02-03T04:05:06Z") = 981173106 gssub (class=string #args=3) gsub (regex). . : gssub("ab.d.fg", ".", "X") "abXdXfg" gsub (class=string #args=3) '$name = gsub($name, "old", "new")': . \1 \9 new (...) old gsub -- DSL . =~ regextract . <> https://miller.readthedocs.io . : gsub("ababab", "ab", "XY") "XYXYXY" gsub("abc.def", ".", "X") "XXXXXXX" gsub("abc.def", "\.", "X") "abcXdef" gsub("abcdefg", "[ce]", "X") "abXdXfg" gsub("prefix4529:suffix8567", "(....ix)([0-9]+)", "[\1 : \2]") "[prefix : 4529]:[suffix : 8567]" haskey (class=collections #args=2) / (True/false) map / 'haskey($*, "a")' 'haskey(mymap, mykey)' / / . map . : Miller -n..-1 1..n . hasvalue (class=collections #args=2) / (True/false) map/ / 'hasvalue($*, "a")' 'hasvalue(myarray, myvalue)' true . map . hex_decode (class=string #args=1) hex (bytes) UTF-8 string() . hex . : hex_decode("6869") 6869 string(hex_decode("6869")) "hi" hex_encode (class=string #args=1) (bytes) hex . : hex_encode("hi") "6869" hexfmt (class=conversion #args=1) int hex 255 "0xff". (class=time #args=1) hms2fsec("01:23:20.250000") = 5000.250000 hms2sec (class=time #args=1) hms2sec("01:23:20") = 5000 hostname (class=system #args=0) (hostname) . index (class=string #args=2) ( ) . -1 . . UTF-8 . : index("abcde", "e") 5 index("abcde", "x") -1 index(12345, 34) 3 index("foret", "t") 5 index([1,2,3], 2) (error) int (class=conversion #args=1,2) int/float/bool/string int. . . . : int("345") 345 ( 10/ ) int("0xff") 255 ( 16/ ) int("0377") 255 ( 8/ ) int("0b11010011") 211 0xd3 ( 2/ ) int("0377", 10) 377 int(345, 16) 345 int(string(345), 16) 837 invqnorm (class=math #args=1) . invqorm(urand()) . is_absent (class=typing #args=1) false true is_array (class=typing #args=1) true . is_bool (class=typing #args=1) (boolean) true . is_boolean. is_boolean (class=typing #args=1) (boolean) true . is_bool. is_bytes (class=typing #args=1) (bytes) true . is_empty (class=typing #args=1) true false . (class=typing #args=1) True map . is_error (class=typing #args=1) True (integer). is_float (class=typing #args=1) True float . is_int (class=typing #args=1) True int . is_map (class=typing #args=1) True map . is_nan (class=typing #args=1) True NaN () . NaN NaN != NaN 'x == NaN' 'is_nan(x)' . is_nonempty_map (class=typing #args=1) True map . is_not_array (class=typing #args=1) True array . is_not_empty (class=typing #args=1) True false. is_not_map (class=typing #args=1) True map . is_not_null (class=typing #args=1) False null ( JSON null) true. is_null (class=typing #args=1) True null ( JSON null) false. is_numeric (class=typing #args=1) True int float . is_present (class=typing #args=1) True false. (class=typing #args=1) ( ) (True) . joink (class=conversion #args=2) / . / . : joink({"a":3,"b":4,"c":5}, ",") = "a,b,c". joink([1,2,3], ",") = "1,2,3". joinkv (class=conversion #args=3) - / . / . : joinkv "=" "," . : joinkv([3,4,5], "=", ",") = "1=3,2=4,3=5" joinkv({"a":3,"b":4,"c":5}, ":", ";") = "a:3;b:4;c:5" joinv (class=conversion #args=2) / . / . : joinv([3,4,5], ",") = "3,4,5" joinv({"a":3,"b":4,"c":5}, ",") = "3,4,5" json_parse (class=collections #args=1) JSON . json_stringify (class=collections #args=1,2) JSON . . true . kurtosis (class=stats #args=1) (sample kurtosis) . / ( void) / . : kurtosis([4,5,9,10,11]) is -1.6703688 latin1_to_utf8 (class=string #args=1) Latin-1 UTF-8 . . : $y = latin1_to_utf8($x) $* = latin1_to_utf8($*) leafcount (class=collections #args=1) () / . / length . leftpad (class=string #args=3) ( ) ( ) (left-pad). . : leftpad("abcdefg", 10 , "*") gives "***abcdefg". leftpad("abcdefg", 10 , "XY") gives "XYabcdefg". leftpad("1234567", 10 , "0") gives "0001234567". length (class=collections #args=1) (top-level) / . . localtime2gmt (class=time #args=1,2) GMT . $TZ . : localtime2gmt("2000-01-01 00:00:00") = "1999-12-31T22:00:00Z" with TZ="Asia/Istanbul" localtime2gmt("2000-01-01 00:00:00", "Asia/Istanbul") = "1999-12-31T22:00:00Z" (class=time #args=1,2) (epoch) . $TZ . : localtime2nsec("2001-02-03 04:05:06") = 981165906000000000 TZ="Asia/Istanbul" localtime2nsec("2001-02-03 04:05:06", "Asia/Istanbul") = 981165906000000000" localtime2sec (class=time #args=1,2) (epoch) . $TZ . : localtime2sec("2001-02-03 04:05:06") = 981165906 TZ="Asia/Istanbul" localtime2sec("2001-02-03 04:05:06", "Asia/Istanbul") = 981165906" log (class=math #args=1) ( e). log10 (class=math #args=1) . log1p (class=math #args=1) log(1-x). logifit (class=math #args=3) m b : $yhat=logifit($x,$m,$b). lstrip (class=string #args=1) . madd (class=arithmetic #args=3) a + b mod m ( ) mapdiff (class=collections #args=variadic) . . . mapexcept (class=collections #args=variadic) ( ) . . 'mapexcept({1:2,3:4,5:6}, 1, 5, 7)' '{3:4}' 'mapexcept({1:2,3:4,5:6}, [1, 5, 7])' '{3:4}' . mapselect (class=collections #args=variadic) . . 'mapselect({1:2,3:4,5:6}, 1, 5, 7)' '{1:2,5:6}' 'mapselect({1:2,3:4,5:6}, [1, 5, 7])' '{1:2,5:6}' . mapsum (class=collections #args=variadic) . =< - . 'mapsum({1:2,3:4},{1:5})' '{1:5,3:4}' . max (class=math #args=variadic) n null . min max min/max / . (class=stats #args=1) . / void / . : maxlen(["ano", "alto"]) 4 md5 (class=hashing #args=1) MD5 . mean (class=stats #args=1) . / void / . : mean([4,5,7,10]) 6.5 meaneb (class=stats #args=1) (error bar) . / void / . : meaneb([4,5,7,10]) 1.3228756 median (class=stats #args=1,2) . / void / . percentiles . : median([3,4,5,6,9,10]) 6 median([3,4,5,6,9,10],{"interpolate_linearly":true}) 5.5 median(["abc", "def", "ghi", "ghi"]) "ghi" mexp (class=arithmetic #args=3) a ** b mod m ( ) min (class=math #args=variadic) n null ( ). min max min/max / . minlen (class=stats #args=1) . / void / . : minlen(["ano", "alto"]) 3 mmul (class=arithmetic #args=3) a * b mod m ( ) mode (class=stats #args=1) () . / . "1" 1 . . : mode([3,3,4,4,4]) 4 mode([3,3,4,4]) 3 msub (class=arithmetic #args=3) a - b mod m ( ) next (class=system #args=0) mlr: . true ( $* ) false ( $* false ). (class=time #args=1,2) (epoch) GMT . . n n . : nsec2gmt(1234567890000000000) = "2009-02-13T23:31:30Z" nsec2gmt(1234567890123456789) = "2009-02-13T23:31:30Z" nsec2gmt(1234567890123456789, 6) = "2009-02-13T23:31:30.123456Z" nsec2gmtdate (class=time #args=1) (epoch) GMT -- . . : sec2gmtdate(1440768801700000000) = "2015-08-28". nsec2localdate (class=time #args=1,2) (epoch) -- . . $TZ . : nsec2localdate(1440768801700000000) = "2015-08-28" TZ="Asia/Istanbul" nsec2localdate(1440768801700000000, "Asia/Istanbul") = "2015-08-28" nsec2localtime (class=time #args=1,2,3) (epoch) . $TZ . . n n : nsec2localtime(1234567890000000000) = "2009-02-14 01:31:30" TZ="Asia/Istanbul" nsec2localtime(1234567890123456789) = "2009-02-14 01:31:30" TZ="Asia/Istanbul" nsec2localtime(1234567890123456789, 6) = "2009-02-14 01:31:30.123456" TZ="Asia/Istanbul" nsec2localtime(1234567890123456789, 6, "Asia/Istanbul") = "2009-02-14 01:31:30.123456" null_count (class=stats #args=1) ( void) JSON null . / . "1" 1 . : null_count(["a", "", "c"]) 1 os (class=system #args=0) . percentile (class=stats #args=2,3) . / void / . percentiles . : percentile([3,4,5,6,9,10], 90) 10 percentile([3,4,5,6,9,10], 90, {"interpolate_linearly":true}) 9.5 percentile(["abc", "def", "ghi", "ghi"], 90) "ghi" percentiles (class=stats #args=2,3) . / void / . . : : percentiles([3,4,5,6,9,10], [25,75]) { "25": 4, "75": 9 } percentiles(["abc", "def", "ghi", "ghi"], [25,75]) { "25": "def", "75": "ghi" } "output_array_not_map" ( "oa") : percentiles([3,4,5,6,9,10], [25,75], {"output_array_not_map":true}) [4, 9] "interpolate_linearly" ( "il") -- : percentiles([3,4,5,6,9,10], [25,75], {"interpolate_linearly":true}) { "25": 4.25, "75": 8.25 } percentiles . -- sort_collection -- "array_is_sorted" (: "ais") : x = [6,5,9,10,4,3] percentiles(x, [25,75], {"ais":true}) { "25": 5, "75": 4 } x = sort_collection(x) percentiles(x, [25,75], {"ais":true}) { "25": 4, "75": 9 } . : : x = splitax("the quick brown fox jumped loquaciously over the lazy dogs", " ") x : ["the", "quick", "brown", "fox", "jumped", "loquaciously", "over", "the", "lazy", "dogs"] -- "dogs" p99 : percentiles(x, [50, 99], {"oa":true, "ais":true}) ["loquaciously", "dogs"] percentiles "the" p99 : percentiles(x, [50, 99], {"oa":true}) ["loquaciously", "the"] percentiles : x = sort(x) # x = sort_collection(x) x : ["brown", "dogs", "fox", "jumped", "lazy", "loquaciously", "over", "quick", "the", "the"] percentiles(x, [50, 99], {"oa":true, "ais":true}) ["loquaciously", "the"] percentiles(x, [50, 99], {"oa":true}) ["loquaciously", "the"] "loquaciously" p99 : x = sort(x, func(a,b) { return strlen(a) <=> strlen(b) } ) x : ["fox", "the", "the", "dogs", "lazy", "over", "brown", "quick", "jumped", "loquaciously"] percentiles(x, [50, 99], {"oa":true, "ais":true}) ["over", "loquaciously"] (class=arithmetic #args=2) . ** . qnorm (class=math #args=1) . reduce (class=higher-order-functions #args=2) map -- . . map map - ( map ) . - map . : : reduce([1,2,3,4,5], func(acc,e) {return acc + e**3}) 225 . map: reduce({"a":1, "b":3, "c": 5}, func(acck,accv,ek,ev) {return {"sum_of_squares": accv + ev**2}}) {"sum_of_squares": 35} . regextract (class=string #args=2) (regular expression) ( ). (capture groups) =~ . : regextract("index ab09 file", "[a-z][a-z][0-9][0-9]") "ab09" regextract("index a999 file", "[a-z][a-z][0-9][0-9]") (absent) . regextract_or_else (class=string #args=3) regextract ( ) ( ) . : regextract_or_else("index ab09 file", "[a-z][a-z][0-9][0-9]", "nonesuch") "ab09" regextract_or_else("index a999 file", "[a-z][a-z][0-9][0-9]", "nonesuch") "nonesuch" rightpad (class=string #args=3) ( ) ( ) (pad) . . : rightpad("abcdefg", 10 , "*") "abcdefg***" . rightpad("abcdefg", 10 , "XY") "abcdefgXY" . rightpad("1234567", 10 , "0") "1234567000" . round (class=math #args=1) . roundm (class=math #args=2) m: roundm($x,$m) round($x/$m)*$m . rstrip (class=string #args=1) (whitespace) . sec2dhms (class=time #args=1) sec2dhms(500000) = "5d18h53m20s" sec2gmt (class=time #args=1,2) (epoch) GMT . . n n . : sec2gmt(1234567890) = "2009-02-13T23:31:30Z" sec2gmt(1234567890.123456) = "2009-02-13T23:31:30Z" sec2gmt(1234567890.123456, 6) = "2009-02-13T23:31:30.123456Z" sec2gmtdate (class=time #args=1) epoch ( ) GMT -- . . : sec2gmtdate(1440768801.7) = "2015-08-28". (class=time #args=1) sec2hms(5000) = "01:23:20" sec2localdate (class=time #args=1,2) epoch ( ) -- . . $TZ . : sec2localdate(1440768801.7) = "2015-08-28" TZ="Asia/Istanbul" sec2localdate(1440768801.7, "Asia/Istanbul") = "2015-08-28" sec2localtime (class=time #args=1,2,3) epoch ( ) . $TZ . . n n . : sec2localtime(1234567890) = "2009-02-14 01:31:30" TZ="Asia/Istanbul" sec2localtime(1234567890.123456) = "2009-02-14 01:31:30" TZ="Asia/Istanbul" sec2localtime(1234567890.123456, 6) = "2009-02-14 01:31:30.123456" TZ="Asia/Istanbul" sec2localtime(1234567890.123456, 6, "Asia/Istanbul") = "2009-02-14 01:31:30.123456" select (class=higher-order-functions #args=2) (map) true . . . : : select([1,2,3,4,5], func(e) {return e >= 3}) [3, 4, 5] . : select({"a":1, "b":3, "c":5}, func(k,v) {return v >= 3}) {"b":3, "c": 5} . sgn (class=math #args=1) +1 0 -1 . sha1 (class=hashing #args=1) SHA1 (bytes). sha256 (class=hashing #args=1) SHA256 (bytes). sha512 (class=hashing #args=1) SHA512 (bytes). sin (class=math #args=1) . sinh (class=math #args=1) (). skewness (class=stats #args=1) . / void / . : skewness([4,5,9,10,11]) -0.2097285 sort (class=higher-order-functions #args=1-2) . (lexical) . "f" ("n" ) "c" "t" . "r" . "v" . a b a < b a == b a > b < 0 0 > 0 ak av bk bv a b < 0 0 > 0 . : : sort([3,"A",1,"B",22]) [1, 3, 20, "A", "B"] . . : sort(["E","a","c","B","d"]) ["B", "E", "a", "c", "d"] . . : sort(["E","a","c","B","d"], "c") ["a", "B", "c", "d", "E"] . : sort(["E","a","c","B","d"], "cr") ["E", "d", "c", "B", "a"] . : sort(["a1","a10","a100","a2","a20","a200"], "t") ["a1", "a2", "a10", "a20", "a100", "a200"] . : sort([5,2,3,1,4], func(a,b) {return b <=> a}) [5,4,3,2,1] . : sort({"c":2,"a":3,"b":1}, func(ak,av,bk,bv) {return bv <=> av}) {"a":3,"c":2,"b":1} . : sort({"c":2,"a":3,"b":1}) {"a":3,"b":1,"c":2} . : sort({"c":2,"a":3,"b":1}, "v") {"b":1,"c":2,"a":3} . : sort({"c":2,"a":3,"b":1}, "vnr") {"a":3,"c":2,"b":1} . (class=stats #args=1) percentiles . sparkline (class=stats #args=1) ( ) ASCII/Unicode. / . : sparkline([1,2,3,4,5,6,7,8]) "" sparkline([3,3,3]) "" splita (class=conversion #args=2) . . : splita("3,4,5", ",") = [3,4,5] splitax (class=conversion #args=2) . . : splitax("3,4,5", ",") = ["3","4","5"] splitkv (class=conversion #args=3) (map) . . : splitkv("a=3,b=4,c=5", "=", ",") = {"a":3,"b":4,"c":5} splitkvx (class=conversion #args=3) (map) ( ). . : splitkvx("a=3,b=4,c=5", "=", ",") = {"a":"3","b":"4","c":"5"} splitnv (class=conversion #args=2) (map) . . : splitnv("a,b,c", ",") = {"1":"a","2":"b","3":"c"} splitnvx (class=conversion #args=2) (map) ( ). . : splitnvx("3,4,5", ",") = {"1":"3","2":"4","3":"5"} sqrt (class=math #args=1) (). ssub (class=string #args=3) sub (regex) . . : ssub("abc.def", ".", "X") "abcXdef" stat (class=system #args=1) : "name" "size" int "mode" (octal) int "modtime" (epoch) int "isdir" (boolean). : stat("./mlr") { "name": "mlr", "size": 38391584, "mode": 0755, "modtime": 1715207874, "isdir": false } stat("./mlr")["size"] 38391584 (class=stats #args=1) . / ( void) / . : stddev([4,5,9,10,11]) is 3.1144823 strfntime (class=time #args=2) (epoch) . https://pkg.go.dev/github.com/lestrrat-go/strftime Miller "%1S" "%9S" . ("%S" .) C ( "man strftime" ) https://miller.readthedocs.io/en/latest/reference-dsl-time . strftime_local . : strfntime(1440768801123456789,"%Y-%m-%dT%H:%M:%SZ") = "2015-08-28T13:33:21Z" strfntime(1440768801123456789,"%Y-%m-%dT%H:%M:%3SZ") = "2015-08-28T13:33:21.123Z" strfntime(1440768801123456789,"%Y-%m-%dT%H:%M:%6SZ") = "2015-08-28T13:33:21.123456Z" strfntime_local (class=time #args=2,3) strfntime $TZ . : strfntime_local(1440768801123456789, "%Y-%m-%d %H:%M:%S %z") = "2015-08-28 16:33:21 +0300" with TZ="Asia/Istanbul" strfntime_local(1440768801123456789, "%Y-%m-%d %H:%M:%3S %z") = "2015-08-28 16:33:21.123 +0300" with TZ="Asia/Istanbul" strfntime_local(1440768801123456789, "%Y-%m-%d %H:%M:%3S %z", "Asia/Istanbul") = "2015-08-28 16:33:21.123 +0300" strfntime_local(1440768801123456789, "%Y-%m-%d %H:%M:%9S %z", "Asia/Istanbul") = "2015-08-28 16:33:21.123456789 +0300" strftime (class=time #args=2) (epoch) . https://pkg.go.dev/github.com/lestrrat-go/strftime Miller "%1S" "%9S" . ("%S" .) C ( "man strftime" ) https://miller.readthedocs.io/en/latest/reference-dsl-time . strftime_local . : strftime(1440768801.7,"%Y-%m-%dT%H:%M:%SZ") = "2015-08-28T13:33:21Z" strftime(1440768801.7,"%Y-%m-%dT%H:%M:%3SZ") = "2015-08-28T13:33:21.700Z" strftime_local (class=time #args=2,3) strftime $TZ . : strftime_local(1440768801.7, "%Y-%m-%d %H:%M:%S %z") = "2015-08-28 16:33:21 +0300" with TZ="Asia/Istanbul" strftime_local(1440768801.7, "%Y-%m-%d %H:%M:%3S %z") = "2015-08-28 16:33:21.700 +0300" with TZ="Asia/Istanbul" strftime_local(1440768801.7, "%Y-%m-%d %H:%M:%3S %z", "Asia/Istanbul") = "2015-08-28 16:33:21.700 +0300" string (class=conversion #args=1) int/float/bool/string/array/map (string). (UTF-8) -- ()bytes. strip (class=string #args=1) (whitespace) . strlen (class=string #args=1) UTF-8 . strmatch (class=string #args=2) / . (capture) regex `strmatch` . : strmatch("a", "abc") is false strmatch("abc", "a") is true strmatch("abc", "a[a-z]c") is true strmatch("abc", "(a).(c)") is true strmatch(12345, "34") is true strmatchx (class=string #args=2) . regex `=~` \1 \2 . `=~` \1 \9 . : strmatchx("a", "abc") returns: { "matched": false } strmatchx("abc", "a") returns: { "matched": true, "full_capture": "a", "full_start": 1, "full_end": 1 } strmatchx("[zy:3458]", "([a-z]+):([0-9]+)") returns: { "matched": true, "full_capture": "zy:3458", "full_start": 2, "full_end": 8, "captures": ["zy", "3458"], "starts": [2, 5], "ends": [3, 8] } (class=time #args=2) strpntime: (epoch) . strpntime_local . : strpntime("2015-08-28T13:33:21Z", "%Y-%m-%dT%H:%M:%SZ") = 1440768801000000000 strpntime("2015-08-28T13:33:21.345Z", "%Y-%m-%dT%H:%M:%SZ") = 1440768801345000000 strpntime("1970-01-01 00:00:00 -0400", "%Y-%m-%d %H:%M:%S %z") = 14400000000000 strpntime("1970-01-01 00:00:00 +0200", "%Y-%m-%d %H:%M:%S %z") = -7200000000000 strpntime_local (class=time #args=2,3) strpntime $TZ . : strpntime_local("2015-08-28T13:33:21Z", "%Y-%m-%dT%H:%M:%SZ") = 1440758001000000000 with TZ="Asia/Istanbul" strpntime_local("2015-08-28T13:33:21.345Z","%Y-%m-%dT%H:%M:%SZ") = 1440758001345000000 with TZ="Asia/Istanbul" strpntime_local("2015-08-28 13:33:21", "%Y-%m-%d %H:%M:%S") = 1440758001000000000 with TZ="Asia/Istanbul" strpntime_local("2015-08-28 13:33:21", "%Y-%m-%d %H:%M:%S", "Asia/Istanbul") = 1440758001000000000 strptime (class=time #args=2) strptime: (epoch) . strptime_local . : strptime("2015-08-28T13:33:21Z", "%Y-%m-%dT%H:%M:%SZ") = 1440768801.000000 strptime("2015-08-28T13:33:21.345Z", "%Y-%m-%dT%H:%M:%SZ") = 1440768801.345000 strptime("1970-01-01 00:00:00 -0400", "%Y-%m-%d %H:%M:%S %z") = 14400 strptime("1970-01-01 00:00:00 +0200", "%Y-%m-%d %H:%M:%S %z") = -7200 strptime_local (class=time #args=2,3) strptime $TZ . : strptime_local("2015-08-28T13:33:21Z", "%Y-%m-%dT%H:%M:%SZ") = 1440758001 with TZ="Asia/Istanbul" strptime_local("2015-08-28T13:33:21.345Z","%Y-%m-%dT%H:%M:%SZ") = 1440758001.345 with TZ="Asia/Istanbul" strptime_local("2015-08-28 13:33:21", "%Y-%m-%d %H:%M:%S") = 1440758001 with TZ="Asia/Istanbul" strptime_local("2015-08-28 13:33:21", "%Y-%m-%d %H:%M:%S", "Asia/Istanbul") = 1440758001 sub (class=string #args=3) '$name = sub($name, "old", "new")': ( ) . \1 \9 new (...) old sub -- DSL . =~ regextract . <> https://miller.readthedocs.io . : sub("ababab", "ab", "XY") gives "XYabab" sub("abc.def", ".", "X") gives "Xbc.def" sub("abc.def", "\.", "X") gives "abcXdef" sub("abcdefg", "[ce]", "X") gives "abXdefg" sub("prefix4529:suffix8567", "suffix([0-9]+)", "name\1") gives "prefix4529:name8567" substr (class=string #args=3) substr substr0 . substr1 . Miller (1-up) Miller . Miller 6 substr . substr0 (class=string #args=3) substr0(s,m,n) s m n ( ) . -len .. -1 0 .. len-1 . substr substr1 . substr1 (class=string #args=3) substr1(s,m,n) s m n ( ) . -len .. -1 1 .. len . substr substr0 . sum (class=stats #args=1) (map) . / . : sum([1,2,3,4,5]) is 15 sum2 (class=stats #args=1) (map) . / . : sum2([1,2,3,4,5]) is 55 (class=stats #args=1) (map) . / . : sum3([1,2,3,4,5]) 225 sum4 (class=stats #args=1) (map) . / . : sum4([1,2,3,4,5]) 979 sysntime (class=time #args=0) (epoch) . system (class=system #args=1) (stdout) (CR) . systime (class=time #args=0) (epoch) . systimeint (class=time #args=0) (epoch) . tan (class=math #args=1) . tanh (class=math #args=1) (). tolower (class=string #args=1) . toupper (class=string #args=1) . truncate (class=string #args=2) . typeof (class=typing #args=1) ( "str") . (debug). unflatten (class=collections #args=2) flatten . -JSON -JSON CSV . (map) . arrayify . <> https://miller.readthedocs.io . : unflatten({"a.b.c" : 4}, ".") {"a": "b": { "c": 4 }}. (class=string #args=2) . -- is_error() . : unformat("{}:{}:{}", "1:2:3") [1, 2, 3] . unformat("{}h{}m{}s", "3h47m22s") [3, 47, 22] . is_error(unformat("{}h{}m{}s", "3:47:22")) true . unformatx (class=string #args=2) unformat . : unformatx("{}:{}:{}", "1:2:3") ["1", "2", "3"] . unformatx("{}h{}m{}s", "3h47m22s") ["3", "47", "22"] . is_error(unformatx("{}h{}m{}s", "3:47:22")) true . upntime (class=time #args=0) Miller . uptime (class=time #args=0) Miller . urand (class=math #args=0) . : : '$n=floor(20+urand()*11)'. urand32 (class=math #args=0) 0 2**32-1 ( ). urandelement (class=math #args=1) . urandint (class=math #args=2) ( ). urandrange (class=math #args=2) [a, b). utf8_to_latin1 (class=string #args=1) UTF-8 Latin-1 . (map) . : $y = utf8_to_latin1($x) $* = utf8_to_latin1($*) variance (class=stats #args=1) (map) . / (void) / . : variance([4,5,9,10,11]) 9.7 version (class=system #args=0) Miller . (class=boolean #args=1) . != (class=boolean #args=2) /. . !=~ (class=boolean #args=2) ( ) ( ) '$name !=~ "^a.*b$"'. % (class=arithmetic #args=2) ( ). & (class=arithmetic #args=2) AND . && (class=boolean #args=2) AND . * (class=arithmetic #args=2) integer*integer float. ** (class=arithmetic #args=2) . pow . + (class=arithmetic #args=1,2) . - (class=arithmetic #args=1,2) . . (class=string #args=2) . (coerced) '"ax".98' . .* (class=arithmetic #args=2) integer-to-integer. .+ (class=arithmetic #args=2) integer-to-integer. .- (class=arithmetic #args=2) integer-to-integer. (class=arithmetic #args=2) . / (class=arithmetic #args=2) . / : 6/3 2 6/4 1.5 . // (class=arithmetic #args=2) . < (class=boolean #args=2) /. . << (class=arithmetic #args=2) . <= (class=boolean #args=2) /. . <=> (class=boolean #args=2) . a <=> b <0 0 >0 a < b a == b a > b . == (class=boolean #args=2) /. . =~ (class=boolean #args=2) ( ) regex ( ) '$name =~ "^a.*b$"'. \1 \9 (...) DSL . <> https://miller.readthedocs.io . : if: if ($url =~ "http.*com") { ... } if: $line = "index ab09 file" $line =~ "([a-z][a-z])([0-9][0-9])" $label = "[\1:\2]" $label "[ab:09]" > (class=boolean #args=2) /. . >= (class=boolean #args=2) /. . >> (class=arithmetic #args=2) . >>> (class=arithmetic #args=2) . (class=boolean #args=3) . ?? (class=boolean #args=2) (absent-coalesce). $a $a ?? 1 1 . ??? (class=boolean #args=2) / (absent/empty-coalesce). $a $a ??? 1 1 . ^ (class=arithmetic #args=2) XOR . ^^ (class=boolean #args=2) XOR . | (class=arithmetic #args=2) OR . || (class=boolean #args=2) OR . ~ (class=arithmetic #args=1) NOT . '$y=~$x' =~ (regex-match) '$y = ~$x' . PUT FILTER (KEYWORDS FOR PUT AND FILTER) all all: "emit1" "emit" "emitp" "unset" @* . begin begin: . . : 'begin { @count = 0 }' bool bool: (boolean) (scope) . : 'bool b = 1' . break break: for/while/do-while . call call: (subroutine) . : 'subr s(k,v) { print k . " is " . v} call s("a", $a)' continue: for/while/do-while . for . do do: "while" do-while . . dump dump: (out-of-stream) JSON stdout . > >> | stdout (redirect ). > >> (write) (append) ( awk) > . | (pipe) . ( > >>) ( |). . : mlr --from f.dat put -q '@v[NR]=$*; end { dump }' : mlr --from f.dat put -q '@v[NR]=$*; end { dump > "mytap.dat"}' : mlr --from f.dat put -q '@v[NR]=$*; end { dump >> "mytap.dat"}' : mlr --from f.dat put -q '@v[NR]=$*; end { dump | "jq .[]"}' edump edump: (out-of-stream) JSON stderr . : mlr --from f.dat put -q '@v[NR]=$*; end { edump }' elif elif: "else if" Miller . . else else: if/elif/elif . . emit1 emit1: . map (side-by-sides) (indexing) map (emit) . : mlr --from f.dat put 'emit1 $*' : mlr --from f.dat put 'emit1 mapsum({"id": NR}, $*)' https://miller.readthedocs.io://johnkerl.org/miller/doc . emit emit: . hashmap emit . > >> | . > >> ( awk) > . | . ( > >>) ( |). . --ocsv --ofs . mlr -h . : mlr --from f.dat put 'emit > "/tmp/data-".$a, $*' : mlr --from f.dat put 'emit > "/tmp/data-".$a, mapexcept($*, "a")' : mlr --from f.dat put '@sums[$a][$b]+=$x; emit @sums' : mlr --from f.dat put --ojson '@sums[$a][$b]+=$x; emit > "tap-".$a.$b.".dat", @sums' : mlr --from f.dat put '@sums[$a][$b]+=$x; emit @sums, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emit @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emit > "mytap.dat", @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emit >> "mytap.dat", @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emit | "gzip > mytap.dat.gz", @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emit > stderr, @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emit | "grep somepattern", @*, "index1", "index2"' https://miller.readthedocs.io://johnkerl.org/miller/doc . emitf: () . > >> | . > >> ( awk) > . | . ( > >>) ( |) . . --ocsv --ofs . mlr -h . : mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf @a' : mlr --from f.dat put --oxtab '@a=$i;@b+=$x;@c+=$y; emitf > "tap-".$i.".dat", @a' : mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf @a, @b, @c' : mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf > "mytap.dat", @a, @b, @c' : mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf >> "mytap.dat", @a, @b, @c' : mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf > stderr, @a, @b, @c' : mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf | "grep somepattern", @a, @b, @c' : mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf | "grep somepattern > mytap.dat", @a, @b, @c' https://miller.readthedocs.io://johnkerl.org/miller/doc . emitp emitp: . Hashmap emitp <<:>> . > >> | . > >> ( awk) > . | . ( > >>) ( |) . . --ocsv --ofs . mlr -h . : mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @sums' : mlr --from f.dat put --opprint '@sums[$a][$b]+=$x; emitp > "tap-".$a.$b.".dat", @sums' : mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @sums, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emitp > "mytap.dat", @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emitp >> "mytap.dat", @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emitp | "gzip > mytap.dat.gz", @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emitp > stderr, @*, "index1", "index2"' : mlr --from f.dat put '@sums[$a][$b]+=$x; emitp | "grep somepattern", @*, "index1", "index2"' https://miller.readthedocs.io://johnkerl.org/miller/doc . end end: . . : 'end { emit @count }' : 'end { eprint "Final count is " . @count }' eprint eprint: stderr . : mlr --from f.dat put -q 'eprint "The sum of x and y is ".($x+$y)' : mlr --from f.dat put -q 'for (k, v in $*) { eprint k . " => " . v }' : mlr --from f.dat put '(NR % 1000 == 0) { eprint "Checkpoint ".NR}' eprintn eprintn: stderr . : mlr --from f.dat put -q 'eprintn "The sum of x and y is ".($x+$y); eprint ""' false false: (boolean). filter filter: . : mlr --from f.dat put 'filter (NR == 2 || $x > 5.4)' put 'filter false' put -q . : : mlr --from f.dat put -q '@running_sum += $x * $y; emit @running_sum' float: . : 'float x = 0' . for for: for . . for : : 'for (k, v in $*) { ... }' for : : 'for (k, v in @counts) { ... }' : 'for ((k1, k2), v in @counts) { ... }' : 'for ((k1, k2, k3), v in @*) { ... }' for C: : 'for (var i = 0, var b = 1; i < 10; i += 1, b *= 2) { ... }' func func: . : 'func f(a,b) { return sqrt(a**2+b**2)} $d = f($x, $y)' funct funct: . : 'func g(num a, num b, funct f) :num { return f(a**2+b**2) }' if if: if/elif/elif . . in in: for . int int: . : 'int x = 0.0' . map map: (map) . : 'map b = 0' . map b = {} . map b = a a . num num: int/float . : 'num b = true' . print print: stdout . : mlr --from f.dat put -q 'print "The sum of x and y is ".($x+$y)' : mlr --from f.dat put -q 'for (k, v in $*) { print k . " => " . v }' : mlr --from f.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' printn printn: stdout . : mlr --from f.dat put -q 'printn "."; end { print "" }' return: . return ( if) absent-null (LHS) . stderr stderr: tee emit emitf emitp print dump . stdout stdout: tee emit emitf emitp print dump . str str: (string) . . subr subr: (subroutine) . : 'subr s(k,v) { print k . " is " . v} call s("a", $a)' tee tee: . ( pprint ). > >> ( awk) > . | (pipe) . ( > >>) ( |) . . --ocsv --ofs . mlr -h . emit tee tee $* . : mlr --from f.dat put 'tee > "/tmp/data-".$a, $*' : mlr --from f.dat put 'tee >> "/tmp/data-".$a.$b, $*' : mlr --from f.dat put 'tee > stderr, $*' : mlr --from f.dat put -q 'tee | "tr \[a-z\\] \[A-Z\\]", $*' : mlr --from f.dat put -q 'tee | "tr \[a-z\\] \[A-Z\\] > /tmp/data-".$a, $*' : mlr --from f.dat put -q 'tee | "gzip > /tmp/data-".$a.".gz", $*' : mlr --from f.dat put -q --ojson 'tee | "gzip > /tmp/data-".$a.".gz", $*' true true: . unset unset: () (out-of-stream) . : mlr --from f.dat put 'unset $x' : mlr --from f.dat put 'unset $*' : mlr --from f.dat put 'for (k, v in $*) { if (k =~ "a.*") { unset $[k] } }' : mlr --from f.dat put '...; unset @sums' : mlr --from f.dat put '...; unset @sums["green"]' : mlr --from f.dat put '...; unset @*' var var: . : 'var a=1', 'var xyz=""' while: while "do" do-while . . ENV ENV: '$home = ENV["HOME"]' FILENAME FILENAME: . FILENUM FILENUM: . FNR FNR: . . IFS IFS: . IPS IPS: . IRS IRS: ( ) LF CRLF . M_E M_E: e. M_PI M_PI: pi. NF NF: . NF : -- put/filter -- NF . NR NR: . . OFS OFS: . .SS "OPS" OPS: . ORS ORS: ( ) LF CRLF . (AUTHORS) Miller John Kerl . Miller Eric MSP Veith . (SEE ALSO) awk(1) sed(1) cut(1) join(1) sort(1) RFC 4180: MIME (CSV) Miller https://miller.readthedocs.io 2026-08-10 MILLER(1)