--------------------------------------------------------------------------------
datamash: basic data analysis
--------------------------------------------------------------------------------

datamash is a command line data (numerical and strings) analysis
tool.  It is designed for strict input validation and robust. awk
can do more than datamash but it is not built to check if the data
are valid.

https://github.com/agordon/datamash/tree/master/examples     [examples]
https://www.gnu.org/software/datamash/alternatives/          [one-liners, cf awk]

$ datamash OPTION op field op field  [stdin]

op refers to operations

OPTIONS:
-C ... skip comment lines ("^ *#" or "^ *;")
-t "X" ... by default IFS=OFS=\t. Use "X" instead.
--output-delimter=OFS
-W ... IFS=OFS=whitespace


-f print entire line (default is to print only grouped keys)
-g groupby fields
-h first line of input and output is header 
-i ... ignore case
-s ... sort the column
-R ... round ouput
-z ... lines end with NUL instead of \n

Primary Operations:
 groupby ... -g
 crosstab X, Y
 transpose
 reverse
 check [N lines] [N fields]

Statistical: 
mean,trimean,median,q1,q3,iqr,perc,mode,antimode,stdev,var,mad,skew,kurt,jarque,dpo,cov,pearson

Numeric:
sum,min,max,absmin,absmx,range

Textual/Numeric:
first,last,select randomly,unique,countunique,comma-separated output list (collapse)

Line Filtering:
 rmdup ... remove duplicate lines 

Per-line operations:
  base64,debase64,bin(histogram),strbin(hash),round/floor/ceil/trunc/frac
  dirname/basename,extension name, barename
  extract number from string 
  cut (copy input field to output field)


------------------------------------------------------------------------
I. Analyze by columns 
------------------------------------------------------------------------

Let us generate a set of 10 random numbers

$ jot -r 10 > a

$ cat a
62
18
31
4
56
95
86
4
78
81

The general format of the command line is
$ datamash "op" "col_number" "op" "col_number" <InputFile


In our case, there is only one column. So to get the sum we execute
$ datamash sum 1 <a      #sum the first column
515

Other simple operations are "min", "max", "absmin", "absmax", "range".
Statistical operations include: mean, trimmean, median, q1, q3, iqr,
perc, mode, antimode, pstdev,  sstdev,  pvar,  svar,  mad, madraw,
pskew, sskew, pkurt, skurt, dpo, jarque, scov, pcov, spearson, ppearson

$ datamash mean 1 q1 1 <a
51.5 21.25

Often, the first line is a header. Say, 
$ cat <(echo "value") a > b

$ cat b
value
62
18
31
4
56
95
86
4
78
81

In this case, to skip the header, 
$ datamash --header-in mean 1 <b

If, you want a pretty header 

$ datamash --header-out mean 1 q1 1 median 1 < a
mean(field-1)	q1(field-1)	median(field-1)
51.5	21.25	59

If, you want to carry the input header to the output then

$ datamash -H mean 1 q1 1 median 1 <b
mean(value)	q1(value)	median(value)       #neat!
51.5	21.25	59

The default input field separator (IFS) or delimiter is \t. You can
change that with $ datamash -t "X"

datamash works on columns. So, if you have a row then you must transfer
it to column (by piping the row to "xargs -n 1"). e.g.

$ echo {01..09} | xargs -n 1 
01
02
03
04
05
06
07
08
09

datamash allows you to transpose a matrix. 

$ datamash transpose < c

------------------------------------------------------------------------
II. Extracting and analyzing groups
------------------------------------------------------------------------

We will use a file used by the datamash developer to illustrate the 
power of datamash.

$ cat score_h.txt
Name    Major   Score
Shawn   Arts    65
Marques Arts    58
Fernando        Arts    78
Paul    Arts    63
Walter  Arts    75
We will use a file used by the datamash developer to illustrate the 
power of datamash.

$ cat score_h.txt
Name    Major   Score
Shawn   Arts    65
Marques Arts    58
Fernando        Arts    78
Paul    Arts    63
Walter  Arts    75
We will use a file used by the datamash developer to illustrate the 
power of datamash.

$ cat score_h.txt
Name    Major   Score
Shawn   Arts    65
Marques Arts    58
Fernando        Arts    78
Paul    Arts    63
Walter  Arts    75
...

The file has a header (first line) and there are three fields "Name",
"Major" and "Score". IFS=\t

$ datamash -g 2 min 3 max 3 < scores_h.txt