* Talk

I just gave a talk about this at [[https://www.socallinuxexpo.org/scale/17x][SCaLE 17x]]. Here are the [[https://www.youtube.com/watch?v=Qvb_uNkFGNQ&t=12830s][video of the talk]] and
the [[https://github.com/dkogan/talk-feedgnuplot-vnlog/blob/master/feedgnuplot-vnlog.org]["slides"]].

* Summary

Vnlog ("vanilla-log") is a toolkit for manipulating tabular ASCII data with
labelled fields using normal UNIX tools. If you regularly use =awk= and =sort=
and =uniq= and others, these tools will make you infinitely more powerful. The
vnlog tools /extend/, rather than replace the standard tooling, so minimal
effort is required to learn and use these tools.

Everything assumes a trivially simple log format:

- A whitespace-separated table of ASCII human-readable text
- A =#= character starts a comment that runs to the end of the line (like in
  many scripting languages)
- The first line that begins with a single =#= (not =##= or =#!=) is a /legend/,
  naming each column. This is required, and the field names that appear here are
  referenced by all the tools.
- Empty fields reported as =-=

This describes 99% of the format, with some extra details [[#format-details][below]]. Example:

#+BEGIN_EXAMPLE
#!/usr/bin/whatever
# a b c
1 2 3
## comment
4 5 6
#+END_EXAMPLE

Such data can be processed directly with almost any existing tool, and /this/
toolkit allows the user to manipulate this data in a nicer way by relying on
standard UNIX tools. The core philosophy is to avoid creating new knowledge as
much as possible. Consequently, the vnlog toolkit relies /heavily/ on existing
(and familiar!) tools and workflows. As such, the toolkit is small, light, and
has a /very/ friendly learning curve.

* Synopsis

I have [[https://raw.githubusercontent.com/dkogan/vnlog/master/dji-tsla.tar.gz][two sets of historical stock data]], from the start of 2018 until now
(2018/11):

#+BEGIN_SRC sh :results output :exports both
< dji.vnl head -n 4
#+END_SRC

#+RESULTS:
: # Date Open High Low Close AdjClose Volume
: 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840
: 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000
: 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000

And

#+BEGIN_SRC sh :results output :exports both
< tsla.vnl head -n 4
#+END_SRC

#+RESULTS:
: # Date Open High Low Close AdjClose Volume
: 2018-11-15 342.33 348.58 339.04 348.44 348.44 4486339
: 2018-11-14 342.70 347.11 337.15 344.00 344.00 5036300
: 2018-11-13 333.16 344.70 332.20 338.73 338.73 5448600

I can add whitespace to make the headers more legible by humans:

#+BEGIN_SRC sh :results output :exports both
< dji.vnl head -n 4 | vnl-align
#+END_SRC

#+RESULTS:
: #  Date      Open     High      Low     Close  AdjClose   Volume 
: 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840
: 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000
: 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000

I can pull out the closing prices:

#+BEGIN_SRC sh :results output :exports both
< dji.vnl vnl-filter -p Close | head -n4
#+END_SRC

#+RESULTS:
: # Close
: 25289.27
: 25080.50
: 25286.49

=vnl-filter= is primarily a wrapper around =awk= or =perl=, allowing the user to
reference columns by name. I can then plot the closing prices:

#+BEGIN_SRC sh :results file link :exports both
< dji.vnl vnl-filter -p Close |
  feedgnuplot --lines --unset grid
#+END_SRC

#+RESULTS:
[[file:guide-1.svg]]

Here I kept /only/ the closing price column, so the x-axis is just the row
index. The data was in reverse chronological order, so this plot is also in
reverse chronological order. Let's fix that:

#+BEGIN_SRC sh :results file link :exports both
< dji.vnl vnl-sort -k Date |
  vnl-filter -p Close |
  feedgnuplot --lines --unset grid
#+END_SRC

#+RESULTS:
[[file:guide-2.svg]]

The =vnl-sort= tool (and most of the other =vnl-xxx= tools) are wrappers around
the core tools already available on the system (such as =sort=, in this case).
With the primary difference being reading/writing vnlog, and referring to
columns by name. Since we just strictly reversed the order of the data, =sort=
was a bit overkill, and we could have equivalently done:

#+BEGIN_SRC sh :results file link :exports both
< dji.vnl vnl-tac |
  vnl-filter -p Close |
  feedgnuplot --lines --unset grid
#+END_SRC

We now have the data in the correct order, but it'd be nice to see the actual
dates on the x-axis. While we're at it, let's label the axes too:

#+BEGIN_SRC sh :results output :exports both
< dji.vnl vnl-filter -p Date,Close | head -n4
#+END_SRC

#+RESULTS:
: # Date Close
: 2018-11-15 25289.27
: 2018-11-14 25080.50
: 2018-11-13 25286.49

#+BEGIN_SRC sh :results file link :exports both
< dji.vnl vnl-sort -k Date |
  vnl-filter -p Date,Close |
  feedgnuplot --lines --unset grid --timefmt %Y-%m-%d --domain \
              --xlabel 'Date' --ylabel 'Price ($)'
#+END_SRC

#+RESULTS:
[[file:guide-3.svg]]

What was the highest value of the Dow-Jones index, and when did it happen?

#+BEGIN_SRC sh :results output :exports both
< dji.vnl vnl-sort -rgk Close |
  head -n2 |
  vnl-align
#+END_SRC

#+RESULTS:
: #  Date      Open     High      Low     Close  AdjClose   Volume 
: 2018-10-03 26833.47 26951.81 26789.08 26828.39 26828.39 280130000

Alrighty. Looks like the high was in October. Let's zoom in on that month:

#+BEGIN_SRC sh :results file link :exports both
< dji.vnl vnl-sort -k Date |
  vnl-filter 'Date ~ /2018-10/' -p Date,Close |
  feedgnuplot --lines --unset grid --timefmt %Y-%m-%d --domain \
              --xlabel 'Date' --ylabel 'Price ($)'
#+END_SRC

#+RESULTS:
[[file:guide-4.svg]]

OK. Is this thing volatile? What was the largest single-day gain, looking at
differences in consecutive closing prices?

#+BEGIN_SRC sh :results output :exports both
< dji.vnl vnl-sort -k Date |
  vnl-filter -p '.,d=diff(Close)' |
  head -n4 |
  vnl-align
#+END_SRC

#+RESULTS:
: #  Date      Open     High      Low     Close  AdjClose   Volume     d  
: 2018-01-02 24809.35 24864.19 24741.70 24824.01 24824.01 341130000 -     
: 2018-01-03 24850.45 24941.92 24825.55 24922.68 24922.68 456790000  98.67
: 2018-01-04 24964.86 25105.96 24963.27 25075.13 25075.13 403280000 152.45

#+BEGIN_SRC sh :results output :exports both
< dji.vnl vnl-sort -k Date |
  vnl-filter -p '.,gain_closeclose=diff(Close)',gain_openclose=Close-Open |
  vnl-sort -rgk gain_closeclose |
  head -n2 |
  vnl-filter -p Date,gain_ |
  vnl-align
#+END_SRC

#+RESULTS:
: #  Date    gain_closeclose gain_openclose
: 2018-03-26 669.4           376.86        

So the best single-gain day was 2018-03-26: the dow gained 669.4 points between
closing on the previous trading day and 2018-03-26. In that, 376.86 points were
gained during trading on 2018-03-26 itself.

What if I looked at maximum trading-day gains?

#+BEGIN_SRC sh :results output :exports both
< dji.vnl vnl-sort -k Date |
  vnl-filter -p '.,gain_closeclose=diff(Close)',gain_openclose=Close-Open |
  vnl-sort -rgk gain_openclose |
  head -n2 |
  vnl-filter -p Date,gain_ |
  vnl-align
#+END_SRC

#+RESULTS:
: #  Date    gain_closeclose gain_openclose
: 2018-02-06 567.02          827.6         

By that metric 2018-02-06 was much better. Since vnlog is a trivially simple
data format, we can use non-vnlog tools to compute statistics such as this. For
instance, we can do the same thing with ministat:

#+begin_src sh :results output :exports both
< dji.vnl vnl-filter -p gain_openclose=Close-Open |
  ministat -A
#+end_src

#+RESULTS:
: x <stdin>
:     N           Min           Max        Median           Avg        Stddev
: x 222      -1041.84         827.6         20.04    -9.3664414     230.67518

Or [[https://www.gnu.org/software/datamash/][datamash]]:

#+begin_src sh :results output :exports both
< dji.vnl vnl-filter -p gain_openclose=Close-Open |
  datamash -CW max 1
#+end_src

#+RESULTS:
: 827.6

Datamash 1.9 knows about vnlog specifically, so we can do a bit better:

#+begin_src sh :results output :exports both
< dji.vnl vnl-filter -p gain_openclose=Close-Open |
  datamash --vnlog max gain_openclose
#+end_src

#+RESULTS:
: # max(gain_openclose)
: 827.6


Let's join the Dow-jones index data and the TSLA data, and let's look at them
together:

#+BEGIN_SRC sh :results output :exports both
vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  head -n4 |
  vnl-align
#+END_SRC

#+RESULTS:
: #  Date    Open_dji High_dji  Low_dji Close_dji AdjClose_dji Volume_dji Open_tsla High_tsla Low_tsla Close_tsla AdjClose_tsla Volume_tsla
: 2018-11-15 25061.48 25354.56 24787.79 25289.27  25289.27     383292840  342.33    348.58    339.04   348.44     348.44        4486339    
: 2018-11-14 25388.08 25501.29 24935.82 25080.50  25080.50     384240000  342.70    347.11    337.15   344.00     344.00        5036300    
: 2018-11-13 25321.21 25511.03 25193.78 25286.49  25286.49     339690000  333.16    344.70    332.20   338.73     338.73        5448600    

#+BEGIN_SRC sh :results output :exports both
vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  vnl-filter -p '^Close' |
  head -n4 |
  vnl-align
#+END_SRC

#+RESULTS:
: # Close_dji Close_tsla
: 25289.27    348.44    
: 25080.50    344.00    
: 25286.49    338.73    

#+BEGIN_SRC sh :results file link :exports both
vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  vnl-filter -p '^Close' |
  feedgnuplot --domain --points --unset grid \
              --xlabel 'DJI price ($)' --ylabel 'TSLA price ($)'
#+END_SRC

#+RESULTS:
[[file:guide-5.svg]]

Huh. Apparently there's no obvious, strong correlation between TSLA and
Dow-Jones closing prices. And we saw that with just a few shell commands,
without dropping down into a dedicated analysis system.

* Build and installation
vnlog is a part of Debian/buster and Ubuntu/cosmic (18.10) and later. On those
boxes you can simply

#+BEGIN_EXAMPLE
$ sudo apt install vnlog libvnlog-dev libvnlog-perl python3-vnlog
#+END_EXAMPLE

to get the binary tools, the C API, the perl and python3 interfaces
respectively.

** Install on non-Debian boxes
Most of this is written in an interpreted language, so there's nothing to build
or install, and you can run the tools directly from the source tree:

#+BEGIN_EXAMPLE
$ git clone https://github.com/dkogan/vnlog.git
$ cd vnlog
$ ./vnl-filter .....
#+END_EXAMPLE

The python and perl libraries can be run from the tree by setting the
=PYTHONPATH= and =PERL5LIB= environment variables respectively. For the C
library, you should =make=, and then point your =CFLAGS= and =LDLIBS= and
=LD_LIBRARY_PATH= to the local tree.

If you do want to install to some arbitrary location to simplify the paths, do
this:

#+BEGIN_EXAMPLE
$ make
$ PREFIX=/usr/local make install
#+END_EXAMPLE

This will install /all/ the components into =/usr/local=.

* Description
Vnlog data is nicely readable by both humans and machines. Any time your
application invokes =printf()= for either diagnostics or logging, consider
writing out vnlog-formatted data. You retain human readability, but gain the
power all the =vnl-...= tools provide.

Vnlog tools are designed to be very simple and light. There's an ever-growing
list of other tools that do vaguely the same thing. Some of these:

- https://github.com/BurntSushi/xsv
- https://csvkit.readthedocs.io/
- https://github.com/johnkerl/miller
- https://github.com/jqnatividad/qsv
- https://github.com/greymd/teip
- https://github.com/eBay/tsv-utils-dlang
- https://www.gnu.org/software/datamash/
- https://stedolan.github.io/jq/
- https://github.com/benbernard/RecordStream
- https://github.com/dinedal/textql
- https://www.visidata.org/
- http://harelba.github.io/q/
- https://github.com/BatchLabs/charlatan
- https://github.com/dbohdan/sqawk

Many of these provide facilities to run various analyses, and others focus on
data types that aren't just a table (json for instance). Vnlog by contrast
doesn't analyze anything, and targets the most trivial possible data format.
This makes it very easy to run any analysis you like in any tool you like. The
main envisioned use case is one-liners, and the tools are geared for that
purpose. The above mentioned tools are much more powerful than vnlog, so they
could be a better fit for some use cases. I claim that

- 90% of the time you want to do simple things, and vnlog is a great fit for the
  task
- If you really do need to do something complex, you shouldn't be in the shell
  writing oneliners anymore, and a fully-fledged analysis system (numpy, etc) is
  more appropriate

In the spirit of doing as little as possible, the provided tools are wrappers
around tools you already have and are familiar with. The provided tools are:

- =vnl-filter= is a tool to select a subset of the rows/columns in a vnlog
  and/or to manipulate the contents. This is an =awk= wrapper where the fields
  can be referenced by name instead of index. 20-second tutorial:

#+BEGIN_SRC sh :results none :exports code
vnl-filter -p col1,col2,colx=col3+col4 'col5 > 10' --has col6
#+END_SRC

  will read the input, and produce a vnlog with 3 columns: =col1= and =col2=
  from the input, and a column =colx= that's the sum of =col3= and =col4= in the
  input. Only those rows for which /both/ =col5 > 10= is true /and/ that have a
  non-null value for =col6= will be output. A null entry is signified by a
  single =-= character.

#+BEGIN_SRC sh :results none :exports code
vnl-filter --eval '{s += x} END {print s}'
#+END_SRC

#+RESULTS:

  will evaluate the given awk program on the input, but the column names work as
  you would hope they do: if the input has a column named =x=, this would
  produce the sum of all values in this column.

- =vnl-sort=, =vnl-uniq=, =vnl-join=, =vnl-tail=, =vnl-ts=, =vnl-tac=,
  =vnl-paste= are wrappers around the corresponding commandline tools. These
  work exactly as you would expect also: the columns can be referenced by name,
  and the legend comment is handled properly. These are wrappers, so all the
  commandline options those tools have "just work" (except options that don't
  make sense in the context of vnlog). As an example, =vnl-tail -f= will follow
  a log: data will be read by =vnl-tail= as it is written into the log (just
  like =tail -f=, but handling the legend properly). And you already know how to
  use these tools without even reading the manpages! Note: I use the Linux
  kernel and the tools from GNU Coreutils exclusively, but this all has been
  successfully tested on FreeBSD and OSX also. Please let me know if something
  doesn't work.

- =vnl-align= aligns vnlog columns for easy interpretation by humans. The
  meaning is unaffected

- =Vnlog::Parser= is a simple perl library to read a vnlog

- =vnlog= is a simple python library to read a vnlog. Both python2 and python3
  are supported

- =libvnlog= is a C library to simplify reading and writing a vnlog. Clearly all
  you /really/ need for writing is =printf()=, but this is useful if we have
  lots of columns, many containing null values in any given row, and/or if we
  have parallel threads writing to a log. In my usage I have hundreds of columns
  of sparse data, so this is handy

- =vnl-make-matrix= converts a one-point-per-line vnlog to a matrix of data.
  I.e.

#+BEGIN_EXAMPLE
$ cat dat.vnl
# i j x
0 0 1
0 1 2
0 2 3
1 0 4
1 1 5
1 2 6
2 0 7
2 1 8
2 2 9
3 0 10
3 1 11
3 2 12

$ < dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp
Writing to '/tmp/x.matrix'

$ cat /tmp/x.matrix
1 2 3
4 5 6
7 8 9
10 11 12
#+END_EXAMPLE

All the tools have manpages that contain more detail. And more tools will
probably be added with time.

* Format details
The high-level description of the vnlog format from [[#Summary][above]] is sufficient to
read/write "normal" vnlog data, but there are a few corner cases that should be
mentioned. To reiterate, the format description from above describes vnlog as:

- A whitespace-separated table of ASCII human-readable text
- A =#= character starts a comment that runs to the end of the line (like in
  many scripting languages)
- The first line that begins with a single =#= (not =##= or =#!=) is a /legend/,
  naming each column. This is required, and the field names that appear here are
  referenced by all the tools.
- Empty fields reported as =-=

For a few years now I've been using these tools myself, and supporting others as
they were passing vnlog data around. In the process I've encountered some
slightly-weird data, and patched the tools to accept it. So today the included
vnlog tools are /very/ permissive, and accept any vnlog data that can possibly
be accepted. Other vnlog tools may not be quite as permissive, and may not be
able to interpret "weird" data. Points of note, describing the included vnlog
tools:

- Leading and trailing whitespace is ignored. Everywhere. So this data file will
  be read properly, with the =x= column containing 1 and 3:

  #+begin_example
 # x y
1 2
  3 4
  #+end_example

- Empty (or whitespace-only) lines anywhere are ignored, and treated as a
  comment
- An initial =#= comment without field names is treated as a comment, and we
  continue looking for the legend in the following lines. So this data file will
  be read properly:

  #+begin_example
## comment
#
# x y
1 2
3 4
  #+end_example

- Trailing comments are supported, like in most scripting languages. So this
  data file will be read properly:

  #+begin_example
# x y
1 2 # comment
3 4
  #+end_example

- Field names are /very/ permissive: anything that isn't whitespace is
  supported. So this data file will be read properly:

  #+begin_example
# x y  # 1+  -
1   2  3  4  5
11 12 13 14 15
  #+end_example

  We can pull out the =#= and =1+= and =-= columns:

  #+begin_src sh
vnl-filter -p '#,1+,-'
  #+end_src

  And we can even operate on them, if we use whitespace to indicate field
  boundaries:

  #+begin_src sh
vnl-filter -p 'x=1+ + 5'
  #+end_src

  Note that this implies that trailing comments in a legend line are /not/
  supported: the extra =#= characters will be used for field names. Field names
  containing =,= or === are currently not accepted by =vnl-filter=, but /are/
  accepted by the other tools (=vnl-sort= and such). I'll make =vnl-filter= able
  to work with those field names too, eventually, but as a user, the simplest
  thing to do is to not pass around data with such field names.

- Duplicated labels are supported whenever possible. So

  #+begin_example
# x y  z  z
1   2  3  4
11 12 13 14
   #+end_example

  will work just fine, unless we're operating on =z=. With this data, both of
  these commands work:

  #+begin_src sh
vnl-filter -p x
vnl-filter -p z
  #+end_src

  Picking =z= selects both of the =z= columns. But neither of these commands can
  work with the non-unique =z= column:

  #+begin_src sh
vnl-filter -p s=z+1
vnl-sort -k z
  #+end_src

* Workflows and recipes
** Storing disjoint data

A common use case is a complex application that produces several semi-related
subsets of data at once. Example: a moving vehicle is reporting both its own
position and the observed positions of other vehicles; at any given time any
number of other vehicles may be observed. Two equivalent workflows are possible:

- a single unified vnlog stream for /all/ the data
- several discrete vnlog streams for each data subset

Both are valid approaches

*** One unified vnlog stream
Here the application produces a /single/ vnlog that contains /all/ the columns,
from /all/ the data subsets. In any given row, many of the columns will be empty
(i.e. contain only =-= ). For instance, a row describing a vehicle own position
will not have data about any observations, and vice versa. It is inefficient to
store all the extra =-= but it makes many things much nicer, so it's often worth
it. =vnl-filter= can be used to pull out the different subsets. Sample
=joint.vnl=:

#+BEGIN_EXAMPLE
# time x_self x_observation
1      10     -
2      20     -
2      -      100
3      30     -
3      -      200
3      -      300
#+END_EXAMPLE

Here we have 3 instances in time. We have no observations at =time= 1, one
observation at =time= 2, and two observations at =time= 3. We can use
=vnl-filter= to pull out the data we want:

#+BEGIN_EXAMPLE
$ < joint.vnl vnl-filter -p time,self

# time x_self
1 10
2 20
2 -
3 30
3 -
3 -
#+END_EXAMPLE

If we only care about our own positions, the =+= modifier in picked columns in
=vnl-filter= is very useful here:

#+BEGIN_EXAMPLE
$ < joint.vnl vnl-filter -p time,+self

# time x_self
1 10
2 20
3 30


$ < joint.vnl vnl-filter -p time,+observation

# time x_observation
2 100
3 200
3 300
#+END_EXAMPLE

Note that the default is =--skipempty=, so if we're /only/ looking at =x_self=
for instance, then we don't even need to =+= modifier:

#+begin_example
$ < joint.vnl vnl-filter -p self

# x_self
10
20
30
#+end_example

Also, note that the =vnlog= C interface works very nicely to produce these
datafiles:

- You can define lots and lots of columns, but only fill some of them before
  calling =vnlog_emit_record()=. The rest will be set to =-=.
- You can create multiple contexts for each type of data, and you can populate
  them with data independently. And when calling =vnlog_emit_record_ctx()=,
  you'll get a record with data for just that context.

*** Several discrete vnlog streams

Conversely, the application can produce /separate/ vnlog streams for /each/
subset of data. Depending on what is desired, exactly, =vnl-join= can be used to
re-join them:

#+BEGIN_EXAMPLE
$ cat self.vnl

# time x_self
1 10
2 20
3 30


$ cat observations.vnl

# time x_observation
2 100
3 200
3 300


$ vnl-join -j time -a- self.vnl observations.vnl

# time x_self x_observation
1 10 -
2 20 100
3 30 200
3 30 300
#+END_EXAMPLE

** Data statistics

A common need is to compute basic statistics from your data. Many of the
alternative toolkits listed above provide built-in facilities to do this, but
vnlog does not: it's meant to be unixy, where each tool has very limited scope.
Thus you can either do this with =awk= like you would normally, or you can use
other standalone tools to perform the needed computations. For instance, I can
generate some data:

#+BEGIN_EXAMPLE
$ seq 2 100 | awk 'BEGIN {print "# x"} {print log($1)}' > /tmp/log.vnl
#+END_EXAMPLE

Then I can compute the mean with =awk=:

#+BEGIN_EXAMPLE
$ < /tmp/log.vnl vnl-filter --eval '{sum += x} END {print sum/NR}'
3.67414
#+END_EXAMPLE

Or I can compute the mean (and other stuff) with a separate standalone tool:

#+BEGIN_EXAMPLE
$ < /tmp/log.vnl ministat
x <stdin>
+----------------------------------------------------------------------------+
|                                                                         xx |
|                                                                  x xxxxxxx |
|                                                             xx xxxxxxxxxxxx|
|                                                x  x xxxxxxxxxxxxxxxxxxxxxxx|
|x       x    x    x  x  x  x x x xx xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|                                         |_______________A____M___________| |
+----------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  99      0.693147       4.60517       3.93183     3.6741353    0.85656382
#+END_EXAMPLE

=ministat= is not a part of the vnlog toolkit, but the vnlog format is generic
so it works just fine.

** Powershell-style filtering of common shell commands

Everything about vnlog is generic and simple, so it's easy to use it to process
data that wasn't originally meant to be used this way. For instance filtering
the output of =ls -l= to report only file names and sizes, skipping directories,
and sorting by file sizes:

#+BEGIN_EXAMPLE
$ ls -l

total 320
-rw-r--r-- 1 dima dima  5044 Aug 25 15:04 Changes
-rw-r--r-- 1 dima dima 12749 Aug 25 15:04 Makefile
-rw-r--r-- 1 dima dima 69789 Aug 25 15:04 README.org
-rw-r--r-- 1 dima dima 33781 Aug 25 15:04 README.template.org
-rw-r--r-- 1 dima dima  5359 Aug 25 15:04 b64_cencode.c
drwxr-xr-x 4 dima dima  4096 Aug 25 15:04 completions
drwxr-xr-x 3 dima dima  4096 Aug 25 15:04 lib
drwxr-xr-x 3 dima dima  4096 Aug 25 15:04 packaging
drwxr-xr-x 2 dima dima  4096 Aug 25 15:04 test
-rwxr-xr-x 1 dima dima  5008 Aug 25 15:04 vnl-align
-rwxr-xr-x 1 dima dima 56637 Aug 25 15:04 vnl-filter
-rwxr-xr-x 1 dima dima  5678 Aug 25 15:04 vnl-gen-header
-rwxr-xr-x 1 dima dima 29815 Aug 25 15:04 vnl-join
-rwxr-xr-x 1 dima dima  3631 Aug 25 15:04 vnl-make-matrix
-rwxr-xr-x 1 dima dima  8372 Aug 25 15:04 vnl-sort
-rwxr-xr-x 1 dima dima  5822 Aug 25 15:04 vnl-tail
-rwxr-xr-x 1 dima dima  4439 Aug 25 15:04 vnl-ts
-rw-r--r-- 1 dima dima   559 Aug 25 15:04 vnlog-base64.h
-rw-r--r-- 1 dima dima  8169 Aug 25 15:04 vnlog.c
-rw-r--r-- 1 dima dima 12677 Aug 25 15:04 vnlog.h


$ (echo '# permissions num_links user group size month day time name';
   ls -l | tail -n +2) |
  vnl-filter 'permissions !~ "^d"' -p name,size |
  vnl-sort -gk size |
  vnl-align

#       name         size
vnlog-base64.h        559
vnl-make-matrix      3631
vnl-ts               4439
vnl-align            5008
Changes              5044
b64_cencode.c        5359
vnl-gen-header       5678
vnl-tail             5822
vnlog.c              8169
vnl-sort             8372
vnlog.h             12677
Makefile            12749
vnl-join            29815
README.template.org 33781
vnl-filter          56637
README.org          69789
#+END_EXAMPLE

With a bit of shell manipulation, these tools can be applied to a whole lot of
different data streams that know nothing of vnlog.

* C interface
** Writing vnlog files
*** Basic usage
For most uses, vnlog files are simple enough to be generated with plain prints.
But then each print statement has to know which numeric column we're populating,
which becomes effortful with many columns. In my usage it's common to have a
large parallelized C program that's writing logs with hundreds of columns where
any one record would contain only a subset of the columns. In such a case, it's
helpful to have a library that can output the log files. This is available.
Basic usage looks like this:

In a shell:

#+BEGIN_SRC sh :results none :exports code
vnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' 'void* binary' > vnlog_fields_generated.h
#+END_SRC

#+RESULTS:

In a C program test.c:

#+BEGIN_SRC C
#include "vnlog_fields_generated.h"

int main()
{
    vnlog_emit_legend();

    vnlog_set_field_value__w(-10);
    vnlog_set_field_value__x(40);
    vnlog_set_field_value__y("asdf");
    vnlog_emit_record();

    vnlog_set_field_value__z(0.3);
    vnlog_set_field_value__x(50);
    vnlog_set_field_value__w(-20);
    vnlog_set_field_value__binary("\x01\x02\x03", 3);
    vnlog_emit_record();

    vnlog_set_field_value__w(-30);
    vnlog_set_field_value__x(10);
    vnlog_set_field_value__y("whoa");
    vnlog_set_field_value__z(0.5);
    vnlog_emit_record();

    return 0;
}
#+END_SRC

Then we build and run, and we get

#+BEGIN_EXAMPLE
$ cc -o test test.c -lvnlog

$ ./test

# w x y z binary
-10 40 asdf - -
-20 50 - 0.2999999999999999889 AQID
-30 10 whoa 0.5 -
#+END_EXAMPLE

The binary field in base64-encoded. This is a rarely-used feature, but sometimes
you really need to log binary data for later processing, and this makes it
possible.

So you

1. Generate the header to define your columns

2. Call =vnlog_emit_legend()=

3. Call =vnlog_set_field_value__...()= for each field you want to set in that
   row.

4. Call =vnlog_emit_record()= to write the row and to reset all fields for the
   next row. Any fields unset with a =vnlog_set_field_value__...()= call are
   written as null: =-=

This is enough for 99% of the use cases. Things get a bit more complex if we
have have threading or if we have multiple vnlog ouput streams in the same
program. For both of these we use vnlog /contexts/.

*** Contexts

To support independent writing into the same vnlog (possibly by multiple
threads; this is reentrant), each log-writer should create a context, and use it
when talking to vnlog. The context functions will make sure that the fields in
each context are independent and that the output records won't clobber each
other:

#+BEGIN_SRC C
void child_writer( // the parent context also writes to this vnlog. Pass NULL to
                   // use the global one
                   struct vnlog_context_t* ctx_parent )
{
    struct vnlog_context_t ctx;
    vnlog_init_child_ctx(&ctx, ctx_parent);

    while(records)
    {
        vnlog_set_field_value_ctx__xxx(&ctx, ...);
        vnlog_set_field_value_ctx__yyy(&ctx, ...);
        vnlog_set_field_value_ctx__zzz(&ctx, ...);
        vnlog_emit_record_ctx(&ctx);
    }

    vnlog_free_ctx(&ctx); // required only if we have any binary fields
}
#+END_SRC

If we want to have multiple independent vnlog writers to /different/ streams
(with different columns and legends), we do this instead:

=file1.c=:
#+BEGIN_SRC C
#include "vnlog_fields_generated1.h"

void f(void)
{
    // Write some data out to the default context and default output (STDOUT)
    vnlog_emit_legend();
    ...
    vnlog_set_field_value__xxx(...);
    vnlog_set_field_value__yyy(...);
    ...
    vnlog_emit_record();
}
#+END_SRC

=file2.c=:
#+BEGIN_SRC C
#include "vnlog_fields_generated2.h"

void g(void)
{
    // Make a new session context, send output to a different file, write
    // out legend, and send out the data
    struct vnlog_context_t ctx;
    vnlog_init_session_ctx(&ctx);
    FILE* fp = fopen(...);
    vnlog_set_output_FILE(&ctx, fp);
    vnlog_emit_legend_ctx(&ctx);
    ...
    vnlog_set_field_value__a(...);
    vnlog_set_field_value__b(...);
    ...
    vnlog_free_ctx(&ctx); // required only if we have any binary fields
    vnlog_emit_record();
}
#+END_SRC

Note that it's the user's responsibility to make sure the new sessions go to a
different =FILE= by invoking =vnlog_set_output_FILE()=. Furthermore, note that
the included =vnlog_fields_....h= file defines the fields we're writing to; and
if we have multiple different vnlog field definitions in the same program (as in
this example), then the different writers /must/ live in different source files.
The compiler will barf if you try to =#include= two different
=vnlog_fields_....h= files in the same source.

*** Remaining APIs

- =vnlog_printf(...)= and =vnlog_printf_ctx(ctx, ...)= write to a pipe like
=printf()= does. This exists primarily for comments.

- =vnlog_clear_fields_ctx(ctx, do_free_binary)= clears out the data in a context
and makes it ready to be used for the next record. It is rare for the user to
have to call this manually. The most common case is handled automatically
(clearing out a context after emitting a record). One area where this is useful
is when making a copy of a context:

#+BEGIN_SRC C
struct vnlog_context_t ctx1;
// .... do stuff with ctx1 ... add data to it ...

struct vnlog_context_t ctx2 = ctx1;
// ctx1 and ctx2 now both have the same data, and the same pointers to
// binary data. I need to get rid of the pointer references in ctx1

vnlog_clear_fields_ctx(&ctx1, false);
#+END_SRC

- =vnlog_free_ctx(ctx)= frees memory for an vnlog context. Do this before
throwing the context away. Currently this is only needed for context that have
binary fields, but this should be called for all contexts anyway, in case this
changes in a later revision

** Reading vnlog files
The basic usage goes like this:

#+begin_src c
#include <stdio.h>
#include <stdbool.h>
#include <vnlog/vnlog-parser.h>
bool parse_vnlog(const char* filename)
{
    FILE* fp = fopen(filename);
    if(fp == NULL)
        return false;

    vnlog_parser_t ctx;
    if(VNL_OK != vnlog_parser_init(&ctx, fp))
        return false;

    // String in the "time" column for the most-recently-parsed row
    const char*const* time_record = vnlog_parser_record_from_key(&ctx, "time");
    if(time_record == NULL)
    {
        vnlog_parser_free(&ctx);
        return false;
    }

    int i_record = 0;
    vnlog_parser_result_t result;
    while(VNL_OK == (result = vnlog_parser_read_record(&ctx, fp)))
    {
        for(int i=0; i<ctx.Ncolumns; i++)
            printf("Record %d: %s = %s\n",
                   i_record,
                   ctx.record[i].key, ctx.record[i].value);
        printf("Record %d: time = %s\n",
               i_record, *time_record);
        i_record++;
    }

    vnlog_parser_free(&ctx);
    return true;
}
#+end_src

The usage should be clear from this example. See =vnlog-parser.h= for details.

** Base64 interface
The C interface supports writing base64-encoded binary data using Chris Venter's
libb64. The base64-encoder used here was slightly modified: the output appears
all on one line, making is suitable to appear in a vnlog field. If we're writing
a vnlog with =printf()= directly without using the =vnlog.h= interface described
above, we allow this modified base64 encoder to be invoked by itself. Usage:

#+BEGIN_SRC C
void* binary_buffer     = ...;
int   binary_buffer_len = ...;

char base64_buffer[vnlog_base64_dstlen_to_encode(binary_buffer_len)];
vnlog_base64_encode( base64_buffer, sizeof(base64_buffer),
                     binary_buffer, binary_buffer_len );
#+END_SRC

Clearly the above example allocates the base64 buffer on the stack, so it's only
suitable for small-ish data chunks. But if you have lots and lots of data,
probably writing it as base64 into a vnlog isn't the best thing to do.
* Python interface
Reading vnlog data into a python program is simple. The =vnlog= Python module
provides three different ways to do that:

1. slurp the whole thing into a numpy array using the =slurp()= function. Basic
   usage:

   #+begin_src python
import vnlog
arr,list_keys,dict_key_index = \
    vnlog.slurp(filename_or_fileobject)
   #+end_src

   This parses out the legend, and then calls =numpy.loadtxt()=. Null data values
   (=-=) are not supported

2. Iterate through the records: =vnlog= class, used as an iterator. Basic usage:

   #+begin_src python
import vnlog
for d in vnlog.vnlog(filename_or_fileobject):
    print(d['time'],d['height'])
   #+end_src

   Null data values are represented as =None=

3. Parse incoming lines individually: =vnlog= class, using the =parse()= method.
   Basic usage:

   #+begin_src python
import vnlog
parser = vnlog.vnlog()
for l in file:
    parser.parse(l)
    d = parser.values_dict()
    if not d:
        continue
    print(d['time'],d['height'])
   #+end_src

Most of the time you'd use options 1 or 2 above. Option 3 is the most general,
but also the most verbose and slowest.

** Structured dtypes in slurp()

The =slurp()= example from above is simple: it doesn't specify a =dtype= (data
type), so =float= (64-bit IEEE-754 float) is used by default, for /all/ the data
in the array. You can specify another =dtype=, for instance:

   #+begin_src python
arr,list_keys,dict_key_index = \
    vnlog.slurp(filename_or_fileobject,
                dtype = int)
   #+end_src

You can also specify a [[https://numpy.org/doc/stable/user/basics.rec.html][/structured/ dtype]] to pull out different columns, with
different individual dtypes. Most notably, this allows interpreting some columns
as strings. Let's say you have a very common vnlog data, such as this "data.vnl":

#+begin_example
#  image   x y z temperature
image1.png 1 2 5 34
image2.png 3 4 1 35
#+end_example

Reading this with a simple =vnlog.slurp()= will fail: the filenames are not
parseable as numerical values. We can instead do this:

#+begin_src python
dtype = np.dtype([ ('image',       'U16'),
                   ('x y z',       int, (3,)),
                   ('temperature', float), ])
arr = vnlog.slurp("data.vnl", dtype=dtype)
#+end_src

This will read the image filename, the xyz points and the temperature into
different sub-arrays, with different types each. Accessing the result looks
like this:

#+begin_example
print(arr['image'])
---> array(['image1.png', 'image2.png'], dtype='<U16')

print(arr['x y z'])
---> array([[1, 2, 5],
            [3, 4, 1]])

print(arr['temperature'])
---> array([34., 35.])
#+end_example

Notes:

- The given structured dtype defines both how to organize the data, and which
  data to extract. So it can be used to read in only a subset of the available
  columns. Here I could have omitted the 'temperature' column, for instance

- Sub-arrays are allowed. In the example I could say either

  #+begin_src python
  dtype = np.dtype([ ('image',       'U16'),
                     ('x y z',       int, (3,)),
                     ('temperature', float), ])
  #+end_src

  or

  #+begin_src python
  dtype = np.dtype([ ('image',       'U16'),
                     ('x',           int),
                     ('y',           int),
                     ('z',           int),
                     ('temperature', float), ])
  #+end_src

  The latter would read =x=, =y=, =z= into separate, individual arrays. Sometime
  we want this, sometimes not.

- Nested structured dtypes are not allowed. Fields inside other fields are not
  supported, since it's not clear how to map that to a flat vnlog legend

- If a structured dtype is given, =slurp()= returns the array only, since the
  field names are already available in the dtype

* numpy interface
If we need to read data into numpy specifically, nicer tools are available than
the generic =vnlog= Python module. The built-in =numpy.loadtxt= =numpy.savetxt=
functions work well (with the caveat that =numpy.loadtxt()= should be followed
by =numpysane.atleast_dims(..., -2)= to make sure that a data array of shape
=(Nrows,Ncols)= is returned even if =Nrows==1=. For example to write to standard
output a vnlog with fields =a=, =b= and =c=:

#+BEGIN_SRC python
numpy.savetxt(sys.stdout, array, fmt="%g", header="a b c")
#+END_SRC

Note that numpy automatically adds the =#= to the header. To read a vnlog from a
file on disk, do something like

#+BEGIN_SRC python
array = numpysane.atleast_dims(numpy.loadtxt('data.vnl'), -2)
#+END_SRC

These functions know that =#= lines are comments, but don't interpret anything
as field headers. That's easy to do, so I'm not providing any helper libraries.
I might do that at some point, but in the meantime, patches are welcome.

* Compatibility

I use GNU/Linux-based systems exclusively, but everything has been tested
functional on FreeBSD and OSX in addition to Debian, Ubuntu and CentOS. I can
imagine there's something I missed when testing on non-Linux systems, so please
let me know if you find any issues.

* Caveats and bugs

These tools are meant to be simple, so some things are hard requirements. A big
one is that columns are whitespace-separated. There is /no/ mechanism for
escaping or quoting whitespace into a single field. I think supporting something
like that is more trouble than it's worth.

* Manpages
** vnl-filter
#+BEGIN_EXAMPLE
NAME
    vnl-filter - filters vnlogs to select particular rows, fields

SYNOPSIS
     $ cat run.vnl

     # time x   y   z   temperature
     3      1   2.3 4.8 30
     4      1.1 2.2 4.7 31
     6      1   2.0 4.0 35
     7      1   1.6 3.1 42


     $ <run.vnl vnl-filter -p x,y,z | vnl-align

     # x  y   z
     1   2.3 4.8
     1.1 2.2 4.7
     1   2.0 4.0
     1   1.6 3.1


     $ <run.vnl vnl-filter -p i=NR,time,'dist=sqrt(x*x + y*y + z*z)' | vnl-align

     # i time   dist
     1   3    5.41572
     2   4    5.30471
     3   6    4.58258
     4   7    3.62905


     $ <run.vnl vnl-filter 'temperature >= 35' | vnl-align

     # time x  y   z  temperature
     6      1 2.0 4.0 35
     7      1 1.6 3.1 42



     $ <run.vnl vnl-filter --eval '{s += temperature} END { print "mean temp: " s/NR}'

     mean temp: 34.5


     $ <run.vnl vnl-filter -p x,y | feedgnuplot --terminal 'dumb 80,30' --unset grid --domain --lines --exit

       2.3 +---------------------------------------------------------------------+
           |           +          +          ***************         +           |
           |                                                **************       |
           |                                                              *******|
       2.2 |-+                                                       ************|
           |                                                 ********            |
           |                                         ********                    |
       2.1 |-+                              *********                          +-|
           |                        ********                                     |
           |                ********                                             |
           |            ****                                                     |
         2 |-+         *                                                       +-|
           |           *                                                         |
           |           *                                                         |
           |           *                                                         |
       1.9 |-+         *                                                       +-|
           |           *                                                         |
           |           *                                                         |
           |           *                                                         |
       1.8 |-+         *                                                       +-|
           |           *                                                         |
           |           *                                                         |
       1.7 |-+         *                                                       +-|
           |           *                                                         |
           |           *                                                         |
           |           *          +           +           +          +           |
       1.6 +---------------------------------------------------------------------+
          0.98         1         1.02        1.04        1.06       1.08        1.1

DESCRIPTION
    This tool is largely a frontend for awk to operate on vnlog files. Vnlog
    is both an input and an output. This tool makes it very simple to select
    specific rows and columns for output and to manipulate the data in
    various ways.

    This is a UNIX-style tool, so the input/output of this tool is strictly
    STDIN/STDOUT. Furthermore, in its usual form this tool is a filter, so
    the format of the output is *exactly* the same as the format of the
    input. The exception to this is when using "--eval", in which the output
    is dependent on whatever expression we're evaluating.

    This tool is convenient to process both stored data or live data; in the
    latter case, it's very useful to pipe the streaming output to
    "feedgnuplot --stream" to get a realtime visualization of the incoming
    data.

    This tool reads enough of the input file to get a legend, at which point
    it constructs an awk program to do the main work, and execs to awk (it's
    possible to use perl as well, but this isn't as fast).

  Input/output data format
    The input/output data is vnlog: a plain-text table of values. Any lines
    beginning with "#" are treated as comments, and are passed through. The
    first line that begins with "#" but not "##" or "#!" is a *legend* line.
    After the "#", follow whitespace-separated field names. Each subsequent
    line is whitespace-separated values matching this legend. For instance,
    this is a valid vnlog file:

     #!/usr/bin/something
     ## more comments
     # x y z
     -0.016107 0.004362 0.005369
     -0.017449 0.006711 0.006711
     -0.018456 0.014093 0.006711
     -0.017449 0.018791 0.006376

    "vnl-filter" uses this format for both the input and the output. The
    comments are preserved, but the legend is updated to reflect the fields
    in the output file.

    A string "-" is used to indicate an undefined value, so this is also a
    valid vnlog file:

     # x y z
     1 2 3
     4 - 6
     - - 7

  Filtering
    To select specific *columns*, pass their names to the "-p" option (short
    for "--print" or "--pick", which are synonyms). In its simplest form, to
    grab only columns "x" and "y", do

     vnl-filter -p x,y

    See the detailed description of "-p" below for more detail.

    To select specific *rows*, we use *matches* expressions. Anything on the
    "vnl-filter" commandline and not attached to any "--xxx" option is such
    an expression. For instance

     vnl-filter 'size > 10'

    would select only those rows whose "size" column contains a value > 10.
    See the detailed description of matches expressions below for more
    detail.

  Context lines
    "vnl-filter" supports the context output options ("-A", "-B" and "-C")
    exactly like the "grep" tool. I.e to print out all rows whose "size"
    column contains a value > 10 *but also* include the 3 rows immediately
    before *and* after such matching rows, do this:

     vnl-filter -C3 'size > 10'

    "-B" reports the rows *before* matching ones and "-A" the rows *after*
    matching ones. "-C" reports both. Note that this applies *only* to
    *matches* expressions: records skipped because they fail "--has" or
    "--skipempty" are *not* included in contextual output.

  Backend choice
    By default, the parsing of arguments and the legend happens in perl,
    which then constructs a simple awk script, and invokes "mawk" to
    actually read the data and to process it. This is done because awk is
    lighter weight and runs faster, which is important because our data sets
    could be quite large. We default to "mawk" specifically, since this is a
    simpler implementation than "gawk", and runs much faster. If for
    whatever reason we want to do everything with perl, this can be
    requested with the "--perl" option.

  Special functions
    For convenience we support several special functions in any expression
    passed on to awk or perl (named expressions, matches expressions,
    "--eval" strings). These generally maintain some internal state, and
    vnl-filter makes sure that this state is consistent. Note that these are
    evaluated *after* "--skipcomments" and "--has". So any record skipped
    because of a "--has" expression, for instance, will *not* be considered
    in prev(), diff() and so on.

    *   rel(x) returns value of "x" relative to the first value of "x". For
        instance we might want to see the time or position relative to the
        start, not relative to some absolute beginning. Example:

         $ cat tst.vnl

         # time x
         100    200
         101    212
         102    209


         $ <tst.vnl vnl-filter -p 't=rel(time),x=rel(x)

         # t x
         0 0
         1 12
         2 9

    *   diff(x) returns the difference between the current value of "x" and
        the previous value of "x". The first row will always be "-".
        Example:

         $ <tst.vnl vnl-filter -p x,'d1=diff(x),d2=diff(diff(x))' | vnl-align

         # x d1 d2
           1  -  -
           8  7  7
          27 19 12
          64 37 18
         125 61 24

    *   sum(x) returns the cumulative sum of "x". As diff(x) can be thought
        of as a derivative, sum(x) can be thought of as an integral. So
        "diff(sum(x))" would return the same value as "x" (except for the
        first row; diff() always returns "-" for the first row).

        Example:

         $ <tst.vnl vnl-filter -p 'x,s=sum(x),ds=diff(sum(x))' | vnl-align

         # x  s   ds
           1   1   -
           8   9   8
          27  36  27
          64 100  64
         125 225 125

    *   prev(x) returns the previous value of "x". One could construct sum()
        and rel() using this, if they weren't already available.

    *   latestdefined(x) returns the most recent value of "x" that isn't
        "-". If "x" isn't "-", this simply returns "x".

ARGUMENTS
  Matches expressions
    Anything on the commandline not attached to any "--xxx" option is a
    *matches* expression. These are used to select particular records (rows)
    in a data file. For each row, we evaluate all the expressions. If *all*
    the expressions evaluate to true, that row is output. This expression is
    passed directly to the awk (or perl) backend.

    Example: to select all rows that have valid data in column "a" *or*
    column "b" *or* column "c" you can

     vnl-filter 'a != "-" || b != "-" || c != "-"'

    or

     vnl-filter --perl 'defined a || defined b || defined c'

    As with the named expressions given to "-p" (described above), these are
    passed directly to awk, so anything that can be done with awk is
    supported here.

  -p|--print|--pick expr
    These option provide the mechanism to select specific columns for
    output. For instance to pull out columns called "lat", "lon", and any
    column whose name contains the string "feature_", do

     vnl-filter -p lat,lon,'feature_.*'

    or, equivalently

     vnl-filter --print lat --print lon --print 'feature_.*'

    We look for exact column name matches first, and if none are found, we
    try a regex. If there was no column called exactly "feature_", then the
    above would be equivalent to

     vnl-filter -p lat,lon,feature_

    This mechanism is much more powerful than just selecting columns. First
    off, we can rename chosen fields:

     vnl-filter -p w=feature_width

    would pick the "feature_width" field, but the resulting column in the
    output would be named "w". When renaming a column in this way regexen
    are *not* supported, and exact field names must be given. But the string
    to the right of the "=" is passed on directly to awk (after replacing
    field names with column indices), so any awk expression can be used
    here. For instance to compute the length of a vector in separate columns
    "x", "y", and "z" you can do:

     vnl-filter -p 'l=sqrt(x*x + y*y + z*z)'

    A single column called "l" would be produced.

    We can also *exclude* columns by preceding their name with "!". This
    works like you expect. Rules:

    *   The pick/exclude directives are processed in order given to produce
        the output picked-column list

    *   If the first "-p" item is an exclusion, we implicitly pick *all* the
        columns prior to processing the "-p".

    *   The exclusion expressions match the *output* column names, not the
        *input* names.

    *   We match the exact column names first. If that fails, we match as a
        regex

    Example. To grab all the columns *except* the temperature(s) do this:

     vnl-filter -p !temperature

    To grab all the columns that describe *something* about a robot (columns
    whose names have the string "robot_" in them), but *not* its temperature
    (i.e. *not* "robot_temperature"), do this:

     vnl-filter -p robot_,!temperature

  --has a,b,c,...
    Used to select records (rows) that have a non-empty value in a
    particular field (column). A *null* value in a column is designated with
    a single "-". If we want to select only records that have a value in the
    "x" column, we pass "--has x". To select records that have data for
    *all* of a given set of columns, the "--has" option can be repeated, or
    these multiple columns can be given in a whitespace-less comma-separated
    list. For instance if we want only records that have data in *both*
    columns "x" *and* "y" we can pass in "--has x,y" or "--has x --has y".
    If we want to combine multiple columns in an *or* (select rows that have
    data in *any* of a given set of columns), use a matches expression, as
    documented below.

    If we want to select a column *and* pick only rows that have a value in
    this column, a shorthand syntax exists:

     vnl-filter --has col -p col

    is equivalent to

     vnl-filter -p +col

    Note that just like the column specifications in "-p" the columns given
    to "--has" must match exactly *or* as a regex. In either case, a unique
    matching column must be found.

  -l|list-columns
    Instead of doing any processing, parse the input to get the available
    columns, print those out, and exit

  -A N|--after-context N
    Output N lines following each *matches* expression, even those lines
    that do not themselves match. This works just like the "grep" options of
    the same name. See "Context lines"

  -B N|--before-context N
    Output N lines preceding each *matches* expression, even those lines
    that do not themselves match. This works just like the "grep" options of
    the same name. See "Context lines"

  -C N|--context N
    Output N lines preceding and following each *matches* expression, even
    those lines that do not themselves match. This works just like the
    "grep" options of the same name. See "Context lines"

  --eval expr
    Instead of printing out all matching records and picked columns, just
    run the given chunk of awk (or perl). In this mode of operation,
    "vnl-filter" acts just like a glorified awk, that allows fields to be
    accessed by name instead of by number, as it would be in raw awk.

    Since the expression may print *anything* or nothing at all, the output
    in this mode is not necessarily itself a valid vnlog stream. And no
    column-selecting arguments should be given, since they make no sense in
    this mode.

    In awk the expr is a full set of pattern/action statements. So to print
    the sum of columns "a" and "b" in each row, and at the end, print the
    sum of all values in the "a" column

     vnl-filter --eval '{print a+b; suma += a} END {print suma}'

    In perl the arbitrary expression fits in like this:

     sub evalexpr
     {
       eval expression;     # evaluate the arbitrary expression
     }
     while(<>) # read each line
     {
       chomp;
       next unless matches; # skip non-matching lines
       evalexpr();
     }

  --function|--sub
    Evaluates the given expression as a function that can be used in other
    expressions. This is most useful when you want to print something that
    can't trivially be written as a simple expression. For instance:

     $ cat tst.vnl
     # s
     1-2
     3-4
     5-6

     $ < tst.vnl
       vnl-filter --function 'before(x) { sub("-.*","",x); return x }' \
                  --function 'after(x)  { sub(".*-","",x); return x }' \
                  -p 'b=before(s),a=after(s)'
     # b a
     1 2
     3 4
     5 6

    See the CAVEATS section below if you're doing something
    sufficiently-complicated where you need this.

  --function-abs|--sub-abs
    Convenience option to add an absolute-value abs() function. This is only
    useful for awk programs (the default, no "--perl" given) since perl
    already provides abs() by default.

  --begin|--BEGIN
    Evaluates the given expression in the BEGIN {} block of the generated
    awk (or perl) program.

  --end|--END
    Evaluates the given expression in the END {} block of the generated awk
    (or perl) program.

  --[no]skipempty
    Do [not] skip records where all fields are blank. By default we *do*
    skip all empty records; to include them, pass "--noskipempty"

  --skipcomments
    Don't output non-legend comments

  --perl
    By default all procesing is performed by "mawk", but if for whatever
    reason we want perl instead, pass "--perl". Both modes work, but "mawk"
    is noticeably faster. "--perl" could be useful because it is more
    powerful, which could be important since a number of things pass
    commandline strings directly to the underlying language (named
    expressions, matches expressions, "--eval" strings). Note that while
    variables in perl use sigils, column references should *not* use sigils.
    To print the sum of all values in column "a" you'd do this in awk

     vnl-filter --eval '{suma += a} END {print suma}'

    and this in perl

     vnl-filter --perl --eval '{$suma += a} END {say $suma}'

    The perl strings are evaluated without "use strict" or "use warnings" so
    I didn't have to declare $suma in the example.

    With "--perl", empty strings ("-" in the vnlog file) are converted to
    "undef".

  --dumpexprs
    Used for debugging. This spits out all the final awk (or perl) program
    we run for the given commandline options and given input. This is the
    final program, with the column references resolved to numeric indices,
    so one can figure out what went wrong.

  --unbuffered
    Flushes each line after each print. This makes sure each line is output
    as soon as it is available, which is crucial for realtime output and
    streaming plots.

  --stream
    Synonym for "--unbuffered"

CAVEATS
    This tool is very lax in its input validation (on purpose). As a result,
    columns with names like %CPU and "TIME+" do work (i.e. you can more or
    less feed in output from "top -b"). The downside is that shooting
    yourself in the foot is possible. This tradeoff is currently tuned to be
    very permissive, which works well for my use cases. I'd be interested in
    hearing other people's experiences. Potential pitfalls/unexpected
    behaviors:

    *   All column names are replaced in all eval strings without regard to
        context. The earlier example that reports the sum of values in a
        column: vnl-filter --eval '{suma += a} END {print suma}' will work
        fine if we *do* have a column named "a" and do *not* have a column
        named "suma". But will not do the right thing if any of those are
        violated. For instance, if a column "a" doesn't exist, then "awk"
        would see "suma += a" instead of something like "suma += $5". "a"
        would be an uninitialized variable, which evaluates to 0, so the
        full "vnl-filter" command would not fail, but would print 0 instead.
        It's the user's responsibility to make sure we're talking about the
        right columns. The focus here was one-liners so hopefully nobody has
        so many columns, they can't keep track of all of them in their head.
        I don't see any way to resolve this without seriously impacting the
        scope of the tool, so I'm leaving this alone.

    *   It is natural to use vnlog as a database. You can run queries with
        something like

         vnl-filter 'key == 5'

        This works. But unlike a real database this is clearly a linear
        lookup. With large data files, this would be significantly slower
        than the logarithmic searches provided by a real database. The
        meaning of "large" and "significant" varies, and you should test it.
        In my experience vnlog "databases" scale surprisingly well. But at
        some point, importing your data to something like sqlite is well
        worth it.

    *   When substituting column names I match *either* a word-nonword
        transition ("\b") *or* a whitespace-nonword transition. The word
        boundaries is what would be used 99% of the time. But the keys may
        have special characters in them, which don't work with "\b". This
        means that whitespace becomes important: "1+%CPU" will not be parsed
        as expected, which is correct since "+%CPU" is also a valid field
        name. But "1+ %CPU" will be parsed correctly, so if you have weird
        field names, put the whitespace into your expressions. It'll make
        them more readable anyway.

    *   Strings passed to "-p" are split on "," *except* if the "," is
        inside balanced "()". This makes it possible to say things like
        vnl-filter --function 'f(a,b) { ... }' -p 'c=f(a,b)'. This is
        probably the right behavior, although some questionable looking
        field names become potentially impossible: "f(a" and "b)" *could*
        otherwise be legal field names, but you're probably asking for
        trouble if you do that.

    *   Currently there're two modes: a pick/print mode and an "--eval"
        mode. Then there's also "--function", which adds bits of "--eval" to
        the pick/print mode, but it feels maybe insufficient. I don't yet
        have strong feelings about what this should become. Comments welcome


#+END_EXAMPLE

** vnl-align
#+BEGIN_EXAMPLE
NAME
    vnl-align - aligns vnlog columns for easy interpretation by humans

SYNOPSIS
     $ cat tst.vnl

     # w x y z
     -10 40 asdf -
     -20 50 - 0.300000
     -30 10 whoa 0.500000


     $ vnl-align tst.vnl

     # w  x   y      z
     -10 40 asdf -
     -20 50 -    0.300000
     -30 10 whoa 0.500000

DESCRIPTION
    The basic usage is

     vnl-align logfile

    The arguments are assumed to be the vnlog files. If no arguments are
    given, the input comes from STDIN.

    This is very similar to "column -t", but handles "#" lines properly:

    1. The first "#" line is the legend. For the purposes of alignment, the
    leading "#" character and the first column label are treated as one
    column

    2. All other "#" lines are output verbatim.


#+END_EXAMPLE

** vnl-sort
#+BEGIN_EXAMPLE
NAME
    vnl-sort - sorts an vnlog file, preserving the legend

SYNOPSIS
     $ cat a.vnl
     # a b
     AA 11
     bb 12
     CC 13
     dd 14
     dd 123

     Sort lexically by a:
     $ <a.vnl vnl-sort -k a
     # a b
     AA 11
     CC 13
     bb 12
     dd 123
     dd 14

     Sort lexically by a, ignoring case:
     $ <a.vnl vnl-sort -k a --ignore-case
     # a b
     AA 11
     bb 12
     CC 13
     dd 123
     dd 14

     Sort lexically by a, then numerically by b:
     $ <a.vnl vnl-sort -k a -k b.n
     # a b
     AA 11
     CC 13
     bb 12
     dd 14
     dd 123

     Sort lexically by a, then numerically by b in reverse:
     $ <a.vnl vnl-sort -k a -k b.nr
     # a b
     AA 11
     CC 13
     bb 12
     dd 123
     dd 14


     Sort by month and then day:
     $ cat dat.vnl
     # month day
     March 5
     Jan 2
     Feb 1
     March 30
     Jan 21

     $ <dat.vnl vnl-sort -k month.M -k day.n
     # month day
     Jan 2
     Jan 21
     Feb 1
     March 5
     March 30

DESCRIPTION
      Usage: vnl-sort [options] logfile logfile logfile ... < logfile

    This tool sorts given vnlog files in various ways. "vnl-sort" is a
    wrapper around the GNU coreutils "sort" tool. Since this is a wrapper,
    most commandline options and behaviors of the "sort" tool are present;
    consult the sort(1) manpage for detail. The differences from GNU
    coreutils "sort" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   The columns are referenced by name, not index. So instead of saying

          sort -k1

        to sort by the first column, you say

          sort -k time

        to sort by column "time".

    *   The fancy "KEYDEF" spec from "sort" is only partially supported. I
        only allow us to sort by full *fields*, so the start/stop positions
        don't make sense. I *do* support the "OPTS" to change the type of
        sorting in a given particular column. For instance, to sort by month
        and then by day, do this (see example above):

          vnl-sort -k month.M -k day.n

    *   "--files0-from" is not supported due to lack of time. If somebody
        really needs it, talk to me.

    *   "--output" is not supported due to an uninteresting technical
        limitation. The output always goes to standard out.

    *   "--field-separator" is not supported because vnlog assumes
        whitespace-separated fields

    *   "--zero-terminated" is not supported because vnlog assumes
        newline-separated records

    *   By default we call the "sort" tool to do the actual work. If the
        underlying tool has a different name or lives in an odd path, this
        can be specified by passing "--vnl-tool TOOL"

    Past that, everything "sort" does is supported, so see that man page for
    detailed documentation. Note that all non-legend comments are stripped
    out, since it's not obvious where they should end up.

COMPATIBILITY
    I use GNU/Linux-based systems exclusively, but everything has been
    tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and
    CentOS. I can imagine there's something I missed when testing on
    non-Linux systems, so please let me know if you find any issues.

SEE ALSO
    sort(1)


#+END_EXAMPLE

** vnl-join
#+BEGIN_EXAMPLE
NAME
    vnl-join - joins log files on a particular field

SYNOPSIS
     $ cat a.vnl
     # a b
     AA 11
     bb 12
     CC 13
     dd 14
     dd 123

     $ cat b.vnl
     # a c
     aa 1
     cc 3
     bb 4
     ee 5
     - 23

     Try to join unsorted data on field 'a':
     $ vnl-join -j a a.vnl b.vnl
     # a b c
     join: /dev/fd/5:3: is not sorted: CC 13
     join: /dev/fd/6:3: is not sorted: bb 4

     Sort the data, and join on 'a':
     $ vnl-join --vnl-sort - -j a a.vnl b.vnl | vnl-align
     # a  b c
     bb  12 4

     Sort the data, and join on 'a', ignoring case:
     $ vnl-join -i --vnl-sort - -j a a.vnl b.vnl | vnl-align
     # a b c
     AA 11 1
     bb 12 4
     CC 13 3

     Sort the data, and join on 'a'. Also print the unmatched lines from both files:
     $ vnl-join -a1 -a2 --vnl-sort - -j a a.vnl b.vnl | vnl-align
     # a  b   c
     -   -   23
     AA   11 -
     CC   13 -
     aa  -    1
     bb   12  4
     cc  -    3
     dd  123 -
     dd   14 -
     ee  -    5

     Sort the data, and join on 'a'. Print the unmatched lines from both files,
     Output ONLY column 'c' from the 2nd input:
     $ vnl-join -a1 -a2 -o 2.c --vnl-sort - -j a a.vnl b.vnl | vnl-align
     # c
     23
     -
     -
      1
      4
      3
     -
     -
      5

DESCRIPTION
      Usage: vnl-join [join options]
                      [--vnl-sort -|[sdfgiMhnRrV]+]
                      [ --vnl-[pre|suf]fix[1|2] xxx    |
                        --vnl-[pre|suf]fix xxx,yyy,zzz |
                        --vnl-autoprefix               |
                        --vnl-autosuffix ]
                      logfile1 logfile2 ...

    This tool joins several vnlog files on a given field. "vnl-join" is a
    wrapper around the GNU coreutils "join" tool. Since this is a wrapper,
    most commandline options and behaviors of the "join" tool are present;
    consult the join(1) manpage for detail. The differences from GNU
    coreutils "join" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   The columns are referenced by name, not index. So instead of saying

          join -j1

        to join on the first column, you say

          join -j time

        to join on column "time".

    *   -1 and -2 are supported, but *must* refer to the same field. Since
        vnlog knows the identify of each field, it makes no sense for -1 and
        -2 to be different. So pass "-j" instead, it makes more sense in
        this context.

    *   "-a-" is available as a shorthand for "-a1 -a2": this is a full
        outer join, printing unmatched records from both of the inputs.
        Similarly, "-v-" is available as a shorthand for "-v1 -v2": this
        will output *only* the unique records in both of the inputs.

    *   "vnl-join"-specific options are available to adjust the field-naming
        in the output:

          --vnl-prefix1
          --vnl-suffix1
          --vnl-prefix2
          --vnl-suffix2
          --vnl-prefix
          --vnl-suffix
          --vnl-autoprefix
          --vnl-autosuffix

        See "Field names in the output" below for details.

    *   A "vnl-join"-specific option "--vnl-sort" is available to sort the
        input and/or output. See below for details.

    *   By default we call the "join" tool to do the actual work. If the
        underlying tool has a different name or lives in an odd path, this
        can be specified by passing "--vnl-tool TOOL"

    *   If no "-o" is given, we output the join field, the remaining fields
        in logfile1, the remaining fields in logfile2, .... This is what "-o
        auto" does, except we also handle empty vnlogs correctly.

    *   "-e" is not supported because vnlog uses "-" to represent undefined
        fields.

    *   "--header" is not supported because vnlog assumes a specific header
        structure, and "vnl-join" makes sure that this header is handled
        properly

    *   "-t" is not supported because vnlog assumes whitespace-separated
        fields

    *   "--zero-terminated" is not supported because vnlog assumes
        newline-separated records

    *   Rather than only 2-way joins, this tool supports N-way joins for any
        N > 2. See below for details.

    Past that, everything "join" does is supported, so see that man page for
    detailed documentation. Note that all non-legend comments are stripped
    out, since it's not obvious where they should end up.

  Field names in the output
    By default, the field names in the output match those in the input. This
    is what you want most of the time. It is possible, however that a column
    name adjustment is needed. One common use case for this is if the files
    being joined have identically-named columns, which would produce
    duplicate columns in the output. Example: we fixed a bug in a program,
    and want to compare the results before and after the fix. The program
    produces an x-y trajectory as a function of time, so both the bugged and
    the bug-fixed programs produce a vnlog with a legend

     # time x y

    Joining this on "time" will produce a vnlog with a legend

     # time x y x y

    which is confusing, and *not* what you want. Instead, we invoke
    "vnl-join" as

     vnl-join --vnl-suffix1 _buggy --vnl-suffix2 _fixed -j time buggy.vnl fixed.vnl

    And in the output we get a legend

     # time x_buggy y_buggy x_fixed y_fixed

    Much better.

    Note that "vnl-join" provides several ways of specifying this. The above
    works *only* for 2-way joins. An alternate syntax is available for N-way
    joins, a comma-separated list. The same could be expressed like this:

     vnl-join -a- --vnl-suffix _buggy,_fixed -j time buggy.vnl fixed.vnl

    Finally, if passing in structured filenames, "vnl-join" can infer the
    desired syntax from the filenames. The same as above could be expressed
    even simpler:

     vnl-join --vnl-autosuffix -j time buggy.vnl fixed.vnl

    This works by looking at the set of passed in filenames, and stripping
    out the common leading and trailing strings.

  Sorting of input and output
    The GNU coreutils "join" tool expects sorted columns because it can then
    take only a single pass through the data. If the input isn't sorted,
    then we can use normal shell substitutions to sort it:

     $ vnl-join -j key <(vnl-sort -s -k key a.vnl) <(vnl-sort -s -k key b.vnl)

    For convenience "vnl-join" provides a "--vnl-sort" option. This allows
    the above to be equivalently expressed as

     $ vnl-join -j key --vnl-sort - a.vnl b.vnl

    The "-" after the "--vnl-sort" indicates that we want to sort the
    *input* only. If we also want to sort the output, pass the short codes
    "sort" accepts instead of the "-". For instance, to sort the input for
    "join" and to sort the output numerically, in reverse, do this:

     $ vnl-join -j key --vnl-sort rg a.vnl b.vnl

    The reason this shorthand exists is to work around a quirk of "join".
    The sort order is *assumed* by "join" to be lexicographical, without any
    way to change this. For "sort", this is the default sort order, but
    "sort" has many options to change the sort order, options which are
    sorely missing from "join". A real-world example affected by this is the
    joining of numerical data. If you have "a.vnl":

     # time a
     8 a
     9 b
     10 c

    and "b.vnl":

     # time b
     9  d
     10 e

    Then you cannot use "vnl-join" directly to join the data on time:

     $ vnl-join -j time a.vnl b.vnl
     # time a b
     join: /dev/fd/4:3: is not sorted: 10 c
     join: /dev/fd/5:2: is not sorted: 10 e
     9 b d
     10 c e

    Instead you must re-sort both files lexicographically, *and* then
    (because you almost certainly want to) sort it back into numerical
    order:

     $ vnl-join -j time <(vnl-sort -s -k time a.vnl) <(vnl-sort -s -k time b.vnl) |
       vnl-sort -s -n -k time
     # time a b
     9 b d
     10 c e

    Yuck. The shorthand described earlier makes the interface part of this
    palatable:

     $ vnl-join -j time --vnl-sort n a.vnl b.vnl
     # time a b
     9 b d
     10 c e

    Note that the input sort is stable: "vnl-join" will invoke "vnl-sort
    -s". If you want a stable post-sort, you need to ask for it with
    "--vnl-sort s...".

  N-way joins
    The GNU coreutils "join" tool is inherently designed to join *exactly*
    two files. "vnl-join" extends this capability by chaining together a
    number of "join" invocations to produce a generic N-way join. This works
    exactly how you would expect with the following caveats:

    *   Full outer joins are supported by passing "-a-", but no other "-a"
        option is supported. This is possible, but wasn't obviously worth
        the trouble.

    *   "-v" is not supported. Again, this is possible, but wasn't obviously
        worth the trouble.

    *   Similarly, "-o" is not supported. This is possible, but wasn't
        obviously worth the trouble, especially since the desired behavior
        can be obtained by post-processing with "vnl-filter".

BUGS AND CAVEATS
    The underlying "sort" tool assumes lexicographic ordering, and matches
    fields purely based on their textual contents. This means that for the
    purposes of joining, 10, 10.0 and 1.0e1 are all considered different. If
    needed, you can normalize your keys with something like this:

     vnl-filter -p x='sprintf("%f",x)'

COMPATIBILITY
    I use GNU/Linux-based systems exclusively, but everything has been
    tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and
    CentOS. I can imagine there's something I missed when testing on
    non-Linux systems, so please let me know if you find any issues.

SEE ALSO
    join(1)


#+END_EXAMPLE

** vnl-tail
#+BEGIN_EXAMPLE
NAME
    vnl-tail - tail a log file, preserving the legend

SYNOPSIS
     $ read_temperature | tee temp.vnl
     # temperature
     29.5
     30.4
     28.3
     22.1
     ... continually produces data

     ... at the same time, in another terminal
     $ vnl-tail -f temp.vnl
     # temperature
     28.3
     22.1
     ... outputs data as it comes in

DESCRIPTION
      Usage: vnl-tail [options] logfile logfile logfile ... < logfile

    This tool runs "tail" on given vnlog files in various ways. "vnl-tail"
    is a wrapper around the GNU coreutils "tail" tool. Since this is a
    wrapper, most commandline options and behaviors of the "tail" tool are
    present; consult the tail(1) manpage for detail. The differences from
    GNU coreutils "tail" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   "-c" is not supported because vnlog really doesn't want to break up
        lines

    *   "--zero-terminated" is not supported because vnlog assumes
        newline-separated records

    *   By default we call the "tail" tool to do the actual work. If the
        underlying tool has a different name or lives in an odd path, this
        can be specified by passing "--vnl-tool TOOL"

    Past that, everything "tail" does is supported, so see that man page for
    detailed documentation.

COMPATIBILITY
    I use GNU/Linux-based systems exclusively, but everything has been
    tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and
    CentOS. I can imagine there's something I missed when testing on
    non-Linux systems, so please let me know if you find any issues.

SEE ALSO
    tail(1)


#+END_EXAMPLE

** vnl-ts
#+BEGIN_EXAMPLE
NAME
    vnl-ts - add a timestamp to a vnlog stream

SYNOPSIS
     $ read_temperature
     # temperature
     29.5
     30.4
     28.3
     22.1
     ... continually produces data at 1Hz

     $ read_temperature | vnl-ts -s %.s
     # time-rel temperature
     0.013893 30.2
     1.048695 28.6
     2.105592 29.3
     3.162873 22.0
     ...

DESCRIPTION
      Usage: vnl-ts [-i | -s] [-m] [--vnl-field t] format < pipe

    This tool runs "ts" on given vnlog streams. "vnl-ts" is a wrapper around
    the "ts" tool from Joey Hess's moreutils
    <https://joeyh.name/code/moreutils/> toolkit. Since this is a wrapper,
    most commandline options and behaviors of the "ts" tool are present;
    consult the ts(1) manpage for details. The differences from "ts" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   The format *must* be passed-in by the user; no default is assumed.

    *   The given format *must not* contain whitespace, so that it fits a
        single vnlog field.

    *   "-r" is not supported: it assumes input timestamps with whitespace,
        which is incompatible with vnlog

    *   A "vnl-ts"-specific option "--vnl-field" is available to set the
        name of the new field. If omitted, a reasonable default will be
        used.

    *   By default we call the "ts" tool to do the actual work. If the
        underlying tool has a different name or lives in an odd path, this
        can be specified by passing "--vnl-tool TOOL"

    Past that, everything "ts" does is supported, so see that man page for
    detailed documentation.

COMPATIBILITY
    By default this calls the tool named "ts". At least on FreeBSD, it's
    called "moreutils-ts", so on such systems you should invoke "vnl-ts
    --vnl-tool moreutils-ts ..."

    I use GNU/Linux-based systems exclusively, but everything has been
    tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and
    CentOS. I can imagine there's something I missed when testing on
    non-Linux systems, so please let me know if you find any issues.

SEE ALSO
    ts(1)


#+END_EXAMPLE

** vnl-uniq
#+BEGIN_EXAMPLE
NAME
    vnl-uniq - uniq a log file, preserving the legend

SYNOPSIS
     $ cat sizes-colors.vnl
     # size  color
     12     blue
     11     yellow
     13     yellow
     53     blue
     34     yellow
     22     orange
     11     orange

     $ < sizes-colors.vnl | vnl-filter -p color | vnl-sort -k color | vnl-uniq -c
     # count color
           2 blue
           2 orange
           3 yellow

     $ < sizes-colors.vnl | vnl-sort -k color | vnl-uniq --group -f -1
     # size color
     12     blue
     53     blue

     11     orange
     22     orange

     11     yellow
     13     yellow
     34     yellow

DESCRIPTION
      Usage: vnl-uniq [options] < logfile

    This tool runs "uniq" on a given vnlog dataset. "vnl-uniq" is a wrapper
    around the GNU coreutils "uniq" tool. Since this is a wrapper, most
    commandline options and behaviors of the "uniq" tool are present;
    consult the uniq(1) manpage for detail. The differences from GNU
    coreutils "uniq" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   "--zero-terminated" is not supported because vnlog assumes
        newline-separated records

    *   Only *one* input is supported (a file on the cmdline or data on
        standard input), and the output *always* goes to standard output.
        Specifying the output as a file on the commandline is not supported.

    *   "--vnl-count NAME" can be given to name the "count" column. "-c" is
        still supported to add the default new column named "count", but if
        another name is wanted, "--vnl-count" does that. "--vnl-count"
        implies "-c"

    *   In addition to the normal behavior of skipping fields at the start,
        "-f" and "--skip-fields" can take a negative argument to skip the
        *all but the last* N fields. For instance, to use only the one last
        field, pass "-f -1" or "--skip-fields=-1".

    *   By default we call the "uniq" tool to do the actual work. If the
        underlying tool has a different name or lives in an odd path, this
        can be specified by passing "--vnl-tool TOOL"

    Past that, everything "uniq" does is supported, so see that man page for
    detailed documentation.

COMPATIBILITY
    I use GNU/Linux-based systems exclusively, but everything has been
    tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and
    CentOS. I can imagine there's something I missed when testing on
    non-Linux systems, so please let me know if you find any issues.

SEE ALSO
    uniq(1)


#+END_EXAMPLE

** vnl-tac
#+BEGIN_EXAMPLE
NAME
    vnl-tac - concatenate and print vnlog data in reverse

SYNOPSIS
     $ cat tst.vnl
     # time temperature
     1.0 29.5
     2.0 30.4
     3.5 28.3
     6.0 22.1


     $ vnl-tac tst.vnl
     # time temperature
     6.0 22.1
     3.5 28.3
     2.0 30.4
     1.0 29.5

DESCRIPTION
      Usage: vnl-tac < input.vnl
             vnl-tac input.vnl

    This tool runs "tac" on given vnlog streams. "vnl-tac" is a wrapper
    around the GNU coreutils "tac" tool. The commandline options of "tac"
    don't make sense in the context of vnlog, so only one vnlog-specific
    option is supported:

    *   By default we call the "tac" tool to do the actual work. If the
        underlying tool has a different name or lives in an odd path, this
        can be specified by passing "--vnl-tool TOOL"

    See the "tac" manpage for detailed documentation.

COMPATIBILITY
    I use GNU/Linux-based systems exclusively, but everything has been
    tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and
    CentOS. I can imagine there's something I missed when testing on
    non-Linux systems, so please let me know if you find any issues.

SEE ALSO
    tac(1)


#+END_EXAMPLE

** vnl-paste
#+BEGIN_EXAMPLE
NAME
    vnl-paste - combine log files record-by-record

SYNOPSIS
    vnl-paste concatenates records from the given data files. Comments are
    stripped to make sure things line up

     $ cat a.vnl
     ## gathered from sensor model xxx
     # humidity temperature
     90 25
     80 20
     81 19
     82 18
     70 15

     $ cat b.vnl
     # position
     10
     20
     ## we moved the sensor
     150
     160
     170

     $ vnl-paste a.vnl b.vnl
     # humidity temperature position
     90 25  10
     80 20  20
     81 19  150
     82 18  160
     70 15  170

    We can also give it more than 2 files and annotate the field labels

     $ cat c.vnl
     # position
     12
     18
     155
     168
     190

     $ vnl-paste --vnl-suffix2 _b a.vnl b.vnl c.vnl | vnl-align
     # humidity temperature position_b position
     90         25           10         12
     80         20           20         18
     81         19          150        155
     82         18          160        168
     70         15          170        190

     $ vnl-paste --vnl-suffix ,_b,_c a.vnl b.vnl c.vnl | vnl-align
     # humidity temperature position_b position_c
     90         25           10         12
     80         20           20         18
     81         19          150        155
     82         18          160        168
     70         15          170        190

     $ vnl-paste --vnl-autosuffix a.vnl b.vnl c.vnl | vnl-align
     # humidity_a temperature_a position_b position_c
     90           25             10         12
     80           20             20         18
     81           19            150        155
     82           18            160        168
     70           15            170        190

DESCRIPTION
      Usage: vnl-paste
                      [ --vnl-[pre|suf]fix[1|2] xxx    |
                        --vnl-[pre|suf]fix xxx,yyy,zzz |
                        --vnl-autoprefix               |
                        --vnl-autosuffix ]
                      logfile1 logfile2 ...

    This tool is essentially a join using the line number as the key. This
    tool concatenates the data in the given vnlog files record-by-record.
    "vnl-paste" is a wrapper around the GNU coreutils "paste" tool. Since
    this is a wrapper, the behavior of the underlying tool is preserved;
    consult the paste(1) manpage for detail. The differences from GNU
    coreutils "paste" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   By default we call the "paste" tool to do the actual work. If the
        underlying tool has a different name or lives in an odd path, this
        can be specified by passing "--vnl-tool TOOL"

    *   None of the "paste" options make sense in a vnlog context, so this
        tool has only the vnlog-specific "--vnl-..." options.

    Note that all non-legend comments are stripped out, since it's not
    obvious where they should end up.

  Field names in the output
    The field name logic works the same way as in "vnl-join", with all the
    suffix and prefix options from that tool being available here.

    By default, the field names in the output match those in the input. This
    is what you want most of the time. It is possible, however that a column
    name adjustment is needed. One common use case for this is if the files
    being pasted have identically-named columns, which would produce
    duplicate columns in the output. Example: we fixed a bug in a program,
    and want to compare the results before and after the fix. The program
    produces an x-y trajectory, so both the bugged and the bug-fixed
    programs produce a vnlog with a legend

     # x y

    Pasting these will produce a vnlog with a legend

     # x y x y

    which is confusing, and *not* what you want. Instead, we invoke
    "vnl-paste" as

     vnl-paste --vnl-suffix1 _buggy --vnl-suffix2 _fixed buggy.vnl fixed.vnl

    And in the output we get a legend

     # x_buggy y_buggy x_fixed y_fixed

    Much better.

    Note that "vnl-paste" provides several ways of specifying this. The
    above works *only* for 2-way pastes. An alternate syntax is available
    for N-way pastes, a comma-separated list. The same could be expressed
    like this:

     vnl-paste --vnl-suffix _buggy,_fixed buggy.vnl fixed.vnl

    Finally, if passing in structured filenames, "vnl-paste" can infer the
    desired syntax from the filenames. The same as above could be expressed
    even simpler:

     vnl-paste --vnl-autosuffix buggy.vnl fixed.vnl

    This works by looking at the set of passed in filenames, and stripping
    out the common leading and trailing strings.

BUGS AND CAVEATS
    If the input data files has a mismatched number of lines, the extra data
    will be output without the null-column markers. *This will result in
    mislabeled data*. Use this tool only if you're sure you have a matching
    number of records. For instance:

     $ vnl-paste <(echo '# x'; seq 3) <(echo '# y'; seq 5)
     # x y
     1      1
     2      2
     3      3
            4
            5

     $ vnl-paste <(echo '# x'; seq 3) <(echo '# y'; seq 5) | vnl-align
     # x y
     1   1
     2   2
     3   3
     4
     5

COMPATIBILITY
    I use GNU/Linux-based systems exclusively, but everything has been
    tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and
    CentOS. I can imagine there's something I missed when testing on
    non-Linux systems, so please let me know if you find any issues.

SEE ALSO
    paste(1)


#+END_EXAMPLE


** vnl-gen-header
#+BEGIN_EXAMPLE
NAME
    vnl-gen-header - create definition for vnlog output from C

SYNOPSIS
     $ vnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' > vnlog_fields_generated.h

DESCRIPTION
    We provide a simple C library to produce vnlog output. The fields this
    library outputs must be known at compile time, and are specified in a
    header created by this tool. Please see the vnlog documentation for
    instructions on how to use the library

ARGUMENTS
    This tool needs to be given a list of field definitions. First we look
    at the commandline, and if the definitions are not available there, we
    look on STDIN. Each definition is a string "type name" (one def per
    argument on the commandline or per line on STDIN). If reading from
    STDIN, we ignore blank lines, and treat any line starting with "#" as a
    comment.

    Each def represents a single output field. Each such field spec in a
    C-style variable declaration with a type followed by a name. Note that
    these field specs contain whitespace, so each one must be quoted before
    being passed to the shell.

    The types can be basic scalars, possibly with set widths ("char",
    "double", "int", "uint32_t", "unsigned int", ...), a NULL-terminated
    string ("char*") or a generic chunk of binary data ("void*").

    The names must consist entirely of letters, numbers or "_", like
    variables in C.


#+END_EXAMPLE

** vnl-make-matrix
#+BEGIN_EXAMPLE
NAME
    vnl-make-matrix - create a matrix from a one-point-per-record vnlog

SYNOPSIS
     $ cat /tmp/dat.vnl
     # i j x
     0 0 1
     0 1 2
     0 2 3
     1 0 4
     1 1 5
     1 2 6
     2 0 7
     2 1 8
     2 2 9
     3 0 10
     3 1 11
     3 2 12

     $ </tmp/dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp --prefix test_
     Writing to '/tmp/test_x.matrix'

     $ cat /tmp/test_x.matrix
     1 2 3
     4 5 6
     7 8 9
     10 11 12

DESCRIPTION
    Vnlog represents each "data item" as a line of text. This is very often
    what one wants, but at times it isn't. One example of this is matrix
    data: we want each line to contain a whole row of a matrix. This script
    exists for convenience, to bridge this gap.

    The input is an vnlog, coming in on STDIN and/or in files given on the
    commandline. This vnlog must have at least two fields: the
    least-significant (slowest-changing) index of each point (must be the
    *first* field), and as many value fields as desired. These points must
    be written out in order, and it is assumed that all entries in the
    matrix are specified. The output is a set of (non-vnlog) matrix files in
    the directory given in the "--outdir" argument. These files are named
    "PREFIX_XXX.matrix" where "PREFIX" comes from --prefix (or empty) and
    "XXX" is the field name. These matrices can be loaded into any analysis
    tool (numpy for instance), or plotted directly with gnuplot:

     set size ratio -1
     plot "/tmp/test_x.matrix" matrix with image
     pause -1


#+END_EXAMPLE

* Repository

https://github.com/dkogan/vnlog/

* Authors

Dima Kogan (=dima@secretsauce.net=) wrote this toolkit for his work at the Jet
Propulsion Laboratory, and is delighted to have been able to release it
publically

Chris Venter (=chris.venter@gmail.com=) wrote the base64 encoder

* License and copyright

This library is free software; you can redistribute it and/or modify it under
the terms of the GNU Lesser General Public License as published by the Free
Software Foundation; either version 2.1 of the License, or (at your option) any
later version.

Copyright 2016-2017 California Institute of Technology

Copyright 2017-2018 Dima Kogan (=dima@secretsauce.net=)

=b64_cencode.c= comes from =cencode.c= in the =libb64= project. It is written by
Chris Venter (=chris.venter@gmail.com=) who placed it in the public domain. The
full text of the license is in that file.
