Use Vim Inside A Unix Pipe Like Sed Or AWK

2016-04-05 - By Robert Elder

Updated April 10, 2016: Removed use of unnecessary brackets as per suggestion

Updated April 7, 2016: Use 'cit' instead of 'vitx', correction about identity functions vs idempotence.

On Hacker News

Introduction

In this article, a method for inserting vim into the middle of a unix pipe is presented.  This method has several caveats, but it can produce extremely terse and powerful text transformations that leverage all of the knowledge you already have about vim.  It can even work with augmentations found in plugins.  The command below appears to work in my version of bash, dash, and zsh.  A friend tells me that it also works on his Mac.

A Small Identity Function Example

echo "asdf" | vim - -esbnN -c 'w!/dev/stderr|q!' 2>&1 >/dev/null

The above command shows an example of an identity function we can use as a starting point that we will build upon for performing more interesting text transformations using the '-c' arguent to vim.  For example:

echo "<h1>Change Me</h1>" | vim - -esbnN -c 'norm citOk I Will' -c 'w!/dev/stderr|q!' 2>&1 >/dev/null

produces this output:

<h1>Ok I Will</h1>

The command we used looks a bit messy (sorry, I tried to make it shorter), but this is all we added to our identity function:

-c 'norm citOk I Will'

This '-c' argument has the effect of going into vim, and typing ESC, :, then 'norm citOk I Will'

The 'norm' is short for 'normal', which is refering to the 'normal mode' that you are in when you start up Vim. The character sequence 'citOk I Will' works as follows:

c         #  Change
it        #  Select the inner tag block of the XML the cursor is over.
Ok I Will #  Since we're in insert mode, this just types 'Ok I Will'.

Search And Modify

echo "The quick brown fox." | vim - -esbnN -c 'exe "norm /brown\ndwired "' -c 'w!/dev/stderr|q!' 2>&1 >/dev/null

produces

The quick red fox.

by doing the equivalent of opening vim, typing '/' for search, then typing brown, pressing enter, using 'dw' to delete a word, then typing 'red ' in insert mode.

Difficult Cases With Parsing Quotes

echo '"Hello \" world"' | vim - -esbnN -c 'norm vi"xiabc' -c 'w!/dev/stderr|q!' 2>&1 >/dev/null

produces

"abc"

by doing the equivalent of opening vim, typing 'vi"' to select everything inside matching quotes, 'x' to delete the selection, 'i' to enter insert mode and type 'abc'.

Find And Replace With Backreferences

echo "23452 33 4 65454" | vim - -esbnN -c '%s/\v([0-9]+)/\1,/g' -c 'w!/dev/stderr|q!' 2>&1 >/dev/null

produces

23452, 33, 4, 65454,

by doing the equivalent of opening vim, and typing a regex replacement that matches numbers and puts commas after them.

Using With Vim Plugins

The following example takes advantage of the vim-surround plugin (tested with commit 2d05440ad23f97a7874ebd9b5de3a0e65d25d85c).

echo '"Hello" "World"' | vim - -esbnN -c 'norm $ds"' -c 'set noeol|w!/dev/stderr|q!' 2>&1 >/dev/null

produces

"Hello" World

by doing the equivalent of opening vim, and typing '$' to move to the end of the line, and 'ds"' to remove the surrounding double quotes.

Just Pipe It Into StackSort

If you read the post on XKCD's StackSort Implemented In A Vim Regex, you're probably thinking "Now I can just pipe everything through Stack Overflow", and if so, you're not only correct but you're also a genius.

echo "sort a list in bash" | vim - -esbnN -c '.s/\(.*\)/\=system('"'"'a='"'"'."https:\/\/api.stackexchange.com\/2.2\/".'"'"'; q=`curl -s -G --data-urlencode "q='"'"'.submatch(1).'"'"'" --compressed "'"'"'."${a}search\/advanced?order=desc&sort=relevance&site=stackoverflow".'"'"'" | python -c "'"'"'."exec(\\\"import sys \\nimport json\\nprint(json.loads('"'"''"'"'.join(sys.stdin.readlines()))['"'"'items'"'"'][0]['"'"'question_id'"'"'])\\\")".'"'"'"`; curl -s --compressed "'"'"'."${a}questions\/$q\/answers?order=desc&sort=votes&site=stackoverflow&filter=withbody".'"'"'" | python -c "'"'"'."exec(\\\"import sys\\nimport json\\nprint(json.loads('"'"''"'"'.join(sys.stdin.readlines()))['"'"'items'"'"'][0]['"'"'body'"'"']).encode('"'"'utf8'"'"')\\\")".'"'"'"'"'"')/' -c 'w!/dev/stderr|q!' 2>&1 >/dev/null

Looking at all the quoting might give you nightmares, but it does work.  It produces the following output:

<p>You need to add quotes around <code>${array[@]}</code>, like this:</p>

<pre><code>for j in "${array[@]}"; do echo "$j"; done | sort -n &gt;&gt; result.txt;
</code></pre>

<p>That will prevent bash reinterpreting the spaces inside your array entries.</p>

Which is the first answer to the first question to 'sort a list in bash' on Stack Overflow which was answered by ams.

Unfortunately, the result contains HTML which we don't want.  No problem, just use a Vim command!  Add this extra command argument after the stacksort one, but before the save to stderr and exit:

-c '%s/\(\_.\+\)<code>\(\_.\+\)<\/code>\(\_.\+\)/\2/g'

That change will get you this output:

for j in "${array[@]}"; do echo "$j"; done | sort -n >> result.txt;

And another -c command to clean up the result a bit:

-c '%s/>/>/g'

And you get:

for j in "${array[@]}"; do echo "$j"; done | sort -n >> result.txt;

Now, just pipe that straight into 'bash' without looking at it and call it a day.

Detailed Explanation Of This Technique

Let's breakdown what's happening in the command presented above:

vim           #  Start vim
-             #  Tell vim to read its input from stdin instead of from a file
-             #  Begin command line options to vim
    e         #  Start vim in ex-mode
    s         #  Silent mode
    b         #  Binary mode
    n         #  Don't create a swap file:  We don't want side effects
    N         #  Do not turn on vi compatibility mode (I needed this for plugins to work)
-c            #  As soon as vim is opened, run the following ex command
    w!/dev/stderr     #  Write the contents of Vim to /dev/stderr
    |                 #  Run another ex command
    q!                #  Quit without saving
2>&1          #  Redirect stderr back to stdout    
>/dev/null    #  Redirect the nasty 'Vim: Reading from stdin...' message to /dev/null

Will Hang And Crash On Non-Terminating Inputs

One very serious caveat is that you cannot use this technique for processing data that can be infinite in length. Vim will always attempt to read stdin until it receives an end of file character, and if this never happens it will simply keep reading forever and use up all of your memory. In addition, you cant use CTRL C to kill this process because vim puts the terminal into an altered state.  After testing this on my machine programs like sort seem to be intelligent when you run them on infinite sized inputs, in that they don't eat up all your memory and will instead just hang forever.  In this case, Vim will just keep using memory and eventually start eating up swap space.

Issues With Empty Files And Newline Cases

I experimented with a number of flags, but unfortunately, there was always at least one case where newlines were modified in some way in the output that did not reflect what was in the input.  Special cases like empty input, input that is a single newline, or input ending in one or more newlines are problematic because of overzealous adding or removing of terminating newlines at the end of files.

You can experiment with using the 'set noeol' command, but this will delete desired terminating newlines from the input as well as the unwanted newlines that are added by Vim:

echo -en "\n" | vim - -esbnN -c 'norm $ds"' -c 'set noeol|w!/dev/stderr|q!' 2>&1 >/dev/null

As far as I can tell, the original command at the top can act as an identity function for every case except for an empty file (where an extra newline is added).  The alternative using 'set noeol' seems to be cause unwanted changes for every input that does not end with a newline because it will always try to remove them.

Use of /dev/stderr

stderr is used as an output to allow us to ignore the devistatingly annoying "Vim: Reading from stdin..." text that appears when reading from stdin. Please, if you ever design anything that runs from the command line, please allow a way to turn off all superfluous output. Even if you find yourself saying "Why would anyone want to do that?".

One unfortunate side-effect of putting the output through stderr and then re-directing it back to stdout, is that your output will now pick up any error messages that are printed to stderr.  I didn't encounter any while testing this, but I did find that in an older version of Vim, that there are extra newlines output to stderr every time that Vim is run.  Very annoying.

If you encounter issues with the output mixing with stderr, you may be able to use the following alternative:

echo "asdf" | vim - -esbnN -c 'w!/dev/fd/3|q!' 3>&1 >/dev/null 2>&1

This will pipe the output through file descriptor 3 and ignore everything from stderr and stdout.  I'm not sure how portable this will be and I'm unsure if there are any other negative side effects to using /dev/fd/3 (for example, a conflict with an internal use of fd 3).  Use at your own risk.

Altered Terminal Mode

When you enter vim, the terminal is put into raw mode where keyboard commands like CTRL +C can be handled directly by the program instead of being handled by the kernel.  If you use the technique introduced here, and you have an error in one of the vim commands, it won't properly exit vim, and because we're starting vim without a UI, the terminal will hang foverer, and CTRL +C won't help you.  This is annoying, but you'll just have to close the terminal.

Conclusion

Tools like AWK or sed are great for those simple hacky instances where you need to automate some form of text processing, but they often lack things that you could do in a few key strokes in Vim.  Using the technique presented in this article, you can do anything in a unix pipe that you could do if you were inside Vim.  This technique does not work for infinite length streams, and it has a few other caveats, but it should prove useful when you need a quick and dirty solution.