Supercharging Vim with Python scripts

Pretty much anything you can do in Vim, you can also do in with Python via the builtin vim module. This may not sound interesting at first, but it can dramatically improve your coding experience. It makes hacking Vim much more feasible, and enjoyable (not to mention testable, which I’ll show below).

But why would you want to hack your editor with Python?

Well, macros are great for smaller one-off tasks, but the moment things get more complicated, debugging them can be a pain in the butt. And no one wants to be debugging Vimscript when you could be using a general purpose programming language, like Python, with all its testing / debugging ecosystem by your side.

With something like Python, the sky truly is the limit.

I tend to make little string manipulation scripts, as I'll show below (here’s all the code). But you could do anything, like pulling data from an API into your editor, or make your own snippets manager, or making a custom web search using text from the current buffer, etc etc.

You get the point. Let's dive in!

Vim setup

As a sanity check, make sure Vim is running Python ok.

Put the following code into your vim.init file, restart Vim, and type gv in a new buffer.

python <<EOF
import vim
def hello():
    vim.command("normal ihello")
EOF

nmap gv :python hello()<cr>

You should see the following.

hello

This means we were able to (1) run Python from Vim and (2) use the pre-installed vim package to run a command.

If you run into issues, make sure:

Vim is actually picking up the changes to your init file (e.g. restart vim, add an echo "hello" line or something to the file to debug, etc.)
Vim’s python is available and working (run :python print("hello") in command mode)
The vim module is available (:python import vim)
Nothing else is overwriting our gv map (like another plugin)

The best resources for learning more about the vim models is actually the help doc from within Vim. You can run :h python-vim and navigate the links with ctrl-] and ctrl-O (to jump back).

Now let's try something more useful. We will write a small script to sort all of the lines inside braces {}.

Replace the python code with this, restart and type gs from within braces.

python <<EOF
import vim
def sort_inside_braces():
    vim.command('normal "ayi{')
    text = vim.eval("@a")
    new_text = "\n".join(sorted(text.split("\n")))
    vim.command(f'normal ci\{{new_text}')
EOF

nmap gs :python sort_inside_braces()<cr>

You should get automatic sorting like this.

Before:
{
  b: 2,
  a: 1,
}

After:
{
  a: 1,
  b: 2,
}

Great! Now let's get into some bigger scripts. But first, we'll need to start organizing our code better.

External scripts

Putting all the code inside your vim.init file is not very scalable. Instead, we can import it from another folder.

You can link to external modules by pointing sys.path to your personal development folder, like this.

import sys
sys.path.insert(0, '/path/to/your/scripts')
# ... then import your code, vim, etc

But what about using third-party modules within Vim?

Linking pip modules

If you want to use third-party modules installed with pip, make sure you run pip install with the same Python executable that Vim is using.

You can find that executable by running :python import sys; print(sys.executable) from inside Vim.

For example, mine was...

/opt/homebrew/opt/[email protected]/bin/python3.10

Use this executable to install new modules. For example, we could install parsley with the following.

vim_python="/opt/homebrew/opt/[email protected]/bin/python3.10"
$vim_python -m pip install parsley

Now you can import parsley from within Vim, like so...

:python import parsley; print(parsley)

... which will print something like this in the status bar.

<module 'parsley' from '/opt/homebrew/lib/python3.10/site-packages/parsley.py'>

Now let's do some heavy lifting. 💪

We're going to use the parsley library to read from our buffer, do some parsing, then write out some resulting text.

Why "parsley"?

Parsley is a parsing library written in Python. It is based on the OMeta language developed by Allesandro Warth and Ian Piumarta (also supported by Alan Kay). I love it because it is easy to learn and scales well.

In parsley, you write a parsing expression grammar (PEG) to parse strings into structures that are easy to manipulate. (Check out my previous post on why you should love PEGs. Did you know that Python itself is parsed with a PEG?)

And for hacking Vim, it's perfect because we will do a lot of parsing to make sense of the surrounding text in the current buffer.

Let's take it for a test drive and write a small script to automatically order HTML attributes.

Example script: order HTML attributes

First, we can describe what we want with a simple unit test.

In a separate folder, make a little test file called test_order_attrs.py.

# test_order_attrs.py
from my_vim_scripts import order_attrs

def test_order_attrs():
    before = '<tag b="on" a="right">'
    after = '<tag a="right" b="on">'
    assert order_attrs(before) == after

(I'm using pytest here, but any testing framework works.)

This makes sure that the a attribute is moved before the b attribute in our dummy 'tag' element.

Pretty straightforward. 🐢

Now for the actual code... One solution is to parse the surrounding HTML tag into a Python dictionary, with shape { "name": str, "attrs": list[str] }, then sort the attrs key. The tag above will become { "name": "tag", "attrs": ['b="on"', 'a="right"'] }.

Here is what that looks like with parsley (check out the parsley docs for more details).

# my_vim_scripts.py
import parsley

# the "->" syntax lets us write Python within the grammar
grammar = """
tag = '<' name:n attrs:a '>' -> {'name': n, 'attrs': a}

name = <letter+>
attrs = (ws attr)*
attr = <name '="' name '"'>
"""

parse = parsley.makeGrammar(grammar, bindings={})

def order_attrs(before):
    tag = parse(before).tag()
    new_attrs = " ".join(sorted(tag['attrs']))
    after = '<' + tag['name'] + ' ' + new_attrs + '>'
    return after

We wrote a little Python code inside the actual grammar definition to reshape the output into the dictionary we want. A small feature, but very powerful for bigger grammars. ⚡️

Now we'll run our test to make sure it passes (don't forget to use the right version of Python).

$ $vim_python -m pytest test_order_attrs.py

============ test session starts =============
platform darwin -- Python 3.10.8, pytest-7.2.2
rootdir: /Users/rex/personal/editor/scripts
collected 1 item

test_order_attrs.py .                    [100%]

============== 1 passed in 0.00s ==============

Great!

Now let's write a Vim function in our init file to hook it up. This will yank the surrounding HTML tag and insert an ordered version instead, using the script from above.

" init.vim
python <<EOF

import sys
sys.path.insert(0,'/Users/rex/personal/editor/scripts')

import vim
from my_vim_scripts import order_attrs

def organize_tag():
    vim.command('normal "aya<') # yank the surrounding tag
    tag = vim.eval("@a")
    new_tag = order_attrs(tag)
    vim.command(f'normal ca<{new_tag}') # overwrite it

EOF

nmap gt :python organize_tag()<cr>

Now for the fun part.

Reload your editor, write a simple HTML tag and type gt from inside it (in normal mode). It should automatically organize the tag's attributes like we tested.

Before:
<tag b="on" a="right">

After:
<tag a="right" b="on">

Pretty cool, right? 😎

Making incremental improvements

Of course, our new script will contain bugs. But let's use a workflow that makes debugging actually feel good.

With our current code, we will get an error if we run our keymap inside the following tag...

<img src="face.jpg" alt="face">

Can you see the issue? The parser gets stuck at the period in face.jpg, since it is not part of the name pattern.

This one is a quick fix, but let’s practice doing it systematically, the low-stress way. 🌱

Copy the exact text that made it fail
Turn that into a new test case, watch it fail
Write just enough code to make it pass
Refactor the code to make it pretty

For example, we could add a test like this…

def test_matches_filename():
    before = '<img src="face.jpg" alt="face">'
    after = '<img alt="face" src="face.jpg">'
    assert order_attrs(before) == after

... watch this fail, then fix our attr pattern with the following…

attr = <name '="' (~'"' anything)* '"'>

And then we can refactor it to make it more readable.

attr = <name '=' '"' quote_contents '"'>
quote_contents = (~'"' anything)*

With this improved version, all we have to do is reload Vim to start using it in our editor. How great is that?

(before)
<img src="face.jpg" alt="face">

(after)
<img alt="face" src="face.jpg">

Looking good!

Test-driven is low-stress

When I started using this workflow for Vim scripts, I gotta say I just fell in love with it.

You work in small steps, your code is verifiably better at each step, and you get faster and faster each step of the way!

You also get frequent “user” feedback, since the script is now part of your personal editor. That's a big plus in my book. 👍

Grow your parsing skills

When I started making these little editor scripts, it felt like I was taking a step backwards at first because I was using my own buggy, unoptimized code. After all, nothing I was writing would be as battle-tested as lxml or Beautiful Soup.

But I think these scripts bring a certain confidence and control over your editor that you just can't beat. It feels great knowing you can build something for yourself, without relying on external APIs, and knowing you can build it fast.

As far as parsing goes, I think it is one of those underrated skills in software that can last your entire career. It is so useful in so many contexts, regardless of whatever programming language you happen to be using. And considering how much time we spend typing things into text files, hacking your editor has got to be one of the best applications of parsing out there.

I hope you give it a shot!

Happy coding, ☕️

~Rex

Rex's Blog