How to tell if your shell is a subshell of Vim

If you’re like me, you use Vim. You probably also use :sh[ell]. You may even forget that you used :sh and open Vim again.

I’ve been getting better about it, but every once in a while I end up with 2 or
3 nested Vim processes and subshells.

This tends to become apparent when I happen to open a file I already have open and Vim barks about there already being a swap file.

Checking the old fashioned way

To help combat this situation, I’ve been running ps before running Vim again
to see if Vim is already running in my current terminal and if so, I can simply
exit and get back to Vim.

Problem there is that I end up running ps quite a bit and still forget to run
it some of the time.

What if there was a way to show that I’m currently inside of a Vim subshell on
my shell’s prompt? That’d be spectacular!

But I wasn’t sure how to accurately determine if I was in actually in a Vim
subshell.

I’ve been researching this off and on for a month or so. Dabbled quite a bit
with pstree and couldn’t come up with anything that didn’t seem a bit too
fragile for day to day usage.

My recent bout of research led me to explore the environment variables that Vim sets at runtime. There are a few of them, $MYVIMRC and $VIMRUNTIME seemed like they could be the ticket.

Unfortunately, both variables provide mixed results.

$MYVIMRC is only present if you have a .vimrc file.

$VIMRUNTIME could be set in your environment and always be present.

With that, YMMV on which variable you would need to interrogate. For me,
$VIMRUNTIME made the most sense as I’ve never set that value directly.

$MYVIMRC would have been sufficient as well as I always have a .vimrc file.

Now that I know what I needed to be checking, all I needed to do was wire it up
to my prompt, similar to how I configured my minimalist git prompt:

zsh

vim_prompt() {
  if [ ! -z $VIMRUNTIME ]; then
    echo "vim ";
  fi
}

PS1='$(vim_prompt)# '

And for consistency with my bash prompt:

bash

vim_prompt() {
  if [ ! -z $VIMRUNTIME ]; then
    echo "vim ";
  fi
}

PS1='$(vim_prompt)$ '

Not much to it! You can configure the indicator however you’d like. For me, I
just prefix my prompt with a little :sh (grey colon, green “sh”).

You can check out the entirety of my prompt and other my other environment
configuration and such over in my dotfiles!