Tuesday, May 19, 2015

Stably monitoring console programs in Linux

So you want to run a program, while keeping a log file for posterity. Obviously you redirect stdout to a log file (and in this case stderr as well, while running it in the background):

myprogram someparams > myprogram.log 2>&1 &

Now you also want to monitor it while it runs to see how things are going (there is also tee for this, but then you have to keep monitoring). So use:

tail -f myprogram.log

But what if you've ssh'ed into the shell so it's possible that you may get disconnected. You don't want to have your program abort just because you lose connection, or your shell ends for whatever reason. You could use bash's disown, if you've already run the program. But if you're thinking ahead, simply:

nohup myprogram someparams > myprogram.log 2>&1 &

You probably already knew that too. But actually, that raises a problem. No nohup is buffering stdout using glibc and so if you want to monitor output, you'll be waiting a while, and only get it in chunks (or possibly even at the end of execution). To the rescue comes unbuffer, part of the expect package. (In ubuntu, install via apt-get install expect). So finally:

unbuffer nohup myprogram someparams > myprogram.log 2>&1 &

This works by tricking the kernel into thinking it's writing to console, which changes it to line buffering. But there's actually now an even better way, using stdbuf, assuming you have it.

stdbuf -oL nohup myprogram someparams > myprogram.log 2>&1 &

And now you can tail to monitor continuously without issues!