Scaling Up UNIX Commands on Files In-Place

I am sharing my experience in debugging a problem scaling up a UNIX command using a template I had earlier.

I had to modify a single line in 49 different files and wanted to do it in-place: the output filenames to be the same as the input filenames.

You’ll see that things are not always straight forward, especially when scaling up commands in UNIX.

This article will take you about four minutes to read.

source and more information

Introduction

I was configuring Private Internet Access for my Linux system and found it annoying to be typing in a randomly generated user name and password when connecting with the OpenVPN client.

To have OpenVPN client read a username & password from a file, the .ovpn configuration file needs to change:

auth-user-pass

to:

auth-user-pass <filename>

where <filename> is a file and it’s contents are:

<username>
<password>

reference

This is exactly what I want. I will use credentials.txt as the filename.

Problem

For a single file, making this change would be easy. Two, a little annoying, Three, this is starting to be not fun… 49?! Forget it! I need to write a script!

note: the number of files to change will depend on your VPN provider, the number of changes may vary. I am using Private Internet Access’ OpenVPN files, available here.

How can I make all these file changes painlessly? Can I work on all the files in-place as there’s 49 files.

Solution: Write a script!

In my scaling up UNIX commands quickly, I took a gif converting command and used xargs to apply the command to every .mov file in the folder:

ls *.mov | sed -E "s/\.[a-z][a-z][a-z]//g" | xargs -I % sh -c "ffmpeg -i %.mov -vf "scale=iw/2:ih/2" -pix_fmt rgb24 -r 15 -f gif - | gifsicle --delay=3 > %.gif"

The core solution for my new problem is:

cat CA Toronto.ovpn | sed 's/auth-user-pass/auth-user-pass credentials.txt/

Which results in:

$ cat CA Toronto.ovpn | grep auth-user-pass
auth-user-pass credentials.txt

Perfect. Now there are 48 more files to process. Let’s scale up this function using the previous template. It’ll be easier this time as I want the output filename to be the same as the input filename.

ls *.ovpn | xargs -d '\n' -I % sh -c "cat '%' | sed 's/auth-user-pass/auth-user-pass credentials.txt/' > '%'"

Running that, I check the result:

$ cat *.ovpn | grep auth-user-pass
# => nil

What the?! Empty files??? Something weird is going on.

Digging further

All the files processed by the scaled up command are empty. Not just missing the auth-user-pass line, the whole file is empty.

This tells me that scaling up the command isn’t as straight-forward as dropping it into the modified template I had before.

The whole file being empty does give me a big hint. The original sed command is only manipulating a line. Why did the rest of the file go blank?

Generalized Shape

The shape of the whole command generalized is:

ls <all files> | xargs sh -c "cat file | sed 'option' > file"

Taking away all the options, I see the command is reading and writing the file at the same time. This creates as paradox and I’m surprised I didn’t open a black hole using this command! :-)

Better Solution: temporary output

Seeing the command is reading and writing the file at the same time, the best solution would be to write the new file to a temporary output then rename that output.

ls <all files> | xargs sh -c "cat file | sed 'option' > file.tmp; mv file.tmp file"

Applying to our situation:

ls *.ovpn | xargs -d '\n' -I % sh -c "cat '%' | sed 's/auth-user-pass/auth-user-pass credentials.txt/' > '%'.temp; mv '%'.temp '%' "

Results in:

$ cat *.ovpn | grep auth-user-pass
auth-user-pass credentials.txt
# ... (48 more times)

Great! This is exactly what I want. The only difference:

> '%'.temp; mv '%'.temp '%'`

Which writes the output to a *.temp file, then renames the file back to the original. This allows the command to read in the file, write output to a new file, once it’s done, rename the new file to the original file, giving the in-place appearance.

Conclusion

I expected modifying 49 files in-place using my handy scaling up command to be a breeze, once I found the command to do what I want on a single file.

When I scaled up a sed command using the previous template, the files became empty. Making me wonder why a command that worked previously didn’t work in this case.

Digging in, I found out the command was reading and writing to the file at the same time. Hence, making a black hole, err, empty files.

Once I understood this, fixing the script template to write to a temporary file then rename that temporary file back to original file resulted in 48 files modified as desired.