Scaling Up UNIX Commands on Files In-Place
I am sharing my experience in debugging a problem scaling up a UNIX command using a template I had earlier.
I had to modify a single line in 49 different files and wanted to do it in-place: the output filenames to be the same as the input filenames.
You’ll see that things are not always straight forward, especially when scaling up commands in UNIX.
This article will take you about four minutes to read.
To have OpenVPN client read a username & password from a file, the
.ovpn configuration file needs to change:
<filename> is a file and it’s contents are:
This is exactly what I want. I will use
credentials.txt as the
For a single file, making this change would be easy. Two, a little annoying, Three, this is starting to be not fun… 49?! Forget it! I need to write a script!
note: the number of files to change will depend on your VPN provider, the number of changes may vary. I am using Private Internet Access’ OpenVPN files, available here.
How can I make all these file changes painlessly? Can I work on all the files in-place as there’s 49 files.
Solution: Write a script!
In my scaling up UNIX commands
I took a gif converting command and used
xargs to apply the command
.mov file in the folder:
The core solution for my new problem is:
Which results in:
Perfect. Now there are 48 more files to process. Let’s scale up this function using the previous template. It’ll be easier this time as I want the output filename to be the same as the input filename.
Running that, I check the result:
What the?! Empty files??? Something weird is going on.
All the files processed by the scaled up command are empty. Not just
auth-user-pass line, the whole file is empty.
This tells me that scaling up the command isn’t as straight-forward as dropping it into the modified template I had before.
The whole file being empty does give me a big hint. The original
command is only manipulating a line. Why did the rest of the file go blank?
The shape of the whole command generalized is:
Taking away all the options, I see the command is reading and writing the file at the same time. This creates as paradox and I’m surprised I didn’t open a black hole using this command! :-)
Better Solution: temporary output
Seeing the command is reading and writing the file at the same time, the best solution would be to write the new file to a temporary output then rename that output.
Applying to our situation:
Great! This is exactly what I want. The only difference:
Which writes the output to a
*.temp file, then renames the file back
to the original. This allows the command to read in the file, write
output to a new file, once it’s done, rename the new file to the
original file, giving the in-place appearance.
I expected modifying 49 files in-place using my handy scaling up command to be a breeze, once I found the command to do what I want on a single file.
When I scaled up a
sed command using the previous template, the
files became empty. Making me wonder why a command that worked
previously didn’t work in this case.
Digging in, I found out the command was reading and writing to the file at the same time. Hence, making a black hole, err, empty files.
Once I understood this, fixing the script template to write to a temporary file then rename that temporary file back to original file resulted in 48 files modified as desired.