Automate boring stuff

2017-01-01 Bash, Git, Python No comments


In my current company all the people who perform creative work (mostly programmers) need to prepare so-called PKUP report. PKUP stands for Podwy┼╝szone Koszty Uzyskania Przychodu in the Polish language. It’s legal regulation in Poland, which allows paying a lower income tax due to the particular type of work. For the regular employee, it means that he or she will simply get a bit higher salary per month.

How the report looks in practice?

As a programmer, I simply create software as a source code. Added, removed and modified lines of code in the existing codebase are treated as my creative work. Luckily, we use Git so I can generate *.diff files from the Git repositories I’m contributing to. Besides that, I need to prepare document as a *.docx file with a short description of my work. My tasks look different every month, but report actually looks almost the same every month. Preparing this report is boring and repeatable stuff.

Let’s automate it!

Generating *.diff files from Git repos

I simply created a shell script, which goes through predefined project directories and saves *.diff files with names the same as project directory with changes performed by me from the 20th day of the last month until now.

Generating *.docx document

Next, I created a python script, which is parametrized and used by shell script. It uses python-docx library for generating *.docx report. I’ve chosen such option, because it’s one of the simplest solutions I’ve found and it’s lightweight. Moreover it can be easily used on Unix systems and integrated with shell scripts.


I wanted to make a script available and usable for everyone, so I created .pkup.conf file, which is responsible for personalization and configuration of the script. I think, it looks pretty straightforward.

yDEV_PROJECTS_LIST=(backoffice platform-backoffice cockpitng backofficesearch pcm pcmbackoffice cockpit cockpit-core)
yDEV_NAME="Your name"
yDEV_SURNAME="Your surname"
yDEV_ROLE="Software Developer"
yDEV_MANAGER="Your manager name and surname"

Installation and uninstallation

I also created installation script, which allows to start using the scripts faster. Installation script install dependencies for python script, copies shell script and python script into /usr/local/bin/ directory and .pkup.conf file into home directory. Configuration file needs to be adjusted by the user manually after installtion. Of course, there’s another script, which can be used for uninstallation.


There are python tests for this solution in file, but they’re quite poor right now due to the limited amount of time. They can be a subject of improvements in the future. Note that such scripting solutions rarely have tests because they’re small and created ad-hoc. Nevertheless, I wanted to follow the philosophy from my last blog article and create tests for any kind of software I make.


I’ve spent some time for preparing this stuff, but it was fun and I think it should save me and hopefully my co-workers some amount of time during creating reports every month. In the future, it can be improved by automatic generation of report messages and sending data to the server.

To sum up, preparing reports manually is boring. Generating reports automatically is exciting!

Complete solution described in this article with documentation is available on GitHub:

Lifting quality of a shell script

2016-11-30 Bash, Linux No comments


In release cycle of our team at work, we need to perform so-called system tests. In order to do that, we need to log into Artifactory, search for the latest release package, check if it’s up to date, download it, unzip it, install internal configuration recipe, compile, initialize & run it. Not all of that can be easily automated, but I thought that at least searching & downloading phase can be done from the terminal in a semi-automated way. That’s why I created ydownloader shell script.

Writing a script with unit tests and continuous integration

I’m not an expert in shell scripting, so I also wanted to learn more about it. In addition, I wanted to apply best software development practices in that script. Someone can say that in the case of a simple shell script proper engineering may be a superfluity, but in my opinion, the simplicity of the project is not an excuse for doing it the right way. Especially, if we want to use it in the future. That’s why I divided this script into smaller functions, added command line arguments handling and help for the users. Moreover, I added unit tests with shunit2 (yes, we can write unit tests for the shell scripts) and continuous integration with Travis CI server. In the “Clean Code” book, we can read that code without unit tests is not clean by definition. After dividing script into smaller functions, it was much easier to test it. My script is accepting command line arguments, so I needed to do the following trick to make it testable and include it in the testing script:

if [ "$TEST_MODE" == "" ]

if [ "$TEST_MODE" = false ] ; then
  # parse command line arguments here...
  echo "TEST_MODE enabled"

Then, I could write unit tests:

. ./ydownloader # load script to be tested

testCutLastChars() {
  # given

  # when
  actualValue=$(echo $valueToCut | cutLastChars 3)

  # then
  assertEquals $expectedValue $actualValue

# more tests goes here...
. ./shunit2/shunit2 # load shunit2

There are also other solutions for unit testing like bats, and others. We can choose what we like. We can also use additional tools like shunit2-colorize to make our console output of shunit2 tests look like a rainbow if we are not fans of monochromatic terminal. Moreover, we can use static code analysis tools for shell scripts like shellcheck.

In addition, I prepared simple install script, which allows to install script locally via curl or wget. Of course, project has sufficient documentation in file.

Short recap

It was really nice coding exercise. Now, I feel much more comfortable with shell scripting, but there’s still a lot to learn. I recommend trying applying a similar approach in your scripts if you haven’t done it yet.

If you want to browse complete project, check it out in my repository: