Unlike a lot of Linux people, I'm not a knee-jerk hater ofsystemd. My "salaried UNIX" background, up through 2008, was primarily with OSes like Solaris and AIX. With Solaris, in particular, I was used tosytemd-type init-systems due to SMF.

That said, making the switch from RHEL and CentOS 6 to RHEL and CentOS 7 hasn't been without its issues. The change fromupstart tosystemd is a lot more dramatic than fromSysV-init toupstart.

Much of the pain withsystemd comes with COTS software originally written to work on EL6. Some vendors really only due fairly cursory testing before saying something is EL7 compatible. Many — especially earlier in the EL 7 lifecycle — didn't bother creatingsystemd services at all. They simply relied onsystemd-sysv-generator utility to do the dirty work for them.

While thesystemd-sysv-generator utility does a fairly decent job, one of the places it can fall down is if the legacy-init script (files hosted in/etc/rc.d/init.d) is actually a symbolic link to someplace else in the filesystem. Even then, it's not super much a problem if "someplace else" is still within the "/" filesystem. However, if your SOPs include "segregate OS and application onto different filesystems", then "someplace else" can very much be a problem — when "someplace else" is on a different filesystem from "/".

Recently, I was asked to automate the installation of some COTS software with the "it works on EL6 so it ought to work on EL7" type of "compatibility". Not only did the software not come withsystemd service files, its legacy-init files linked out to software installed in/opt. Our shop's SOPs are of the "applications on their own filesystems" variety. Thus, the/opt/<APPLICATION> directory is actually its own filesystem hosted on its own storage device. After doing the installation, I'd reboot the system. ...And when the system came back, even though there was a boot script in/etc/rc.d/init.d, the service wasn't starting. Poring over the logs, I eventually found:

 systemd-sysv-generator[NNN]: stat() failed on /etc/rc.d/init.d/<script\_name> No such file or directory

This struck me odd given that the link and its destination very muchdid exist.

Turns out,systemd invokes thesystemd-sysv-generator utility very early in the system-initialization proces. It invokes itso early, in fact, that the /opt/ filesystem has yet to be mounted when it runs. Thus,when it's looking to do the conversion the file the sym-link points to actually doesnot yet exist.

My first thought was, "screw it: I'll just write asystemd service file for the stupid application." Unfortunately, the application's starter was kind of a rats nest of suck and fail; complete and utter lossage. Trying to invoke it from directly via asystemd service definition resulted in the application's packaged controller-process not knowing where to find a stupid number of its sub-components. Brittle. So, I searched for other alternatives...

Eventually, my searches led me to both the nugget about whensystemd invokes thesystemd-sysv-generator utility andhow to overcome the "sym-link to a yet-to-be-mounted-filesystem" problem. Undersystemd-enabled systems, there's a new-with-systemd mount-option you can place in/etc/fstab —x-initrd.mount. You also need to make sure that your filesystem'sfs_passno is set to0 ...and if your filesystem lives on an LVM2 volume, you need to update your GRUB2 config to ensure that the LVM gets onlined prior tosystemd invoking thesystemd-sysv-generator utility. Fugly.

At any rate, once I implemented this fix, thesystemd-sysv-generator utility became happy with the sym-linked legacy-init script ...And my vendor's crappy application was happy to restart on reboot.

Given that I'm deploying on AWS, I was able to accommodate setting these fstab options by doing:

 mounts:   - ["/dev/nvme1n1", "/opt/<application> , "auto", "defaults,x-initrd.mount", "0", "0"]

Within mycloud-init declaration-block. This should work in any context that allows you to usecloud-init.

I wish I could say that this was the worst problem I've run into with this particular application. But, really, this application is an all around steaming pile of technology.