Category Archives: Linux

MariaDB 10.1 and SystemD start issue

I’ve always compiled my software on servers from source. Call me a perfectionist / idiot, whatever, but I do it for several reasons, including making sure I’ve got the latest features and security patches, as well as being able to tailor the software I use on servers so it’s a minimal install of the software, still providing what I require.

Anyways, I’ve just found a quirk with getting MariaDB to starting up. As far as I could tell from the provided systemd .service file, it was being run by the user that owned the data directory I’d configured at compile time, this directory was specified correctly in the my.cnf, and when I tried to run the service manually it worked. but something was blocking it when it came to running it via systemctl.

I was receiving the following messages:

Sep 25 22:37:58 centos7 systemd: Starting MariaDB database server...
Sep 25 22:37:58 centos7 mysqld: 2016-09-25 22:37:58 139993404594304 [Note] /usr/local/bin/mysqld (mysqld 10.1.17-MariaDB) starting as process 7978 ...
Sep 25 22:37:58 centos7 mysqld: 2016-09-25 22:37:58 139993404594304 [Warning] Can't create test file /usr/local/mariadb/data/centos7.lower-test
Sep 25 22:37:58 centos7 mysqld: 2016-09-25 22:37:58 139993404594304 [ERROR] mysqld: File './mysql-bin.index' not found (Errcode: 30 "Read-only file system")
Sep 25 22:37:58 centos7 mysqld: 2016-09-25 22:37:58 139993404594304 [ERROR] Aborting
Sep 25 22:37:58 centos7 systemd: mariadb.service: main process exited, code=exited, status=1/FAILURE
Sep 25 22:37:58 centos7 systemd: Failed to start MariaDB database server.
Sep 25 22:37:58 centos7 systemd: Unit mariadb.service entered failed state.
Sep 25 22:37:58 centos7 systemd: mariadb.service failed.

I was inspecting the provided mariadb.service file, trying to spot anything that might indicate why the user couldn’t write to a folder it owned, when I came across this:

# Prevent writes to /usr, /boot, and /etc
ProtectSystem=full

So there you have it. I’d configured it to have the data directory under the mariadb program directory under /usr. MariaDB was allowing me to configure a data directory at compile time that it knew it wouldn’t be able to boot the daemon and write to (if I used systemd, which, lets face it, is becoming the norm, at-least on RedHat based distros). There’s a lot of guides on the net that as an example have the data directory created under /usr/local/mysql/data. This obviously isn’t an excuse. Using Javascript as an example there’s a lot of bad guides on the internet that produce quick hacks rather than quality long term code.

I actually both understand and agree with the systemd setting especially when you take into account the Filesystem Hierarchy Standard (FHS)

Possible Solutions:

  • Move the data directory to somewhere else. According to the FHS this should lie under /var/lib/mariadb
  • Set the ProtectSystem value to false

I’d recommend the former

CloudFlare and Dynamic DNS

Recently I changed ISPs.  This generally isn’t a big deal. It didn’t cause any difference in speed or  connection reliability, however my old ISP had, regardless of me not ordering one, given me a static IP address.  I’d taken advantage of this and assigned the IP to a DNS record to give me easy to remember access to my files when I’m away from home.  Anyway when I changed ISPs I discovered my new one changed my IP address every few days, which is an unfortunate thing to discover when you’re not at home.

I took a look at my router and found it only supported DynDns, which I then found out that, apart from to some DLink users, didn’t provide provide a free version of their service to the general public, and I don’t have a DLink router.

The next option was a free alternative to DynDns, there were several, and they all required running some kind of linux demon / cron script, which I would then have to alter my existing A record DNS entry to a CNAME to point to the new dynamic DNS record.

Before setting this up I thought I’d look into my existing DNS provider just incase there was any way of updating DNS records on the fly without having to maintain two separate DNS providers. I found that CloudFlare had an API, which I could update records with. The example provided was a Curl example, which made me wonder, surely I could just write a bash script to update the record with via a cron job. I didn’t want to have to install any major scripting language on my minimal Linux distro, excluding any php, perl etc. solution I’d found that worked with CloudFlare.

I checked CloudFlare’s API rate limiting and found it certainly wouldn’t be limiting for what I was trying to do (it’s approximately 1200 requests every 5 mins if you’re wondering). I then found MyExternalIP that could provide my external IP address with a rate limit that was easy to deal with of roughly 1/s, after which it was just a case of writing the bash script.

I worked a script out using Curl requests to get the information I needed including the specific id of the DNS record I was trying to update and then used sed to get the information I needed out each requests response. I made it compatible with the CloudFlareAPI call I was making that way any parameter passed to the call could be passed by a command line argument to the script itself.

The resultant script I added to my crontab to run every half hour.  Everything worked perfectly so I thought  I’d create a gist at github so others could take advantage of this:

#!/bin/sh
#
# CloudFlare Dynamic DNS
#
# Updates CloudFlare records with the current public IP address
#
# Takes the same basic arguments as A/CNAME updates in the CloudFlare v4 API
# https://www.cloudflare.com/docs/client-api.html#s5.2
#
# Use with cron jobs etc.
#
# e.g.
#
# manually run:
# cloudflare_dyn_dns.sh -key 404613183ab3971a2118ae5bf03d63e032f9e -email test@example.com -zone example.com -name extra
#
# cronjob entry to run every 5 minutes:
# */5 * * * * /path/to/cloudflare_dyn_dns.sh -key 404613183ab3971a2118ae5bf03d63e032f9e -email test@example.com -zone example.com -name extra >> /path/to/cloudflare_dyn_dns.log
# 
# will both set the type A DNS record for extra.example.com to the current public IP address for user test@example.com with the provided API key

key=
email=
zone=
zone_id=
type=A
rec_id=
name=
content=
ttl=1
proxied=false

while [ "$1" != "" ]; do
    case $1 in
        -key )     shift
                   key=$1
                   ;;
        -email )   shift
                   email=$1
                   ;;
        -zone )    shift
                   zone=$1
                   ;;
        -zone_id ) shift
                   zone_id=$1
                   ;;
        -type )    shift
                   type=$1
                   ;;
        -rec_id )  shift
                   rec_id=$1
                   ;;
        -name )    shift
                   name=$1
                   ;;
        -content ) shift
                   content=$1
                   ;;
        -ttl )     shift
                   ttl=$1
                   ;;
        -proxied ) shift
                   proxied=$1
                   ;;
        * )        echo "unknown parameter $1"
                   exit 1
    esac
    shift
done

if [ "$content" = "" ]
then
    content=`curl -s http://myexternalip.com/raw`
    if [ "$content" = "" ]
    then
        date
        echo "No IP address to set record value with."
        exit 1
    fi
fi

if [ "$name" = "" ]
then
    echo "You must provide the name of the record you wish to change."
    exit 1
fi

if [ "$zone" = "" ]
then
    echo "You must provide the domain you wish to change."
    exit 1
fi

existing_content=`host -t $type $name.$zone | sed -E 's/.+?\s+([^\s]+)$/\1/'`

if [ "$content" = "$existing_content" ]
then
    echo "Existing record value $existing_content is the same as provided content $content. Exiting."
    exit
fi

if [ "$key" = "" ]
then
    echo "You must provide your user API token."
    exit 1
fi

if [ "$email" = "" ]
then
    echo "You must provide your user email."
    exit 1
fi

# Get the zone id for the entry we're trying to change if it's not provided
if [ "$zone_id" = "" ]
then
    zone_response_json=`curl -X GET "https://api.cloudflare.com/client/v4/zones?name=$zone" -H "X-Auth-Email: $email" -H "X-Auth-Key: $key" -H "Content-Type: application/json"`
    zone_id=`echo $zone_response_json | sed -E "s/.+\"result\":\[\{\"id\":\"([a-f0-9]+)\"[^\}]+$zone.+/\1/g"`
    if [ "$zone_id" = "" ]
    then
        echo "Cloudflare DNS Zone id could not be found, please make sure it exists"
        exit 1
    fi
fi

# Get the record id for the entry we're trying to change if it's not provided
if [ "$rec_id" = "" ]
then
    rec_response_json=`curl -X GET "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records?name=$name.$zone" -H "X-Auth-Email: $email" -H "X-Auth-Key: $key" -H "Content-Type: application/json"`
    rec_id=`echo $rec_response_json | sed -E "s/.+\"result\":\[\{\"id\":\"([a-f0-9]+)\"[^\}]+\"type\":\"$type\"[^\}]+$name.$zone.+/\1/g"`
    if [ "$rec_id" = "" ]
    then
        echo "Cloudflare DNS Record id could not be found, please make sure it exists"
        exit 1
    fi
fi

# Update the DNS record
update_response=`curl -X PUT "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records/$rec_id" -H "X-Auth-Email: $email" -H "X-Auth-Key: $key" -H "Content-Type: application/json" --data "{\"id\":\"$rec_id\",\"type\":\"$type\",\"name\":\"$name.$zone\",\"content\":\"$content\",\"ttl\":\"$ttl\",\"proxied\":$proxied}"`

success_val=`echo $update_response | sed -E "s/.+\"success\":(true|false).+/\1/g"`
if [ "$success_val" = "true" ]
then
    echo "Record Updated."
else
    echo "Record update failed."
    exit 1
fi

Firewalld, firewall-cmd and Fail2Ban

Basically I’ve been looking at upgrading a web-server to the latest version of Fedora 19, or when it’s released later this year CentOS 7.0 (providing it’s easy for them when RHEL 7.0 is released), however knowing that iptables is now becoming redundant in favour of firewalld in Fedora I started looking at updating my web-server install script to work with firewalld.  Knowing part of that is Fail2Ban and that uses iptables my first port of call was finding a way of getting these two working together.

My first obvious search for “firewalld fail2ban” returned nothing helpful whatsoever, just people wanting a conf file to get it working with no actually helpful response, however once I found that firewalld uses firewall-cmd on the command line to control the rules I searched for that. This turned up a current bug posted on RedHat’s BugZilla: https://bugzilla.redhat.com/show_bug.cgi?id=979622 , where it turns out a very helpful soul, Edgar Hoch, has created an action.d conf file to get it all working: https://bugzilla.redhat.com/attachment.cgi?id=791126

Fedora 19, GNOME 3 and nVidia graphics

After upgrading to the latest version of Fedora a few months ago I was terribly un-impressed. The box in question had been upgraded every 6 months (-ish, thanks to Fedora 18) since Fedora 14 and I’d never had any issues, but then came Fedora 19.

To be fair it wasn’t Fedora’s fault per say, it was GNOME 3 and the open source nVidia graphics drivers. The desktop looked ok when you booted the box, but if you tried to use the Activities section, none of the transparency worked, and a lot of the Favourite icons in the dock had a luminous green behind them when you hovered over them.  What was worse was trying to launch a non-favourite application, click to do that and you could see the first 6 frequently used ones, but no others, and none under the “all” tab. This obviously made the whole experience pretty much unusable.

I went through the obvious investigations, straight away looking for some better nVidia graphics drivers.  I didn’t expect to find any official nvidia drivers after Linus’ hilarious rant last year. However, it turned out there was. I first tried downloading them from nVidia but their installers were less than helpful, and none wanted to install on my system regardless of fulfilling their dependencies. I then tried looking elsewhere and remembered the trusty basic linux guide site If !1 0. I found a guide on there for Fedora 18, and adapted it for Fedora 19, but unfortunately that wouldn’t work due to a mass amount of package conflicts.  I’d been meaning to wipe the system for a while and start again, so backed up the /etc/ folder to another drive, wiped the partitions, then installed Fedora 19 and used the guide again and all was fine and dandy.  The boot screen is the basic plymouth one rather than the more graphical splash one, but apart from that everything works and I don’t have awful un-usable graphics anymore.

If you want the latest guide, which is for Fedora 19, go here: http://www.if-not-true-then-false.com/2013/fedora-19-nvidia-guide/

Fedora 18 – non-graphical boot and ifconfig

After downloading Fedora 18 and creating a minimal install virtual machine I went straight to doing the default I always do.  As I was creating a test server I need to be able to see what’s happening during boot so I disable the graphical boot.

Previously (at-least up to Fedora 15) I’d used the plymouth commands to switch the graphical boot to details which had worked.

[[email protected] ~]# plymouth-set-default-plugin details
[[email protected] ~]# /usr/libexec/plymouth/plymouth-update-initrd

However, when I tried this with Fedora 18, nope, not working, the community to the rescue. Another individual  (Nigel Smith) had had the same issue and found the solution in editing the default grub configuration file. So if you need the same visit here:

http://nwsmith.blogspot.co.uk/2012/10/customizing-grub2-boot-options-in.html

While I was at it I also noticed that ifconfig produced a “Command not found”. Erm … wha? Where did that go? Isn’t that like a basic linux command? Turns out in Fedora 18 it’s been removed due to the alternative and replacement iproute package being in place for “many years”. So you’ve now got 2 choices:

1. use the ip command:

ip addr

2. install the net-tools package:

yum -y install net-tools

Installing MongoDB on a Linux distro using SystemD instead of inittab

While trying to get MongoDB working on a Fedora 15 test server I found that there was no way of installing the latest version with SystemD control due to a pre-existing known bug.

To Mongo’s credit there is a file provided in the source under rpm/init.d-mongod, however when it comes to this being used by SystemD, well it just errors.

This was frustrating, but as I was doing it on a VM, I did what any logical person would do. Installed Mongo from the package manager, took a copy of the required SystemD files, reverted to a snapshot, then installed the latest version of Mongo from source.

After this I adapted the mongod.service file to use the correct locations, ensured all the required directories, files and users were present on the system then started the service et voila, working latest version of MongoDB on Fedora 15 with SystemD

the two required files are:
/lib/systemd/system/mongod.service

[Unit]
Description=High-performance, schema-free document-oriented database
After=syslog.target network.target
 
[Service]
Type=forking
User=mongod
Group=mongod
PIDFile=/var/run/mongodb/mongod.pid
EnvironmentFile=/etc/sysconfig/mongod
ExecStart=/usr/local/bin/mongod $OPTIONS run
 
[Install]
WantedBy=multi-user.target

/etc/sysconfig/mongod

Or you can just use the RedHat distro based install script I created:

#!/bin/sh

# MongoDB Version
MONGODB_VER='2.2.2'

# Get all the dependencies up to date
yum -y update
yum -y install scons gcc-c++ glibc-devel

# Get the source
cd /usr/local/src/
wget http://downloads.mongodb.org/src/mongodb-src-r$MONGODB_VER.tar.gz
tar xfz mongodb-src-r$MONGODB_VER.tar.gz
cd mongodb-src-r$MONGODB_VER

# Compile and Install
scons all
scons install

# Create the SystemD dependant files
echo '[Unit]
Description=High-performance, schema-free document-oriented database
After=syslog.target network.target
 
[Service]
Type=forking
User=mongod
Group=mongod
PIDFile=/var/run/mongodb/mongod.pid
EnvironmentFile=/etc/sysconfig/mongod
ExecStart=/usr/local/bin/mongod $OPTIONS run
 
[Install]
WantedBy=multi-user.target' > /lib/systemd/system/mongod.service

echo 'OPTIONS="--quiet -f /etc/mongod.conf"' > /etc/sysconfig/mongod

# Setup the required user and group
useradd -r -U mongod

# Setup the required directories
mkdir -p /var/run/mongodb/
mkdir -p /var/log/mongo/
mkdir -p /var/lib/mongo/
chown mongod:mongod /var/run/mongodb/
chown mongod:mongod /var/log/mongo/
chown mongod:mongod /var/lib/mongo/
chmod 0755 /var/log/mongo/
chmod 0755 /var/run/mongodb/
chmod 0755 /var/lib/mongo

# Start the new service and enable it on boot
systemctl --system daemon-reload
systemctl start mongod.service
systemctl enable mongod.service

Installing the default Linux Kernel on a Linode CentOS 6 box

While creating the new web-server for my employers, to replace a Fedora 10 box which gets no security updates, I needed to compile some software from source, meaning I needed the kernel sources.

Since I couldn’t easily obtain these I needed to install the Kernel provided by the distribution rather than the more recent kernel provided by Linode themselves.

The Linode Library provided a way of doing this for CentOS 5 but not for CentOS 6, thus I adapted the provided script for v5 into one that works with CentOS 6 et voila, distro provided kernel.

Here’s the full source available as a gist on github:

### Starting from a fresh CentOS 6 or newer Linode
### Enable the native kernel to boot from pvgrub
### It will autoconfigure itself with each yum update.
### This is adapted from a previous script for CentOS 5.5 found here:
### http://www.linode.com/docs/assets/542-centos5-native-kernel-selinux-enforcing.sh
### Provided via the linode wiki
### https://www.linode.com/docs/tools-reference/custom-kernels-distros/run-a-distributionsupplied-kernel-with-pvgrub#centos-5
### Provided without warranty, although since it should only be run
### on first box build if your box gets broken simply rebuild it

mkdir /boot/grub/

DISTRO_PLATFORM=`uname -p`
AWK_VERSION_MATCH="{if(\$1==\"kernel.$DISTRO_PLATFORM\") print \$2}"
KERNEL_VERSION=`yum -q list kernel | awk "$AWK_VERSION_MATCH"`

### Write template grub.conf
cat > /boot/grub/grub.conf << EOF
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initramfs paths are relative to /boot/, eg.
#          root (hd0)
#          kernel /boot/vmlinuz-version ro root=/dev/xvda
#          initrd /boot/initramfs-version.img
#boot=/dev/xvda
default=0
timeout=3
title CentOS ($KERNEL_VERSION.$DISTRO_PLATFORM)
        root (hd0)
        kernel /boot/vmlinuz-$KERNEL_VERSION.$DISTRO_PLATFORM root=/dev/xvda
        initrd /boot/initramfs-$KERNEL_VERSION.$DISTRO_PLATFORM.img
EOF

ln -s /boot/grub/grub.conf /boot/grub/menu.lst
yum -y install kernel
if [ $? -ne 0 ]; then
    echo "ERROR aborting..."
    exit 1
fi