Cron tasks on load balanced identical web servers

A common setup for web applications is multiple identical web servers behind a load balancer. The benefit of having such a setup is horizontal scaling. When all the web servers are exactly the same, there is no single point of failure. If there is an issue with any one of the servers, the problematic server can be quickly replaced with a new healthy instance.

A challenge that comes up with this setup is the cron configuration. Since all the servers are identical, the same cron task will execute from multiple machines at the same time when we only want it to execute once.

One possible solution is to configure only one server to execute the cron. This solution however makes one of the servers different from the others and increases the risk that this server will take down the entire application with it.

A better solution would be to use an application-wide central lock. The lock is created using redis and shared by all the web servers.

When it's time to execute the cron, the server that will be the fastest by a fraction of a millisecond will create a temporary lock using redis and then execute the cron job. All the following servers will see that the lock already exists and exit immediately.

The "setnx" command creates a key if it does not already exist and returns 1 if a key was set and 0 if it was not. We will use this command to check if a lock was already created and create it if not. The "expire" command sets a timeout on a key. After the timeout has expired, the key will automatically be deleted. We will use this command to set the lock to expire after a short period of time.

MutexLock is a php library that implements this solution.

Setting up the library:

MutexLock\lock::init([
    // monolog logger:
    'logger' => $log,
    // redis connection:
    'host'   => '127.0.0.1',
    'port'   => '6379',
]);

Then we check if the lock was created and create it if it wasn't. If the lock was already created, the set function will return true and we'll exit.

if (!MutexLock\Lock::set(LOCK_KEY, LOCK_TIME)) {
    return;
}