Running Cron Jobs on Cloud Foundry

Many cloud applications need to run some sort of scheduled tasks for chores like processing statistics or doing internal housekeeping. There are two different strategies to implement scheduled tasks to cloud applications running on Cloud Foundry. You can either build scheduling and tasks into the application itself or schedule and run the task in separate containers

Some application frameworks like Spring include some built-in scheduling support. However, this scheduling support does not include a distributed coordination mechanism. This means that an application horizontally scaled to multiple instances will each run the scheduled task individually. Depending on the nature of the task, this may cause observable side effects like sending emails to your customers to be repeated.

It's thus preferable to have a central entity for scheduling. You could of course use e.g. a Java Spring App that needs approximately 1G of RAM to do that for you, but that would be very wasteful. Instead, we can build a simple cron scheduler that runs on 16 MB of RAM to get reliable task scheduling for just a few cents per month.

The task scheduler can then execute arbitrary scripts or code, for example to:

• invoke an https endpoint on your application to perform the task
• queue a message on RabbitMQ for processing by a worker
• trigger execution of the job in a seperate Cloud Foundry Task Container

meshcloud's cf-cron scheduler

Our sample repository demonstrates how to run scheduled tasks on Cloud Foundry with a very small footprint (8 to 16 MB RAM) using a traditional crontab. Traditional cron daemons need to run as root and have opinionated defaults for logging and error notifications. This makes them unsuitable for running in a containerized environment like Cloud Foundry. Instead of a system cron daemon we're thus using supercronic to run our cron tab.

How it works

This application is built using the binary buildpack and executes supercronic on the crontab file. The crontabfile specifies all your cron jobs. To add additional jobs, simply add a new line which specifies a schedule and command to the crontab.

Note: By default, supercronic will log all output to stderr so we redirect that to stdout in our cf manifest.

You can also include additional scripts and binaries to execute more complex actions. This example allows you to install apt and debian packages to use in your cronjobs. You can specify these packages in apt.yml and they will be installed during staging by apt-buildpack
courtesy of the magic multi-buildpack.

After cf pushing this sample app to Cloud Foundry, you can see that it happily executes the jobs from the crontab in the log output:

2018-03-05T10:59:00.00+0100 [APP/PROC/WEB/0] OUT time="2018-03-05T09:59:00Z" level=info msg=starting iteration=237 job.command="echo "hello world, every 2 seconds"" job.position=1 job.schedule="*/2 * * * * * *"
2018-03-05T10:59:00.00+0100 [APP/PROC/WEB/0] OUT time="2018-03-05T09:59:00Z" level=info msg="hello world, every 2 seconds" channel=stdout iteration=237 job.command="echo "hello world, every 2 seconds"" job.position=1 job.schedule="*/2 * * * * * *"
2018-03-05T10:59:00.00+0100 [APP/PROC/WEB/0] OUT time="2018-03-05T09:59:00Z" level=info msg="job succeeded" iteration=237 job.command="echo "hello world, every 2 seconds"" job.position=1 job.schedule="*/2 * * * * * *"
2018-03-05T10:59:00.05+0100 [APP/PROC/WEB/0] OUT time="2018-03-05T09:59:00Z" level=info msg="cf version 6.34.1+bbdf81482.2018-01-17" channel=stdout iteration=7 job.command="cf --version" job.position=0 job.schedule="*/1 * * * *"
2018-03-05T10:59:00.05+0100 [APP/PROC/WEB/0] OUT time="2018-03-05T09:59:00Z" level=info msg="job succeeded" iteration=7 job.command="cf --version" job.position=0 job.schedule="*/1 * * * *"

Scheduling Cloud Foundry Tasks

While the cron container here is designed to be small and lightweight, you may want to use it to trigger more resource intensive tasks and processes. When a simple curl to an http endpoint is not enough to kick off such a task on your existing app, Cloud Foundry Tasks are a great solution to run these processes.

This sample repository thus includes instructions to install the cf cli tool which you can use to trigger such a task using a meshcloud Service User.

by Johannes Rudolph

Deploying Meteor Apps on Cloud Foundry

This repository demonstrates different ways to deploy Meteor applications on Cloud Foundry.
There are at least three different ways to deploy Meteor JS applications on Cloud Foundry:

• using a specialized meteor-js buildpack
• using the general node-js buildpack
• using your own docker container image

While a specialized meteor buildpack like meteor-buildpack-horse or cf-meteor-buildpack is a useful option, recent versions of meteor support simplified deployment on servers that provide just a node.js runtime environment. For this demonstration repository, we're focusing on this approach because the node.js buildpack is more widely used and better maintained.

The application we're going to deploy is located in the ./try-meteor folder of this repository. We also assume you have provisioned a Mongo DB service on your space. At meshcloud's Cloud Foundry, you can create a dedicated service instance suitable for use with meteor like this:

cf create-service MongoDB M meteor-mongo

Using the node.js Buildpack

Build a meteor app bundle

On your local machine with the meteor cli installed, build a distributable package into the deploy-buildpack folder:

cd try-meteor && meteor build ../deploy-buildpack/. --server-only --architecture os.linux.x86_64

Building generates a try-meteor.tar.gz file with the meteor application bundled as a plain node.js application, with some helfpful instructions in its README.md file:

This is a Meteor application bundle. It has only one external dependency:
Node.js v8.9.3. To run the application:

$ (cd programs/server && npm install)
$ export MONGO_URL='mongodb://user:password@host:port/databasename'
$ export ROOT_URL='http://example.com'
$ export MAIL_URL='smtp://user:password@mailhost:port/'
$ node main.js

Use the PORT environment variable to set the port where the
application will listen. The default is 80, but that will require
root on most systems.

node.js buildpack wrapper

To deploy this tar.gz file on Cloud Foundry with the node.js buildpack we need to:

1. upload and unpack the tar.gz bundle
2. run npm install on the extracted bundle
3. set the correct environment variables using a launcher js script

We can easily achieve that through a custom package.json that uses npm's postinstall and start script to execute these actions. You can find the package.json and all required files for the deployment in the ./deploy-buildpack folder.

Note: at the time of writing the bundles generated by meteor 1.6.0.1 machine lack the meteor-deque dependency so we just explicitly add that by hand.

{
"name": "try-meteor",
"private": true,
"scripts": {
"start": "node launcher.js",
"postinstall": "tar -xf try-meteor.tar.gz && (cd bundle/programs/server && npm install)"
},
"engines": {
"node": "8.9.3"
},
"dependencies": {
"meteor-deque": "~2.1.0",
"cfenv": "1.0.4"
}
}

Have a look at the launcher.js file if you want to change service names etc.
The final bit that we need is a Cloud Foundry Manifest file to describe our application:

---
applications:
- name: try-meteor-app
memory: 512M
instances: 1
buildpack: https://github.com/cloudfoundry/nodejs-buildpack
services:
- meteor-mongo

We're all set now, a simple cf push and your app should be up and running on the cloud.

Using a Docker Container

The next option is to use a docker-based deployment of the application. This requires that we build our own docker image of the application and publish it to a docker registry.

You can find the code for the docker-based deployment of our sample application in the MARKDOWN_HASHd786272e16707f3074a294d21f8b83a0MARKDOWNHASH folder. The docker image used in this example is based on the [node:8-alpine](https://hub.docker.com//node/) base image. However, before we can build our container we need to build our meteor application and extract it:

cd try-meteor && meteor build ../deploy-docker/. --server-only --architecture os.linux.x86_64
cd ../deploy-docker && tar -xf try-meteor.tar.gz && rm try-meteor.tar.gz

The docker deployment demonstrated in this repository also uses the same launcher.js script introduced above to automatically initialize meteor environment variables from their Cloud Foundry counterparts. With that out of the way, let's build and push the docker image

docker build -t meshcloud/meteor-cf-example .
docker push meshcloud/meteor-cf-example:latest

With the container available in a docker registry, we can push it to Cloud Foundry by specifying the docker image in the manifest:

---
applications:
- name: try-meteor-app-docker
memory: 512M
instances: 1
docker:
image: meshcloud/meteor-cf-example
services:
- meteor-mongo

Now all that's left to do is a simple cf push and your app should be up and running on Cloud Foundry in no time.

iDevBook01:deploy-docker jr (master *) $ cf push
Pushing from manifest to org meshcloud-demo / space aproject as c9f7d64c-404d-4b29-b719-b2359f6c8157...
Using manifest file /Users/jr/dev/demo/meteor/deploy-docker/manifest.yml
Getting app info...
Updating app with these attributes...
name: try-meteor-app-docker
docker image: meshcloud/meteor-cf-example
command: node launcher.js
disk quota: 1G
health check type: port
instances: 1
memory: 512M
stack: cflinuxfs2
services:
meteor-mongo
routes:
try-meteor-app-docker.cf.eu-de-darz.msh.host

Updating app try-meteor-app-docker...
Mapping routes...

Stopping app...

Waiting for app to start...

name: try-meteor-app-docker
requested state: started
instances: 1/1
usage: 512M x 1 instances
routes: try-meteor-app-docker.cf.eu-de-darz.msh.host
last uploaded: Mon 12 Feb 10:25:29 CET 2018
stack: cflinuxfs2
docker image: meshcloud/meteor-cf-example
start command: node launcher.js

state since cpu memory disk details
#0 running 2018-02-12T10:12:23Z 0.1% 38.2M of 512M 1.3M of 1G

iDevBook01:deploy-docker jr (master *) $

by Johannes Rudolph