Custom Service Control

Custom Service Control (CSC) is a tool that enables you to manage your own services on a flex server. Custom services include backends for web applications or daemons with specific functions. Using Hostpoint’s CSC framework, you can set up your applications so that in the event of a server reboot they are automatically stopped, carefully shut down, and then restarted.

Requirements

CSC is available for Managed Flex Servers M and higher.

What functions does Custom Service Control offer?

• They can function as a daemon, starting and stopping your applications. This could be a node.js application that you integrate via an NGINX proxy function.
• CSC can detect and restart crashed applications. Using Event Listener you can execute actions if this happens, e.g. send an e-mail.
• CSC can manage logging and log rotation for your application, ensuring that your application logs are always organized.
• Prior to a server reboot (e.g. for system updates), CSC can safely stop your applications and custom services and then restart them once your server is back up and running again.

Overview of Custom Service Control framework

CSC has two components:

• The hpservices tool, which manages the whole framework and activates and controls the supervisor service.
• The supervisor service Supervisord, which allows the actual monitoring and control of the processes. The popular Supervisord is used for the supervisor service.

Brief overview of the hpservices utility

The hpservices utility is the control center for CSC administration. This tool activates, starts, and stops Supervisord and helps to set up new custom services.

Overview of key functions:

hpservices supervisord start

Starts Supervisord. Your server will then regularly check (approximately every 60 seconds) that Supervisord is running. If it is not, it will be automatically started. When you start Supervisord for the first time, the basic configuration is created and Supervisord is activated. This state is then saved. After a server reboot (e.g. after an update), Supervisord is restarted using your defined configuration.

hpservices supervisord stop

Stops Supervisord and all its applications. This state is also saved until you restart Supervisord.

hpservices supervisord status

Displays whether Supervisord is running or not.

hpservices supervisord restart

Stops and restarts Supervisord. The configuration is then reloaded from the start. Any applications managed by Supervisord are also stopped and restarted. This way, you can test whether your configuration achieves the desired result after a reboot.

hpservices supervisord add example1

Creates a skeleton file using the relevant records and a template for the Supervisord configuration for the example1 service.

hpservices supervisord remove example2

Removes the records and the Supervisord configuration for the example2 service. ​

hpservices supervisord list

Lists all applications that you have added using hpservices supervisord add.​

Working with CSC

Just as your application runs as a “custom service” on your server, CSC also runs on your server and is controlled, configured, and managed from there. To use CSC log onto your server with ssh.

Starting Supervisord

You need to activate Supervisord on your server so that you can start, stop, and manage your own services using CSC. When it is started for the first time, the relevant records and an appropriate configuration are created.

hpservices supervisord start
supervisord successfully started

Use the status subcommand to check whether your Supervisord is running:

hpservices supervisord status
supervisord is running as pid 1337

If your Supervisord crashes or is manually stopped (killed), CSC automatically restarts Supervisord using your defined configuration, and your applications. If you want to stop the Supervisord function, you can either deactivate the relevant applications using supervisorctl (works until the next Supervisord reboot) or stop Supervisord using the hpservices utility (works for all applications until you restart it using hpservices).

How to stop Supervisord

To deactivate the CSC’s and Supervisord’s functions, stop them.

hpservices supervisord stop
supervisord has been stopped

This will stop Supervisord and any applications it has started. This state is then saved. This also means that after a reboot of your server, Supervisord and any configured applications will not start automatically.

How to add a new service

Using the example of a “hellojs” application, we will show you how to set up an application as a custom service.

If you want to run your own application using CSC, you have to add the application to Supervisord’s configuration. The Supervisord’s configuration files can be found in the ~/.services/supervisord/ folder in your home folder (see also the “Appendix: Directory and File Structure” section at the end of these instructions). The subcommand hpservices supervisord add followed by your chosen application name creates a skeleton file from the relevant directories and the service configuration for Supervisord. For our hellojs example application, it looks like this:

hpservices supervisord add hellojs
successfully created dir for hellojs: /home/username/.services/supervisord/hellojs
successfully created dir for hellojs: /home/username/.services/supervisord/hellojs/log
supervisord service config file written: /home/username/.services/supervisord/hellojs/service.conf

Installing the application

Our best practice recommendation is to install your application in the subfolder ~/app/applicationname/. This creates a clear structure in your home folder and ensures that your application’s source code is located outside your web server’s document root. It is also possible to install your application in a different location in your home folder.

cd app/hellojs

var express = require('express');
var app = express();
				
app.get('*', function (req, res) {
  res.send('Hello World!\n'.concat(process.argv[2],'\n'));
  console.log(req.method,req.url.concat('\n'),req.headers);
});
				
app.listen(process.argv[2], function () {
  console.log('Example app listening on port '.concat(process.argv[2],'!'));
});

ls
start.js
npm install express
ls
log     node_modules     package-lock.json     start.js

node start.js 1234
Example app listening on port 1234!
^C

Configuration as a Supervisord service

The hpservices utility has created a subfolder for our hello.js application in Supervisord’s configuration directory: ~/.services/supervisord/hellojs/. This contains a commented-out example configuration as a starting point for you and a “log” folder where your service’s standard output (stdout) and standard error output (stderr) log files can be saved (see also the detailed description of logging options below):

ls -l
total 8
drwxr-xr-x 2 username username 4096 Jan 22 17:45 log
-rw-r--r-- 1 username username 882 Jan 22 17:45 service.conf

In the service.conf file you can configure the new service, the hellojs application. It contains some basic suggestions that you can accept, modify, or expand. Lines beginning with a semicolon (“;”) are comments and are ignored. To start a service, Supervisord needs a minimum configuration consisting of a program block and a command:

[program:hellojs]
command=/usr/local/bin/node %(ENV_HOME)s/app/hellojs/start.js 1234 ; the program (can take args)
;directory=/home/username/app/hellojs/ ; directory to cwd to before exec (default no cwd)
;autostart=true ; start application at supervisord start (default: true)
;stopwaitsecs=10 ; max num secs to wait before SIGKILL (default 10)
;stdout_logfile=/home/username/.services/supervisord/hellojs/log/default.log
;stdout_logfile_maxbytes=1MB ; filesize at which to rotate logfiles (default ist 50MB)
;stdout_logfile_backups=10 ; number of stdout logfile backups (0 means none, default 10)
;stderr_logfile=/home/username/.services/supervisord/hellojs/log/default.err
;stderr_logfile_maxbytes=1MB ; filesize at which to rotate logfiles (default is 50MB)
;stderr_logfile_backups=10 ; number of stderr logfile backups (0 means none, default 10)

Service options

[program]

Name of service. This is used to manage the service using the supervisorctl command.

command

The command that should be used to start the service. In our example, we enter the interpreter “usr/local/bin/node”, our program “%(ENV_HOME)s/app/hellojs/start.js” along with a single argument, the port “1234”, expected by our hellojs app.

directory

Service's working directory. Before starting the application, Supervisord “switches” to this directory.

autostart

If you set this value to “false”, the program will not be automatically started when Supervisord is started. This can be useful when developing services.

stopwaitsecs

Defines the maximum time that Supervisord waits before stopping the application during the stop routine. Once this time has elapsed, it forcibly ends the application (KILL).
Warning: Only increase this value if absolutely necessary. Doing so increases your server’s downtime during maintenance work! The maximum possible waiting time is 120 seconds.

Logging options

If your program issues outputs to the standard output (stdout) or standard error output (stderr), you can write them to a log file. This is a very elegant logging method for your application. You can configure Supervisord so that it manages the accounting and rotation of these logs in a clearly organized way. This ensures that you can always find all your log files. The suggested, commented-out lines include appropriate default settings that you can simply apply if you want to.

stdout_logfile

Defines the path to the standard output log file

stdout_logfile_maxbytes

The maximum size of a standard output log file. If this size is reached, the current log file is moved to a backup file and a new one started.

stdout_logfile_backups

The number of backup standard output log files to be stored.

stderr_logfile

Defines the path to the standard error output log file

stderr_logfile_maxbytes

The maximum size of a standard error output log file. If this size is reached, the current log file is moved to a backup file and a new one started.

stderr_logfile_backups

The number of backup standard error output log files to be stored.

You can find details and other service configuration options in the official Supervisord documentation.

Starting the new service

When you have installed and tested your application, and configured the corresponding service in Supervisord, you can instruct your Supervisord to load the new configuration. There are different ways to do this.

supervisord restart

The simplest option is to completely restart Supervisord. However, this will also restart any applications that it manages. The advantage is that this enables you test at the same time whether your CSC configuration is correct and functioning.

hpservices supervisord restart
supervisord has been stopped
supervisord successfully started

Alternatively, you can read the new configuration manually using supervisorctl and activate the new service. There are two ways of doing this:

supervisorctl update

The update subcommand rereads and reapplies the Supervisord configuration. A new service is then automatically started:

supervisorctl
hello-world                   RUNNING     pid 11106, uptime 21:47:53
supervisor> update
hellojs: added process group
supervisor> status
hello-world                   RUNNING     pid 11106, uptime 21:47:59
hellojs                       RUNNING     pid 11105, uptime 21:47:59
supervisor>

supervisorctl reread & add

A more cautious method of adding it is using the reread and add subcommands. This enables you to see in advance which changes will be applied by an update, using add you can perform these selectively.

1. Reload the configuration. Supervisord detects that a new service is available and communicates this:
   

   supervisorctl reread
   hellojs: available

   However, the new application is not yet active:

   supervisorctl status
   hello-moon                    RUNNING    pid 2408, uptime 0:11:36
   supervisorctl avail
   hello-moon                    in use     auto      999:999
   hellojs                       avail      auto      999:999

2. Add our "hellojs" to the applications which are running:
   

   supervisorctl add hellojs
   hellojs: added process group

   The new service is added and automatically started:

   supervisorctl status
   hello-moon                    RUNNING     pid 2408,   uptime 0:11:36
   hellojs                       RUNNING     pid 234269, uptime 0:0:07

Best Practices

To ensure that your application runs as reliably and smoothly as possible, we recommend that you observe the following best practices:

• Log into your application on stdout and stderr and let Supervisord write the log files and manage their rotation and organization.
• If your application writes to a log file, make sure that it is written somewhere that makes it easy to manage your log files. For example, this could be ~/app/appname/log/. Set up an automatic routine that regularly rotates and erases your log files. By actively managing your logs, you can identify any problems and prevent your web space from filling up with log notifications until it eventually crashes.
• After making any changes, check that your Supervisord configuration and application both work as intended. You can do this by restarting the whole CSC framework with hpservices supervisord restart, for example.
• Set up an E-Mail account for technical messages. You will then be informed if your Supervisord could not be activated after a system start due to an error in your configuration.

Working with the services: supervisorctl

The application that you set up as a service in Supervisord is easy to control using the supervisorctl utility.

When you start supervisorctl via ssh, it opens as an interactive shell where you can invoke different subcommands directly.

supervisorctl
hello-world                   RUNNING    pid 11106, uptime 0:00:03
hellojs                       RUNNING    pid 11105, uptime 0:00:03
hellojs2                      STOPPED    Not started
supervisor>

The interactive supervisorctl shell supports a simple form of command-line completion using the tab key (e.g. press the letter “h” followed by the tab key or the letters “st” and then the tab key twice). Some assistance is also built in:

However, you can also execute subcommands directly without using the interactive mode:

Detailed instructions on using supervisorctl and its options can be found in the official supervisorctl documentation.

Example

In the following section, we would like to invite you on a short tour. We would like to show you some ways you can develop your applications, how supervisorctl can help you, and how to use this tool.

In our example, we will continue working on our previously mentioned hello.js. To ensure a smooth test, we have made a copy and set up a new service: hellojs2. Meanwhile, the original is still in actively running. First, we would like to start the unaltered hellojs2 application exactly the same as our hellojs application:

supervisor> start hellojs2
hellojs2: ERROR (spawn error)

Oh! Unfortunately, there was an error. This is also clear when executing the status subcommand:

supervisor> status
hello-world                   RUNNING    pid 11106, uptime 0:20:52
hellojs                       RUNNING    pid 11105, uptime 0:20:52
hellojs2                      FATAL      Exited too quickly (process log may have details)

In the stderr log file written by Supervisord for us, we can see the reason for the error:

tail -18 .services/supervisord/hello.js2/log/default.err
Error: listen EADDRINUSE: address already in use :::1234
    at Server.setupListenHandle [as _listen2] (net.js:1290:14)
    at listenInCluster (net.js:1338:12)
    at Server.listen (net.js:1425:7)
    at Function.listen
(/home/username/app/hello.js2/node_modules/express/lib/application.js:618:24)
    at Object.<anonymous> (/home/username/app/hello.js2/index.js:14:5)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)
Emitted 'error' event at:
    at emitErrorNT (net.js:1317:8)
    at process._tickCallback (internal/process/next_tick.js:63:19)
    at Function.Module.runMain (internal/modules/cjs/loader.js:745:11)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)

Correct! Remember: we previously used port 1234 for the productive hellojs app. Only one listening socket can be opened on a TCP port of a given IP address, so obviously this caused an error: Port is already occupied. To fix the problem, we modify the command line in the Supervisord configuration for hellojs2 and select a different, unused port:

command=/usr/local/bin/node /home/username/app/hellojs2/index.js 1235 ; the program (relative uses PATH, can take args)

We can then read and activate the new configuration:

supervisor> update
hellojs2: stopped
hellojs2: updated process group
supervisor> status
hello-world                   RUNNING    pid 11106, uptime 0:20:02
hellojs                       RUNNING    pid 11105, uptime 0:20:02
hellojs2                      STOPPED    Not started

The next start attempt is successful, and we are ready to continue working on hellojs2:

supervisor> start hellojs2
hellojs2: started
supervisor> status
hello-world                   RUNNING    pid 11106, uptime 0:20:52
hellojs                       RUNNING    pid 11105, uptime 0:20:52
hellojs2                      RUNNING    pid 21305, uptime 0:00:04

We can now restart our new application when we need to test any changes:

supervisor> restart hellojs2
hellojs2: stopped
hellojs2: started

Eventually, even the most energetic programmer becomes tired. We can then stop our hellojs2 application until the next development session.

supervisor> stop hellojs2
hellojs2: stopped

And it will remain in this state since we set autostart to “false” in the service configuration for hellojs2.

Removing a service

If you want to permanently remove (delete) your own applications, you can do this with the hpservices tool and supervisorctl. This deletes both the folder structure containing the relevant Supervisord configuration (~/.services/supervisord/example) and the folder containing the application.

First you need to stop the service with the following command:

supervisorctl stop ahoi.js

Then using the hpservices tool, remove the configuration:

hpservices supervisord remove ahoi.js
Are you sure to delete the service in the following directory with all its content?
/home/username/.services/supervisord/ahoi.js
Are you sure? (y/n): y
successfully removed service ahoi.js
Are you sure to delete the application directory with all its content?
/home/username/app/ahoi.js
Are you sure? (y/n): y
successfully removed application directory ahoi.js

And finally remove the service from supervisor with the following command:

supervisorctl remove ahoi.js

Deactivating instead of removing a service

Alternatively, you can adjust the Supervisord configuration for this app so that Supervisord no longer starts it automatically. To do this, change the value of autostart to “false”.

autostart=false          ; start at supervisord start (default: true)

If you would like to deactivate, rather than delete, any services you no longer need, remember that unmaintained software on your server can pose a security risk. Years later, non-updated software can be suddenly, inadvertently restarted. It's better to create a backup of data and applications you no longer need, and delete them from your productive server.

Appendix: directory and file structure

~/.services/supervisord/hostpoint.conf

Supervisord main configuration file

~/.services/supervisord/myservice/service.conf

Supervisord service configuration file for myservice

~/.services/supervisord/myservice/log/

Directory for the stdout and stderr log files for myservice