Elasticsearch on Managed Flex Servers

Elasticsearch is a search engine based on the Lucene library. The Java program saves its data in a NoSQL format (JSON). Communication with the clients is processed via a RESTful API. Along with Solr, Elasticsearch is the most commonly used search engine. This article explains how to install and use Elasticsearch on your Managed Flex Server.

Requirements

Elasticsearch requires a minimum of a Managed Flex Server M.

Installation

We recommend creating an ~/app/ directory for your applications and then creating a subdirectory for each of your applications. This ensures that your applications are organized and easily accessible. So, for Elasticsearch, we will create the directory ~/app/elasticsearch.

mkdir -p ~/app/elasticsearch

First, install the relevant (latest) version of Elasticsearch. You can find them on the elastic website. Select the version for LINUX X86_64. Extract the archive to the directory created above.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-X.X.X.tar.gz -O /tmp/elastic.tar.gz
tar -xzf /tmp/elastic.tar.gz -C ~/app/elasticsearch
rm -rf /tmp/elastic.tar.gz
cd ~/app/elasticsearch

ln -s elasticsearch-X.X.X current

You can now see Elasticsearch in the ~/app/elasticsearch/current/ folder. Using the current Symlink makes it easy (e.g. when upgrading to newer versions) to operate and manage parallel installations.

Java has been pre-installed on your Managed Flex Server.
In order for Elasticsearch to find OpenJDK, you need to run the following commands:

echo 'export ES_JAVA_HOME="/usr/local/openjdk17"' >> ~/.profile
echo 'export ES_JAVA_OPTS="-Xms1g -Xmx1g"' >> ~/.profile			
source ~/.profile
			

Configuring Elasticsearch

Elasticsearch is configured with the config/elasticsearch.yml file in your elasticsearch directory. The file is commented copiously and contains plenty of information on options.

Paths

Pay particular attention to the paths to the data (path.data) and the logs (path.logs). For the latter, it is important to ensure that they are periodically deleted, so that your webspace does not suddenly fill up. By default, each path is written to a subdirectory of the installation folder.

Network

By default, the Elasticsearch REST API is available via IP 127.0.0.1 and port 9200 (port 9300 for the node communication). No other changes are generally needed. However, for security reasons, never set it to a public IP address!

X-Pack

Because X-Pack is not supported under FreeBSD (similar to other platforms), you need to disable it in the configuration. Edit the configuration file using a text editor of your choice or add the configuration directly via the shell:

echo -e "\n\n# disable xpack as it is not supported on FreeBSD, $USER, $(date)\nxpack.ml.enabled: false" >> config/elasticsearch.yml

Disk Watermark

The limit for the minimum of available storage space is set as a percentage by default and is too early to take effect due to the large amount of storage space. The values should therefore be defined in GB.

echo -e "\n\n# adjust low disk watermark, $USER, $(date)\ncluster.routing.allocation.disk.watermark.low: 20G" >> ~/app/elasticsearch/current/config/elasticsearch.yml
echo -e "\n\n# adjust high disk watermark, $USER, $(date)\ncluster.routing.allocation.disk.watermark.high: 10G" >> ~/app/elasticsearch/current/config/elasticsearch.yml				
echo -e "\n\n# adjust watermark flood stage, $USER, $(date)\ncluster.routing.allocation.disk.watermark.flood_stage: 5G" >> ~/app/elasticsearch/current/config/elasticsearch.yml

Starting Elasticsearch

Once you have successfully installed and configured Elasticsearch, you can start testing the service using ~/app/elasticsearch/current/bin/elasticsearch:

./bin/elasticsearch
Issues of Elastic Search

Please note that it can take up to thirty seconds until Elasticsearch has started up and is ready to use.

You can find details and other Elasticsearch configuration options in the official documentation for Elasticsearch.

Integration

Using Custom Service Control

Previously, you started Elasticsearch manually via the shell. While this is an easy method for testing and development purposes, it is not an option for production operations. The Custom Service Control (CSC) can handle the operation and automatic management of Elasticsearch, e.g. starting and stopping it when performing maintenance on your server.

The configuration in the CSC is fairly simple. To do this, create a new custom service for Elasticsearch:

hpservices supervisord add elasticsearch
successfully created dir for elasticsearch: /home/username/.services/supervisord/elasticsearch
successfully created dir for elasticsearch: /home/username/.services/supervisord/elasticsearch/log
supervisord service config file written: /home/username/.services/supervisord/elasticsearch/service.conf

This creates a configuration framework for the new service in the file ~/.services/supervisord/elasticsearch/service.conf. For Elasticsearch, the command must be adapted and the environment variable added, otherwise you can accept all suggested values (delete the semicolon at the beginning of the line).

[program:elasticsearch]
command=%(ENV_HOME)s/app/elasticsearch/current/bin/elasticsearch ; the program (can take args)
directory=%(ENV_HOME)s/app/elasticsearch/ ; directory to cwd to before exec (default no cwd)
autostart=true ; start application at supervisord start (default: true)
stopwaitsecs=10 ; max num secs to wait before SIGKILL (default 10)
stdout_logfile=%(ENV_HOME)s/.services/supervisord/elasticsearch/log/default.log
stdout_logfile_maxbytes=1MB ; filesize at which to rotate logfiles (default ist 50MB)
stdout_logfile_backups=10 ; number of stdout logfile backups (0 means none, default 10)
stderr_logfile=%(ENV_HOME)s/.services/supervisord/elasticsearch/log/default.err
stderr_logfile_maxbytes=1MB ; filesize at which to rotate logfiles (default is 50MB)
stderr_logfile_backups=10 ; number of stderr logfile backups (0 means none, default 10)
environment=ES_JAVA_HOME="/usr/local/openjdk17",ES_JAVA_OPTS="-Xms1g -Xmx1g"

Using supervisorctl update, you can apply the new configuration for the Elasticsearch service:

supervisorctl update
elasticsearch: added process group

Attention: If you receive an error, you must uniquely execute before "hpservices supervisord start".

For more detailed information on working with CSC and the options it offers, refer to the guide for Custom Service Control.