This page will guide you through steps to setup high availability Magnolia CMS follow our previous how to article Guide - How to install your perfect linux server for Magnolia and Guide - How to configure an apache load balancer using another technology (HAProxy and memcached session management - msm) to provide pretty much the same functionalities on Ubuntu 16.04 or above operating system versions.

Concepts

-- https://assets.digitalocean.com/articles/high_availability/ha-diagram-animated.gif

HAProxy terminology

First of all, as we start working with HAProxy and incase you are new to it there are some specific concepts that could help you easier catch up with what we are doing provided by Mitchell Anicas on his Digital Ocean introduction to haproxy and loadbalancing concepts. You can skip this section and move on to Overview section if you already familiar with all of this and related concepts.

There are many terms and concepts that are important when discussing load balancing and proxying. We will go over commonly used terms in the following sub-sections.
Before we get into the basic types of load balancing, we will talk about ACLs, backends, and frontends.

acl url_blog path_beg /blog

backend web-backend
   balance roundrobin
   server web1 web1.yourdomain.com:80 check
   server web2 web2.yourdomain.com:80 check

backend blog-backend
   balance roundrobin
   mode http
   server blog1 blog1.yourdomain.com:80 check
   server blog1 blog1.yourdomain.com:80 check

frontend http
  bind *:80
  mode http

  acl url_blog path_beg /blog
  use_backend blog-backend if url_blog

  default_backend web-backend

Load-balancing and Sticky-sessions

Then if you are new to Load-balancing and Sticky sessions, please reference to below excerpt which I extracted from Ricky's Hodgepodge Blogspot for your information.

Session replication themes

Moreover there are several themes of Session replication from Thomas Andraschko Blogspot that you probably want to reference with some nice pictures and pros/cons analysis below.

High-availability clustered infrastructure

Apache Ignite web session clustering

Finally you might want to do a web session clustering, please referenece to Apache Ignite project to figure out how to integrate it to your existing system instead of using msm - memcached session management library. A short cross reference from Apache Ignite Web Session Clustering:

So shall we start!

Overview

We will simulate above architecture using 2 Debian based instances, one will not only acts as a load balancer but also as an application server node 1. The other one (node 2) will barely a tomcat instance which quite similar to the first one. Also in this guide we will setup HAProxy layer 7 load balancing (application layer) and memcached sticky sessions using kryo library (reference here).

Prerequisite

Note that please have JAVA 8 and Apache Tomcat app server available in your instances before we go on, for short I'll skip these basic steps and go on with some notices:

JAVA 8

Your JAVA_HOME might look like this:

JAVA_HOME="/usr/lib/jvm/java-8-oracle"

Apache Tomcat

And your CATALINA_HOME might be configured as below:

CATALINA_HOME="/opt/tomcat"

Remember to adjust your firewall so that traffics can go through Tomcat port 8080:

$ sudo ufw allow 8080 

For Tomcat automatically startup, use this:

$ sudo systemctl enable tomcat 

To enable Tomcat administrator login, add this line above your end tag of </tomcat-users> in your $CATALINA_HOME/conf/tomcat-users.xml (feel free to use your prefered name and password)

<user username="admin" password="password" roles="manager-gui,admin-gui"/>

PsiProbe

Deploy a correct PSI Probe version for sessions management and control (for example I'm using Tomcat 8.5 which required Probe v3.x or above). Download here https://github.com/psi-probe/psi-probe/releases

Checkpoint

Please make sure that Probe web-app is accessible from outside of our local network to any instance including our HAProxy address as below image before we move on.

You can use this link: http://haproxy_www_public_IP:8888/probe/ or any of http://tomcat-server-IP:8080/probe/ 

Memcached installation and configuration

Memcached is nicely publicly available to us, just deliver this command to install:

$ apt install memcached

Configure memcached for external referencing

By default memcached is listening on port 11211 and its configuration file is under '/etc/memcached.conf', you will need to modify its '-l' startup option using your instance IP address and open your instances firewall on this port accessing from outside. Here is how it looks like:

# Specify which IP address to listen on. The default is to listen on all IP addresses

# This parameter is one of the only security measures that memcached has, so make sure

# it's listening on a firewalled interface.

-l 192.168.56.109

And here is how to open your firewall on port 11211

$ sudo ufw allow 11211

Restart memcached service and check its status as below:

$ sudo service memcached restart

$ sudo service memcached status

Output:

● memcached.service - memcached daemon
   Loaded: loaded (/lib/systemd/system/memcached.service; enabled)
   Active: active (running) since Thu 2017-08-31 03:14:45 PDT; 9s ago
 Main PID: 5771 (memcached)
   CGroup: /system.slice/memcached.service
           └─5771 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 192.168.56.108


Aug 31 03:14:45 deb88jessie systemd[1]: Started memcached daemon.

Checkpoint

Go to any public manchine (outside your local network) and use this command to verify memcached service is up and running on both instances:

$ echo "stats settings" | nc tomcat-instance-1-IP 11211

Output:

$ echo "stats settings" | nc 192.168.56.108 11211

STAT maxbytes 67108864

STAT maxconns 1024

STAT tcpport 11211

STAT udpport 11211

STAT inter 192.168.56.108

STAT verbosity 0

STAT oldest 0

STAT evictions on

STAT domain_socket NULL

STAT umask 700

STAT growth_factor 1.25

STAT chunk_size 48

STAT num_threads 4

STAT num_threads_per_udp 4

STAT stat_key_prefix :

STAT detail_enabled no

STAT reqs_per_event 20

STAT cas_enabled yes

STAT tcp_backlog 1024

STAT binding_protocol auto-negotiate

STAT auth_enabled_sasl no

STAT item_size_max 1048576

STAT maxconns_fast no

STAT hashpower_init 0

STAT slab_reassign no

STAT slab_automove 0

STAT lru_crawler no

STAT lru_crawler_sleep 100

STAT lru_crawler_tocrawl 0

STAT tail_repair_time 0

STAT flush_enabled yes

STAT hash_algorithm jenkins

END

viet-mbp:~ viet.nguyen$ 

HAProxy

Install haproxy

$ sudo apt install haproxy

We need to enable the HAProxy init script, so HAProxy will start and stop along with your VPS.

$ sudo vi /etc/default/haproxy

Change the value of ENABLED to 1 to enable the HAProxy init script:

ENABLED=1

Configure haproxy

First, let's make a copy of the default haproxy.cfg file:

$ sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig

Now open haproxy.cfg in a text editor:

$ sudo vi /etc/haproxy/haproxy.cfg

You will see that there are two sections already defined: global and defaults. First we will take a look at some of the default parameters.

Under defaults, look for the following lines:

mode    http
option  httplog

Selecting http as the mode configures HAProxy to perform layer 7, or application layer, load balancing. This means that the load balancer will look at the content of the http requests and forward it to the appropriate server based on the rules defined in the frontend.

Frontend Configuration

The first thing we want to add is a frontend. For a basic layer 7 reverse proxying and load balancing setup, we will want to define an ACL that will be used to direct our traffic to the appropriate backend servers. 

At the end of the file, let's add our frontend, www. Be sure to replace haproxy_www_public_IP with the public IP of your haproxy-www VPS, also note that I'm using port 8888 for demo session:

frontend www
	bind haproxy_www_public_IP:8888
	option http-server-close
	# acl url_mgnlha path_beg /mgnlha
	# use_backend mgnlha-backend if url_mgnlha
	default_backend web-backend

Backend Configuration

After you are finished configuring the frontend, continue adding your first backend by adding the following lines. Be sure to replace the highlighted words with the appropriate values:

backend web-backend
   server tomcat1 192.168.56.108:8080 cookie n1 check
   server tomcat2 192.168.56.109:8080 cookie n2 check
   appsession JSESSIONID len 52 timeout 90s request-learn prefix
   #HAProxy version 1.6.3 will need this for sticky session, rem above line
   cookie JSESSIONID prefix nocache

HAProxy Stats

If you want to enable HAProxy stats, which can be useful in determining how HAProxy is handling incoming traffic, you will want to add the following into your configuration:

listen stats 
   bind  :1936
   stats enable
   stats scope www
   stats scope web-backend
   # stats scope mgnl-backend
   stats uri /stats
   stats realm Haproxy\ Statistics
   stats auth user:password
   stats refresh 30s
   stats show-node

Enabling HAProxy Logging

Enabling logging in HAProxy is very simple. First edit the rsyslog.conf file:

sudo vi /etc/rsyslog.conf

Then find the following two lines, and uncomment them to enable UDP syslog reception. It should look like the following when you are done:

$ModLoad imudp

$UDPServerRun 514

$UDPServerAddress 127.0.0.1

Now restart rsyslog to enable the new configuration:

$ sudo service rsyslog restart

HAProxy logging is is now enabled! The log file will be created at /var/log/haproxy.log once HAProxy is started.

Restart haproxy service

$ sudo service haproxy restart

$ sudo service haproxy status

Output:

● haproxy.service - HAProxy Load Balancer

   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled)

   Active: active (running) since Thu 2017-08-31 03:58:45 PDT; 6s ago

     Docs: man:haproxy(1)

           file:/usr/share/doc/haproxy/configuration.txt.gz

  Process: 5924 ExecStartPre=/usr/sbin/haproxy -f ${CONFIG} -c -q (code=exited, status=0/SUCCESS)

 Main PID: 5928 (haproxy-systemd)

   CGroup: /system.slice/haproxy.service

           ├─5928 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

           ├─5929 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

           └─5930 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds




Aug 31 03:58:45 deb88jessie systemd[1]: Started HAProxy Load Balancer.

Aug 31 03:58:45 deb88jessie haproxy-systemd-wrapper[5928]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc... -Ds

Aug 31 03:58:45 deb88jessie haproxy[5929]: Proxy www started.

Aug 31 03:58:45 deb88jessie haproxy[5929]: Proxy www started.

Aug 31 03:58:45 deb88jessie haproxy[5929]: Proxy web-backend started.

Aug 31 03:58:45 deb88jessie haproxy[5929]: Proxy web-backend started.

Aug 31 03:58:45 deb88jessie haproxy[5929]: Proxy stats started.

Aug 31 03:58:45 deb88jessie haproxy[5929]: Proxy stats started.

Hint: Some lines were ellipsized, use -l to show in full.

Checkpoint

Validate HAProxy Stats from external computer by going to your haproxy_www_public_IP port 1936 and context path 'stats' (as configured above) and verify that it is looking like this:

MSM - memcached session management integration

There are several session serialization strategies available, as they are described on SerializationStrategies. The default strategy uses java serialization and is already provided by the memcached-session-manager jar. Other strategies are provided by separate jars, in the section below you'll see which jars are required for which strategy.

Add memcached-session-manager jars to tomcat

Independent of the chosen serialization strategy you always need the memcached-session-manager-${version}.jar and either memcached-session-manager-tc6-${version}.jar for tomcat6, memcached-session-manager-tc7-${version}.jar for tomcat7 (attention: tomcat 7.0.23+) or memcached-session-manager-tc8-${version}.jar for tomcat8.

If you're using memcached, you also need the spymemcached-2.11.1.jar.

If you're using couchbase, you need additionally these jars: couchbase-client-1.4.0.jarjettison-1.3.jar, commons-codec-1.5.jar, httpcore-4.3.jar, httpcore-nio-4.3.jar, netty-3.5.5.Final.jar.

If you're using Redis, you need the jedis-2.9.0.jar.

Please download the appropriate jars and put them in $CATALINA_HOME/lib/.

We have selected below jars for you as of September 2017 which are compatible with memcached-session-manager-2.1.1, Tomcat 8, Kryo suite and spymemcached-2.11.1 (download below)

memcached.zip

Unzip it and copy them to your $CATALINA_HOME/lib folder and restart Tomcat to get it works. Which includes:

asm-5.0.4.jar

kryo-4.0.0.jar

kryo-serializers-0.41.jar

memcached-session-manager-2.1.1.jar

memcached-session-manager-tc8-2.1.1.jar

minlog-1.3.0.jar

msm-kryo-serializer-2.1.1.jar

objenesis-2.2.jar

reflectasm-1.11.3.jar

spymemcached-2.11.1.jar

Add custom serializers to your webapp (optional)

If you want to use java's built in serialization nothing more has to be done. If you want to use a custom serialization strategy (e.g. because of better performance) this has to be deployed with your webapp so that they're available in WEB-INF/lib/.

As msm is available in maven central (under groupId de.javakaffee.msm) you can just pull it in using the dependency management of your build system. With maven you can use this dependency definition for the kryo-serializer:

<dependency>
    <groupId>de.javakaffee.msm</groupId>
    <artifactId>msm-kryo-serializer</artifactId>
    <version>1.9.7</version>
    <scope>runtime</scope>
</dependency>

For javolution the artifactId is msm-javolution-serializer, for xstream msm-xstream-serializer and for flexjson it's msm-flexjson-serializer.

If you're not using a dependency management based on maven repositories these are the jars you need for the different serializers:

• kryo-serializer: msm-kryo-serializer, kryo-serializers-0.34+, kryo-3.x, minlog, reflectasm, asm-5.x, objenesis-2.x
• javolution-serializer: msm-javolution-serializer, javolution-5.4.3.1
• xstream-serializer: msm-xstream-serializer, xstream, xmlpull, xpp3_min
• flexjson-serializer: msm-flexjson-serializer, flexjson

Configure memcached-session-manager as <Context> Manager

Update the <Context> element (in META-INF/context.xml or what else you choose for context definition, please check the related tomcat documentation for this) so that it contains the Manager configuration for the memcached-session-manager, like in the examples below.

In this guideline as we are working with Magnolia CMS, let's do it in our Root context of tomcat server 1:

$ sudo vi $CATALINA_HOME/conf/context.xml

Put this above your closed tag </Context>

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"

    memcachedNodes="n1:192.168.56.108:11211,n2:192.168.56.109:11211"

	sticky="true"

	sessionBackupAsync="true"

    storageKeyPrefix="static:mgnlctx"

    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

    transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"

    />

Do it again on your tomcat server 2 with out any modification as we don't have any specific change for each of them.

Restart your tomcat instances and look into your logs to make sure that it's working as below:

$ sudo service tomcat restart

$ sudo tail -f $CATALINA_HOME/logs/catalina.out

Sample output:

31-Aug-2017 09:10:35.928 FINE [localhost-startStop-1] de.javakaffee.web.msm.MemcachedSessionService.loadTranscoderFactoryClass Loading transcoder factory class de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory using classloader ParallelWebappClassLoader

  context: examples

  delegate: false

----------> Parent Classloader:

java.net.URLClassLoader@27d6c5e0




31-Aug-2017 09:10:35.928 INFO [localhost-startStop-1] de.javakaffee.web.msm.serializer.kryo.KryoTranscoder.<init> Starting with initialBufferSize 102400, maxBufferSize 204800000 and defaultSerializerFactory de.javakaffee.web.msm.serializer.kryo.DefaultFieldSerializerFactory

31-Aug-2017 09:10:35.928 INFO [localhost-startStop-1] de.javakaffee.web.msm.MemcachedSessionService.startInternal --------

-  finished initialization:

- sticky: true

- operation timeout: 1000

- node ids: [n1, n2]

- failover node ids: []

- storage key prefix: mgnlctx_

--------

Make sure that you can 'see' this line (of couse the timestamp should not be the same)

31-Aug-2017 09:10:36.056 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 25338 ms

Which indicates that your Tomcat instance was successfully started up without any obstacle.

Check point

Go to this location from your external computer to make sure that either your HAProxy or Sticky sessions are working

http://haproxy_www_public_IP:8888/probe/appsummary.htm?webapp=%2fprobe&size=

Deploy Magnolia CMS

Final step in this tutorial is to install Magnolia CMS to your Tomcat instances and make sure that it works correctly.

Here is Download link for Magnolia CMS 5.5.6 demo WAR file.

Please reference to our Release notes for Magnolia CORE 5.5.6 for more information.

Stop your tomcat instances, copy downloaded war file to $CATALINA_HOME/webapps, rename it to 'mgnl' or whatever name that you prefer, then start tomcat again with a closer look at our 'catalina.out' for changes and notifications.

Check point

After all your instances up and running, start your HAProxy service and go to its public IP, login and open our Travel demo page for a sanity test. Use this link:

http://haproxy_www_public_IP:8888/mgnl/.magnolia/admincentral#app:pages:detail;/travel:edit

Known issues and best practices

Known issues

Q1: Wrong configuration may cause this issue:

storage (info.magnolia.ui.framework.ioc.BeanStore)
attributes (com.vaadin.server.VaadinSession)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:362)
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:303)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
	at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:629)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:87)
    ...
Caused by: java.util.ConcurrentModificationException
	at java.util.Vector$Itr.checkForComodification(Vector.java:1184)
	at java.util.Vector$Itr.next(Vector.java:1137)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:99)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
	... 252 more

Q2: Cannot load MagnoliaProWidgetSet.

→ Increase kryo cache buffer size as below best practice #2 - set -Dmsm.kryo.bufferSize.max=204800000

Q3: java.io.NotSerializableException:

→ Use 3rd party serialization solution such as Kryo in this tutorial

Best practices

1. Use Magnolia CMS Memcached module in combination with our deployment, documentation here: https://documentation.magnolia-cms.com/display/DOCS/Memcached+implementation

2. Enable Memcached logger, increase msm-kryo max buffer size and increase Tomcat memory in '$CATALINA_HOME/bin/catalina.sh' as below 

   CATALINA_OPTS="$CATALINA_OPTS -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SunLogger -Dmsm.kryo.bufferSize.max=204800000"
   CATALINA_OPTS="$CATALINA_OPTS -Xms1600m -Xmx1600m"

3. Setting up Jackrabbit clustering (documentation here) for content sharing and to save publish efforts on less important big contents (LOBs)

4. Please follow Grégory Joseph recommendation in this document Re: Guide - How to install your perfect linux server for Magnolia 

   • don't add stuff to catalina.sh. env variables need to be set in setenv.sh; and even then, I'd recommend using magnolia_control.sh or something similar, to avoid those vars to be set ALSO when calling the shutdown program. (which IS an issue if you have jmx or debug options in there)
   • likewise, I wouldn't recommend to set JAVA_HOME system-wide, but just in setenv.sh (more explicit, less magic, and you can use different java versions for different tomcats)
   • you might want to double-check the scripts, there are lost \ in there.

Contact support@magnolia-cms.com for more detail and support, please note that we would prefer JIRA tickets on our SUPPORT project for proper SLA reaction rather than personal emails or accidentally ping on Skype or Facebook 

TO BE CONTINUED!