Push logs in batches to Elasticsearch from Magnolia log4j2

This guideline will walk you through directly pushing Magnolia logs to ElasticSearch without any 3rd party program like fluentd. Since ElasticSearch is provided as an AWS service so you can easily use this guideline if you're hosting Magnolia CMS on the AWS cloud. By passing 3rd party intervention, we can reduce system loads as well as simplify our deployment model, saving maintenance time, running costs and system management efforts.

Introduction

Below is a typical deployment model for fluentd which uses td-agent program when people integrate your logs and push it to ElasticSearch. It has some weakness with performance impact and system overhead risks that you will find later when we propose the direct logs pushing from our running Tomcat to the server.

Typical integration model

https://www.fluentd.org/guides/recipes/elasticsearch-and-s3

Apply to Magnolia deployment

Weakness

Once Tomcat instance run into error, td-agent will keep reading 'catalina.out' file because you have lots of output in your error log. So Fluentd need to process a lot of output log to get the latest log. td-agent also have to push updated info via its channel to fluentd log aggregator in log collector instance. This would make your system run into overhead of bandwidth, processing and memory.

Proposed model

Implement the model

Update your pom.xml to use integration libraries

After researching for a possible integration way, we can go with https://github.com/rfoltyns/log4j2-elasticsearch which includes few libraries that you can update your pom.xml as below:

<dependency>
     <groupId>org.appenders.log4j</groupId>
     <artifactId>log4j2-elasticsearch-jest</artifactId>
     <version>1.2.0</version>
</dependency>
<dependency>
     <groupId>org.appenders.log4j</groupId>
     <artifactId>log4j2-elasticsearch6-bulkprocessor</artifactId>
     <version>1.2.0</version>
</dependency>

Remember to build your webapp and bundle it again.

Update your log4j2.xml for asynchronous bulk appender

Add Elasticsearch appender

Setting your log4j2.xml follow their guideline by adding below appender to your existing one:

<Elasticsearch name="elasticsearchAsyncBatch">
    <IndexName indexName="mgnl_log4j2" />
    <AsyncBatchDelivery>
        <JestHttp serverUris="http://localhost:9200" />
    </AsyncBatchDelivery>
</Elasticsearch>

Add a new AppenderRef to your existing appenders

<AppenderRef ref="elasticsearchAsyncBatch"/>

Example:

Running and verifying results

Run Elasticsearch

Go to its 'bin' folder and run

./elasticsearch

Run Kibana

Go to its folder and run

./bin/kibana

Run your webapp

It should have some console output such as:

2018-11-16 14:37:13,183 INFO  info.magnolia.init.DefaultMagnoliaInitPaths       : servername is localhost, rootPath is /private/var/folders/9b/89d4rhb523768s90by9gy7sh0000gn/T/1-mgnlwebapp55, webapp is 1-mgnlwebapp55, contextPath is /mgnlwebapp55
Initializing Log4J
2018-11-16 14:37:14,405 INFO  io.searchbox.client.AbstractJestClient            : Setting server pool to a list of 1 servers: [http://localhost:9200]
2018-11-16 14:37:14,408 INFO  io.searchbox.client.JestClientFactory             : Using multi thread/connection supporting pooling connection manager
2018-11-16 14:37:14,592 INFO  io.searchbox.client.JestClientFactory             : Using default GSON instance
2018-11-16 14:37:14,592 INFO  io.searchbox.client.JestClientFactory             : Node Discovery disabled...
2018-11-16 14:37:14,592 INFO  io.searchbox.client.JestClientFactory             : Idle connection reaping disabled...
2018-11-16 14:37:15,436 INFO  info.magnolia.init.MagnoliaServletContextListener : Property sources loaded: [DefaultMagnoliaConfigurationProperties with sources: [SystemPropertySource][ServletContextPropertySource from WEB-INF/config/default/magnolia.properties][ModulePropertiesSource from modules [core, dam, freemarker-support, license, categorization-support, ce-product-descriptor, imaging-support, publishing-core, resource-loader, rest-client, rest-integration, config, i18n, publishing-receiver, publishing-sender, publishing-transactional-core, rest-services, cache, publishing-transactional-receiver, publishing-transactional-sender, rendering, ui-framework, virtual-uri, appswitcher, dam-jcr, diff, ehcache3, imaging, link-unfurl, resteasy-client, scheduler, templating, ui-mediaeditor, block-templating, resources, task-management, templating-jsp, ui-admincentral, about-app, backup, cache-app, groovy, log-tools, magnolia-resources-templating, messages-app, pages, password-manager, publishing-app, resources-app, sample-app, security-app, site, tools, ui-contentapp, cache-browser-app, contacts, dam-app, dam-imaging, dam-templating, definitions-app, diff-pages-integration, enterprise, google-sitemap, jcr-browser-app, jcr-tools, mail, mte-imaging, publishing-transactional-app, site-app, soft-locking, workflow, cookie-manager, dependencies, ee-std-product-descriptor, form, mte, multisite, personalization-integration, rest-content-delivery, translation, visitor-manager, workflow-jbpm, advanced-cache, advanced-cache-personalization, ee-pro-product-descriptor, javascript-models, marketing-tags, mtk, personalization-components, personalization-preview-app, personalization-segmentation-app, privacy-form, privacy-ui, public-user-registration, rssaggregator, translation-pages-integration, advanced-cache-app, advanced-cache-dpc, categorization, content-editor, content-tags-core, personalization-personas-app, content-tags-ui, lang-de, lang-fr, lang-ja, personalization-samples, personalization-traits, stories-app, travel-demo, privacy-sample, tours, travel-demo-marketing-tags, travel-demo-component-personalization, travel-demo-content-tags, travel-demo-multisite, travel-demo-personalization, travel-demo-stories-app, webapp]][InitPathsPropertySource]]
---------------------------------------------
MAGNOLIA LICENSE
---------------------------------------------
Version number : 5.7.1
Core version   : 5.7.1
Build          : 27. July 2018 (rev. 089ebfa5a98ef8a81cfb6c1eca951735edd0050c of UNKNOWN)
Edition        : Enterprise Edition Pro
Provider       : Magnolia International Ltd. (info@magnolia-cms.com)
2018-11-16 14:37:15,449 INFO  info.magnolia.cms.beans.config.ConfigLoader       : Initializing content repositories
2018-11-16 14:37:15,449 INFO  info.magnolia.repository.DefaultRepositoryManager : Loading JCR
2018-11-16 14:37:15,458 INFO  info.magnolia.repository.DefaultRepositoryManager : Loading JCR magnolia
....

Then discover your Kirbana log maybe using this link (depends on your deployment) http://localhost:5601/app/kibana#/discover?_g=()

Double check it with our console output to verify its accuracy

Hope this helps!

Page tree