Sending Abiquo logs to a Graylog server

Abiquo tomcat webapps can be configured to ship logs to a graylog2 server for log aggregation an analysis. To get log shipping working we will need to add a plugin for logback (the logging framework used by the Abiquo webapps) that will send the logs to the graylog server.

Installing and configuring graylog is out of the scope of this document and there are plenty of good docs on the graylog documentation site.

logstash-gelf

The plugin we are going to use is called logstash-gelf. This plugin can handle other logging frameworks besides logback.

Download a release for logstash-gelf from its releases page in the github repo to your Abiquo servers. At the time of writing, the latest version is 1.11.0, so if you download a newer version, you'll need to adapt the filenames listed in this document.

Once we have the jar file for the plugin on the servers, we will need to copy the jar file into the lib folder of each webapp:

# for dir in $(ls /opt/abiquo/tomcat/webapps); do cp logstash-gelf-1.11.0.jar $dir/WEB-INF/lib/; done

logback.xml

In order to make Abiquo send logs to the graylog instance, we will also need to modify the logging config file, logback.xml for each webapp, similar to what's described on the Configure webapp logging page.

In this case, we will add a new appender that will be in charge of sending the logs to graylog. The following shows an already modified version of the log config file for the appliance manager webapp.

<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="60 seconds">
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>%d{HH:mm:ss.SSS} %-5level %logger{36} - %msg%n</Pattern>
        </layout>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
    </appender>
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${catalina.base}/logs/am.log</file>
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>%d{HH:mm:ss.SSS} %-5level %logger{36} - %msg%n</Pattern>
        </layout>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <MaxHistory>30</MaxHistory>
            <fileNamePattern>${catalina.base}/logs/am-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>50MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>
    <appender name="GELF" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
        <host>udp:10.60.13.47</host>
        <port>12201</port>
        <originHost>nardo40</originHost>
        <version>1.1</version>
        <facility>java-test</facility>
        <extractStackTrace>true</extractStackTrace>
        <filterStackTrace>true</filterStackTrace>
        <mdcProfiling>true</mdcProfiling>
        <timestampPattern>yyyy-MM-dd HH:mm:ss,SSSS</timestampPattern>
        <maximumMessageSize>8192</maximumMessageSize>
        <!-- This are static fields -->
        <additionalFields>abiquoenv=nardo40,webapp=am</additionalFields>
        <!-- Optional: Specify field types -->
        <additionalFieldTypes>abiquoenv=String,webapp=String</additionalFieldTypes>
        <!-- This are fields using MDC -->
        <mdcFields>mdcField1,mdcField2</mdcFields>
        <dynamicMdcFields>mdc.*,(mdc|MDC)fields</dynamicMdcFields>
        <includeFullMdc>true</includeFullMdc>
    </appender>
    <logger name="com.abiquo">
        <level value="DEBUG" />
    </logger>
    <logger name="org.apache.wink">
        <level value="WARN" />
    </logger>
    <root>
        <level value="INFO" />
        <appender-ref ref="FILE" />
        <appender-ref ref="STDOUT" />
        <appender-ref ref="GELF" />
    </root>
</configuration>

The areas added to get log shipping working are:

Appender configuration. This section configures the behavior of the plugin, setting the target for the logs and options in the plugin. In this case, of special interest are:

host: The graylog host to send the logs to. It also specifies using standard UDP Gelf logging.
originHost: The string that will appear as the source field in graylog.

additionalFields: Some additional fields to add to the messages to give them more context. In this case, note we are setting a webapp field to specify which of the webapps is sending the logs.

    <appender name="GELF" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
        <host>udp:10.60.13.47</host>
        <port>12201</port>
        <originHost>nardo40</originHost>
        <version>1.1</version>
        <facility>java-test</facility>
        <extractStackTrace>true</extractStackTrace>
        <filterStackTrace>true</filterStackTrace>
        <mdcProfiling>true</mdcProfiling>
        <timestampPattern>yyyy-MM-dd HH:mm:ss,SSSS</timestampPattern>
        <maximumMessageSize>8192</maximumMessageSize>

        <!-- This are static fields -->
        <additionalFields>abiquoenv=nardo40,webapp=am</additionalFields>
        <!-- Optional: Specify field types -->
        <additionalFieldTypes>abiquoenv=String,webapp=String</additionalFieldTypes>

        <!-- This are fields using MDC -->
        <mdcFields>mdcField1,mdcField2</mdcFields>
        <dynamicMdcFields>mdc.*,(mdc|MDC)fields</dynamicMdcFields>
        <includeFullMdc>true</includeFullMdc>
    </appender>

Add the appender to the logger output. At the end of the log config file, you need to add this appender to the output so it actually gets to send any log lines
```
    <root>
        ...
        <appender-ref ref="GELF" />
    </root>
```

Wrap up

The last step is to restart the Abiquo tomcat service. Since we added a new plugin we need to restart tomcat so it takes that new plugin into account. On your graylog server, you should now be able to see the messages coming in.