X
Popular Searches

How to Set up RSysLog with Loggly

Bash Shell

In the Linux operating system, many applications send logs to syslog, the primary logging mechanism. Best security practices often dictate that it is ideal to send your logs to a separate system, like Loggly, a SaaS solution for log data management.

There are many reasons to do this, but the primary ones are:

  • Backup of critical log data
  • Ability to verify log integrity and detect data changes
  • Consolidation of logs across systems

Loggly has many features, as shown below, but one feature, in particular, is Loggly’s agent-free deployment ability.

  • Monitoring multiple cloud systems, internal logs, and application logs
  • Integration with third-party systems, such as Slack, GitHub, or PagerDuty
  • Analysis and visualization of data using graphs and KPIs
  • Agent-free deployment

In this article, we are going to explain how to integrate Loggly with rsyslog, a common and powerful replacement for syslogd. This means that there is no agent needed on a Linux system, and rsyslog can be configured to directly send the logs to Loggly.

Installing rsyslog

Many systems come with rsyslog preinstalled, but if it is not, a few steps are needed for installation. Even though rsyslog is commonly included in the repositories of many distributions, it may not be the latest version. In this article, Ubuntu is the distribution of choice, notably that of 18.04. Instructions for other distributions are located here.

This section may say installing from source, but the repositories listed will allow you to install the latest versions of the package.

Adding the rsyslog Repository

In the case of Ubuntu, we need to add the Adiscon PPA repository.

apt-get update && apt-get install -y software-properties-common
add-apt-repository -y ppa:adiscon/v8-stable

Installing rsyslog

There are two main packages we need to install for rsyslog. Beyond the base package itself, the rsyslog-gnutls package allows us to use an encrypted connection to the Loggly service.

apt-get update
apt-get install rsyslog
apt-get install rsyslog-gnutls

Adding the Loggly TLS Certificates

Before we configure rsyslog with TLS, we first need to download and make accessible the Loggly certificates.

sudo mkdir /etc/rsyslog.d/keys/ca.d
wget -O /etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt <https://logdog.loggly.com/media/logs-01.loggly.com_sha12.crt>

Verifying the Work Directory Permissions

There are some cases where the /var/spool/rsyslog permissions are incorrect, and the following code will verify that the permissions are set correctly (in the context of Ubuntu 18.04).

sudo chown -R syslog:adm /var/spool/rsyslog
sudo chmod -R 644 /var/spool/rsyslog

Configuring Loggly

Over time, the configuration directives and syntax have changed for rsyslog. Often, there is a mix of the old and new directives available. Below is a default configuration for rsyslog using the new syntax only. The primary reason to modify this configuration is to point rsyslog to the certificate that we just downloaded.

#################
#### MODULES ####
#################

module( load="imuxsock" )
module( load="imklog" )
module( load="builtin:omfile"
        fileOwner="syslog"
        fileGroup="adm"
        fileCreateMode="0644"
        dirOwner="syslog"
        dirGroup="adm"
        dirCreateMode="0755"
)
module( load="impstats"
        interval="600"
        severity="7"
        log.syslog="off"
        log.file="/var/log/rsyslog_stats.log"
)

###########################
#### GLOBAL DIRECTIVES ####
###########################

global (
    maxMessageSize="64k"
    defaultNetstreamDriverCAFile="/etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt"
    defaultNetstreamDriver="gtls"
    workDirectory="/var/spool/rsyslog"
)

#
# Include all config files in /etc/rsyslog.d/
#
include(file="/etc/rsyslog.d/*.conf")

The important part of the code is the defaultNetstreamDriverCAFile and defaultNetstreamDriver. These directives need to be configured correctly to point to the downloaded certificate.

After signing in to your Loggly account, you will need to create a customer token. This is located under Source Setup > Customer Tokens. As there is a new user interface coming, included below are screenshots for both the old and new interfaces.

 Source Setup > Customer Tokens to create a new customer token
New User Interface.
 Source Setup > Customer Tokens to create a new customer token
Old User Interface.

Once you have navigated to Customer Tokens, click on the “Add New” button to generate a new token. It is best to give this token a description.

Adding a description to a customer token.

Finally, you will have a token that you can use. Copy this for use later in our configuration files.

A customer token.

Configuring rsyslog for Loggly

The final step for configuring rsyslog for Loggly is to define our configuration file and restart rsyslog. Below is a default configuration file located in /etc/rsyslog.d/22-remote.conf that will tell rsyslog to send syslog events to Loggly.

The number 22 is not important, this is merely a way to define the order of loading the configuration files. Choose a number that makes sense in your configuration.

template(
    name="LogglyFormat"
    type="string"
    string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [ca35d899-0232-4888-a8d5-118fbf5caf8d tag=\\"RsyslogTLS\\"] %msg%\\n"
)

# Send all Syslog Messages to Remote
action(
    type="omfwd"
    protocol="tcp"
    target="logs-01.loggly.com"
    port="6514"
    template="LogglyFormat"
    StreamDriver="gtls"
    StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
    StreamDriverPermittedPeers="*.loggly.com"
    ResendLastMSGOnReconnect="on"
    queue.spoolDirectory="/var/spool/rsyslog"
    queue.filename="queue_sendToLoggly"
    queue.size="5000"
    queue.dequeuebatchsize="300"
    queue.highwatermark="4500"
    queue.lowwatermark="3500"
    queue.maxdiskspace="1g"
    queue.saveonshutdown="on"
    queue.type="LinkedList"
)

Within this configuration, we define the format for sending the logs over, which helps Loggly to properly categorize the logs. The action configuration is for sending all Syslog messages to Loggly by default. Finally for this configuration to take effect, we need to restart rsyslog.

sudo service rsyslog restart

rsyslog Troubleshooting

If you are encountering issues with the configuration, which tend to be either permissions or a mis-configuration, then you can view the current log here: /var/log/rsyslog.log. Additionally, to turn on a higher level of logging, add the following lines to your rsyslog.conf file and restart rsyslog.

$DebugFile /var/log/rsyslog_debug.log
$DebugLevel 2

Conclusion

The combination of rsyslog and Loggly is a powerful one. Allowing you to consolidate, validate, and analyze your logs is important to maintaining a proper security posture. You will quickly find that the advanced features of Loggly are well worth the setup when used in conjunction with your advanced logging configuration for rsyslog.

Adam Bertram Adam Bertram
Adam Bertram is a 20+ year veteran of IT and an experienced online business professional. He’s a consultant, Microsoft MVP, blogger, trainer, published author and content marketer for multiple technology companies. Catch up on Adam’s articles at adamtheautomator.com, connect on LinkedIn, or follow him on Twitter at @adbertram. Read Full Bio »

The above article may contain affiliate links, which help support CloudSavvy IT.