Adding TLS To Graylog2 With Logstash

I really love Graylog2. The web interface is great, search is a breeze, and it does dashboards and analytics. And it supports LDAP and multiple users, which is something that Kibana doesn't do (though otherwise I think Kibana is pretty rad too). Graylog2 is also super fast: I've pushed my modestly spec'd servers to well over 1000 msgs/s without breaking a sweat.

But the recent 0.20.x releases have been missing some key features like TLS encryption and log filtering. It seems likely that these features will be added in over time, but they just aren't there right now.

Fortunately Logstash supports both TLS and filtering, plus a whole lot of other cool stuff due to its robust collection of plugins. You can deploy a Logstash in front of Graylog2 and wind up with a pretty good solution that does everything you want.

Log Flow

It's really simple:

(Client) --TLS SYSLOG--> (Logstash on 0.0.0.0) --GELF--> (Graylog2 on 127.0.0.1)

1. Clients send TLS encrypted syslog to Logstash
2. Logstash decrypts, filters and otherwise molests log data
3. Logstash forwards GELF formatted messages to Graylog2, which is listening on localhost.

About Certificates

Before setting up TLS, you need to have PKI deployed so you can generate certificates. That's outside the scope of this post, but there's lots of information out there about how to get that stuff set up.

Technically you don't need a CA if you use self signed certs... but that's not a great idea. We're in a post Snowden age and PKI is a basic mission critical service like DHCP or DNS or NTP. You just need it.

Logstash Configuration

Here's a simple example config that I adapted from this Logstash Cookbook recipe. A TCP input is listening for SSL, and a GELF output is sending on localhost.

The Grok plugin parses messages that are syslog formatted and splits the text into fields that can be used for filtering or for other outputs. For example you could add a statsd output to count SSH auth failures by analyzing syslog_program and syslog_message. That's sort of a dumb example because you can count things in Graylog, but maybe you want them in Graphite too. I don't know man. Everybody likes graphs.

This is basically the same as what I have in production, using Logstash 1.4.1 and Graylog2 0.20.1. Of course, you will need to configure the host/port/other stuff to fit your environment.

# /etc/logstash/conf.d/logstash.conf

input {
  tcp {
    host       => "0.0.0.0"
    mode       => "server"
    port       => 5144
    ssl_enable => true
    ssl_cert   => "/etc/ssl/puppet/logstash.crt"
    ssl_key    => "/etc/ssl/puppet/logstash.key"
    ssl_cacert => "/etc/ssl/certs/my_ca_cert.pem"
    ssl_verify => false
    type       => "syslog"
  }
}
 
filter {
  if "-- MARK --" in [message] {
    drop { }
  }
 
  if [type] == "syslog" {
    grok {
      match => {
        "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
      }
    }
 
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
 
    if !("_grokparsefailure" in [tags]) {
      mutate {
        add_field => { "syslog_full" => "%{message}" }
      }
      mutate {
        replace => [ "message", "%{syslog_message}" ]
      }
 
      # example filter
      if [syslog_program] == "snoopy" {
        drop { }
      }
    }
  }
}
 
output {
  # You can use the stdout output to debug your conf
  # stdout { codec => rubydebug }
  gelf {
    level         => ["%{syslog_severity}"]
    host          => "127.0.0.1"
    port          => 12201
    sender        => "%{syslog_hostname}"
    short_message => "%{message}"
    full_message  => "%{syslog_full}"
    ship_metadata => true
    ship_tags     => true
    ignore_metadata => [
      "type", "syslog_pri", "syslog_timestamp", "syslog_hostname", "syslog_message", "syslog_severity", "syslog_severity_code", "syslog_facility_code", "syslog_full"
    ]
  }
}

Graylog Configuration

Configuring a Graylog2 input is super easy:

Client Configuration

Finally, here's a sample syslog-ng conf that sends logs to a TLS destination. The filter section is just what I happen to use, you will want to modify it to fit your environment.

# /etc/syslog-ng/conf.d/99-graylog2.conf

filter f_graylog {
    not (level(info..notice) and program(CRON))
    and not (level(info..notice) and program("crontab"))
    and not (level(info..notice) and program("puppet-agent"))
    and not (level(info..notice) and program("logstash-forwarder"))
;};
 
destination graylog2 {
    tcp ("graylog2.mydomain.local" port(5144)
        tls(ca_dir("/etc/ssl/certs"))
    );
};
 
log { source(s_src); filter(f_graylog); destination(graylog2); };