How to create a single-node Graylog instance and analyze FortiGate logs

Firewall logs provide a wealth of information about a network. They can be used to identify devices, troubleshoot policies, and even help determine the impact of a cyber attack. Graylog is a powerful open source log collection and analysis platform that is well-suited for managing firewall logs. This guide explains how to create a production-ready single node Graylog instance with bidirectional authentication to the firewalls, and how it can be used to analyze FortiGate firewall logs with premade dashboards.

Check out the presentation I made on this topic here.

Download the Debian ISO and create a bootable flash drive using Rufus. At the time of this writing. the latest stable release is Debian 12, codename bookworm. Install Debian on a dedicated workstation or server by booting from the flash drive and selecting the non-graphical install option. Follow the prompts to set the system keyboard, hostname, root password (make sure you remember what you set that to!), standard user account, time zone, and system partitions. When you reach the “select and install software” step, use the arrow keys and spacebar to deselect Debian Desktop Environment and GNOME, and select SSH server for remote shell access.

Sign in with your standard account, then switch to the root user using the su – command.

su -

Add your standard user account to the sudo group, so you can use the sudo command to run commands as root without having to switch accounts (replace username with your actual standard account username.

usermod -aG sudo username

Run the exit command twice to log out, then log back in for the change to take effect.

A Graylog node requires two dependencies: MongoDB and OpenSearch.

Install Graylog dependencies

Run the following commands to install MongoDB Community Edition.

Note: MongoDB does not support Debian 12 “Bookworm” because it requires libssl1.1, which is only available in Debian 11 “Buster”. A workaround is to use the MongoDB package repository for Ubuntu 22.04 Jammy Jellyfish.

sudo apt-get install -y gnupg curl
curl -fsSL https://pgp.mongodb.com/server-7.0.asc | \
   sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg \
   --dearmor
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo systemctl daemon-reload
sudo systemctl enable mongod.service
sudo systemctl restart mongod.service

Run the following commands to install OpenSearch.

curl -o- https://artifacts.opensearch.org/publickeys/opensearch.pgp | sudo gpg --dearmor --batch --yes -o /usr/share/keyrings/opensearch.pgp
echo "deb [signed-by=/usr/share/keyrings/opensearch.pgp] https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable main" | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list
sudo apt update
sudo OPENSEARCH_INITIAL_ADMIN_PASSWORD=$(tr -dc A-Z-a-z-0-9_@#%^-_=+ < /dev/urandom  | head -c${1:-32}) apt-get -y install opensearch

Configure OpenSearch

Before starting OpenSearch, it must be properly configured. The nano command is a CLI text editor that is friendly for new users. Use nano to edit the OpenSearch configuration file.

sudo nano /etc/opensearch/opensearch.yml

Most of the options can be kept at their default values.

Set the cluster name to graylog.

cluster.name: graylog

In the network section, configure OpenSearch to listen on the loopback interface.

network.host: 127.0.0.1

Add the discovery type in the discovery section.

discovery.type: single-node

Add these options in the various section.

action.auto_create_index: false
plugins.security.disabled: true

Add this setting at the bottom of the file.

# Custom settings
search.max_buckets: 200000

Save the changes to the file by pressing ctrl-o, and then enter. Press ctrl-x to exit nano.

The JVM heap limits for OpenSearch must be set to half the size of the system memory.

sudo nano /etc/opensearch/jvm.options

For example, if your system has 16 GB of RAM, use the following settings.

-Xms8g
-Xmx8g

The -Xms and -Xmx values must be identical.

Save the changes to the file by pressing ctrl-o, and then enter. Press ctrl-x to exit nano.

Configure kernel parameters by running the following commands.

sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf

Start OpenSearch by running the following commands.

sudo systemctl daemon-reload
sudo systemctl enable opensearch
sudo systemctl start opensearch

Install Graylog

Run the following commands to install Graylog.

wget https://packages.graylog2.org/repo/packages/graylog-5.2-repository_latest.deb
sudo dpkg -i graylog-5.2-repository_latest.deb
sudo apt-get update && sudo apt-get install graylog-server

Configure Graylog

Edit the Graylog configuration file.

sudo nano /etc/graylog/server/server.conf

The password_secret value must be random. To generate a random secret, open a new terminal session in a separate window, and run the following commands.

sudo apt install -y pwgen
pwgen -N 1 -s 96

The password for the Graylog root user is stored in the configuration file as a SHA256 hash. To generate the hash, use the second open terminal and run the following command.

Warning: The password will be displayed in plain text as you type it. Make sure no one is watching your screen.

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

Leave the http_bind_address option commented out to keep it at its default value.

Set http_publish_url to the HTTPS URL that will be used to access Graylog. For example, https://graylog.example.com/

Set elasticsearch_hosts to http://127.0.0.1:9200. The configuration file says this is the default value, but I have found that it must be explicitly set or Graylog will not find any data nodes.

Use the ctrl-w command in nano to locate the allow_leading_wildcard_searches option. Set allow_leading_wildcard_searches and allow_highlighting to true.

Save the changes to the file by pressing ctrl-o, and then enter. Press ctrl-x to exit nano.

Start Graylog by running the following commands.

sudo systemctl daemon-reload
sudo systemctl enable graylog-server
sudo systemctl start graylog-server

This will start a setup web interface, which is different than the normal web interface and uses different credentials. Even the root credentials you configured in server.conf will not work.

Configure NGINX as a reverse proxy

Graylog’s HTTP interfaces are configured by default to only bind to 127.0.0.1. That’s a good thing for security. NGINX can be used as a reverse proxy to accept HTTPS connections and route them to Graylog over the loopback interface.

Install NGINX using the following command.

sudo apt install -y nginx

Disable the default NGINX configuration.

sudo rm /etc/nginx/sites-enabled/default

Create a new NGINX configuration for Graylog. Replace graylog.example.com with the actual hostname that you wilol use to access Graylog in a web

sudo nano /etc/nginx/sites-available/graylog
server
{
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;
    server_name graylog.example.org;

    location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL https://$server_name/;
      proxy_pass http://127.0.0.1:9000;
    }
}

Enable the new NGINX configuration and restart NGINX.

sudo ln -s /etc/nginx/sites-available/graylog /etc/nginx/sites-enabled/graylog
sudo service nginx restart

Deploy HTTPS on NGINX with Let’s Encrypt DNS verification

Let’s Encrypt is a non-profit Certificate Authority that uses automation to verify domain ownership and issue free certificates that are honored by browsers and devices. The Electronic Frontier Foundation (EFF) created a tool called certbot that automatically obtains Let’s Encrypt certificates, configures popular server software to use them, and manages the certificate renewal process.

To obtain a certificate without exposing a web server to the internet, certbot has a variety of DNS plugins for many DNS nameserver hosting providers. These plugins use API credentials to add a TXT record to a DNS zone, which is then checked by Let’s Encrypt to verify domain ownership before issuing a certificate. You do not need to add a hostname to the public DNS zone. Instead, create an A record in a shadow DNS zone for your domain on a DNS server on the local network. That way, local systems can find the local IP address of the Graylog server.

Run the following commands to install certbot.

sudo apt install -y snapd
sudo snap install core
sudo snap install certbot
sudo snap set certbot trust-plugin-with-root=ok

Install the certbot DNS plugin for your nameserver hosting provider. For example, for GCP DNS run the following command.

sudo snap install certbot-dns-google

Follow the documentation of the plugin and run the certbot certonly command to obtain a certificate. The exact process and command arguments will vary depending on the plugin being used. Follow the documentation for that plugin. For example, to obtain a certificate for a domain with DNS nameservers hosted in GCP, the following command could be used.

sudo certbot certonly --dns-google -d graylog.example.com --dns-google-credentials gcp-dns-credentials.json

Once the certificate has been obtained, have certbot configure HTTPS on NGINX by running the following command. Replace graylog.example.com with the actual HTTPS hostname of the Graylog server.

sudo certbot --nginx -d graylog.example.com

certbot will state that a certificate already exists. Select the “Attempt to reinstall this existing certificate” option.

Congratulations! You now have a working single-node Graylog server with HTTPS configured. Log into Graylog using the Graylog root account that was configured earlier.

Create a FortiGate Syslog index set

In the Graylog web interface, navigate to System> Indices. Create an index set called FortiGate Syslog. Set a time-based index rotation of one day (P1D) and base the retention on the number on the number of days (i.e., indices) that you what to retain.

Install the FortiGate Syslog content packs

I have created two Graylog content packs for FortiGate syslog data. The first content pack, (FortiGate syslog) contains a stream and dashboard. A stream tells Graylog what data to direct to a particular index set or pipeline. Searches can also be filtered by stream. After you install the first content pack install the Graylog syslog pipeline content pack. This installs a pipeline that sets fields used by the dashboard to their proper datatype and removes redundant fields. The full original message is still available in the message field.

Configure the FortiGate Syslog stream to use the FortiGate Syslog index set

Once you have created the index set and installed the content packs, navigate to Streams, edit the FortiGate Syslog stream, select the FortiGate Syslog index set you created, and click Update Stream.

Prepare Graylog to accept logs from FortiGate firewalls

Create a self-signed certificate for accepting logs over TLS.

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/graylog/server/graylog-selfsigned.key -out /etc/graylog/server/graylog-selfsigned.crt
sudo chown graylog /etc/graylog/server/graylog-selfsigned.key
sudo chown graylog /etc/graylog/server/graylog-selfsigned.crt

Use a file transfer tool like Cyberduck to transfer a copy of the certificate at /etc/graylog/server/graylog-selfsigned.crt to your local system over SFTP. Log into your FortiGate web interface. Import the certificate to the FortiGate as a Remote CA certificate by navigating to System> Certificates> Create/Import> CA Certificate> File.

While still at the Certificates page, download a copy of the root CA certificate used by the firewall, which is named Fortinet_CA_SSL by default. This will be used by the Graylog server to authenticate the connection from the firewall, so unauthorized devices cannot send logs to the Graylog server. Use a file transfer tool to upload the certificate to the server, then run the following commands to move the certificate into place.

sudo mkdir -p /etc/graylog/server/trustedcerts.d
sudo mv Fortinet_CA_SSL.cer /etc/graylog/server/trustedcerts.d
sudo chown graylog -R /etc/graylog/server/trustedcerts.d
sudo chmod ugo=rX -R /etc/graylog/server/trustedcerts.d

Note: A previous version of this guide attempted to use the CEF log format. That turned out to be very buggy, so this content has been updated to use the default Syslog format, which works very well.

In Graylog, navigate to System> Indices. Create a new index for FortiGate logs with the title FortiGate Syslog, and the index prefix fortigate_syslog. Configure the index rotation and retention settings to match your needs. For example, to retain a year of logs set the rotation period to P1D and set the max number of indices to 365.

Download the FortiGate Syslog Graylog content pack JSON file by right-clicking on this link and clicking “Save link as.” Be sure to add yourself as a watcher to the GitHub project to be notified of new Content Pack releases that fix bugs or add more features.

In Graylog, navigate to System> Content Packs. Click Upload, choose the content_pack.json file, and click Upload. Click Install across from the FortiGate Syslog content pack in the list of content packs.

Navigate to Streams. Edit the FortiGate Syslog stream by clicking on More Actions> Edit Stream. Select the FortiGate Syslog index set, make sure “Remove matches from ‘Default Stream’” is checked, and Click Update Stream.

Navigate to System> Inputs. Launch a new Syslog TCP input.

Title: Syslog TCP 
Bind address: 0.0.0.0
Port: 6514 
Time zone: GMT
TLS cert file: /etc/graylog/server/graylog-selfsigned.crt
TLS private key file: /etc/graylog/server/graylog-selfsigned.key
Enable TLS: True
TLS client authentication: Required
TLS Client Auth Trusted Certs: /etc/graylog/server/trustedcerts.d

Configure FortiGate to send logs to Graylog

Use the CLI to configure the FortiGate.

To simplify and unify log management, it is important that every firewall be configured to use the GMT time zone, which for logging purposes is equivalent UTC. This ensures that times are consistent regardless of a firewall’s geographic location, or local factors such as daylight savings.

config system global
    set timezone "Etc/GMT"
end

By default, logs sent to the syslog server are not filtered. To ensure that the Graylog Input gets all logs, reset all log filter options to their default settings.

config log syslogd filter
    unset severity
    unset forward-traffic
    unset local-traffic
    unset multicast-traffic
    unset sniffer-traffic
    unset anomaly
    unset voip
end

Finally, configure the syslog output over TLS in Syslog format.

config log syslogd setting
    set status enable
    set server "graylog.example.com"
    set mode reliable
    set port 6514
    set format default
    set enc-algorithm high
    set certificate "Fortinet_CA_SSL"
end

Logs will begin to appear in your Graylog server.

Patching the server

To update all software on the server run the following command.

sudo apt update && sudo apt-dist upgrade -y && sudo apt autoremove -y

Server application updates do not take effect until the service is restarted. Linux kernel or firmware updates do not take effect until the system is rebooted. Stop the graylog-server service before restarting the mongodb or opensearch services, then start the graylog service again.

sudo service stop graylog-server
sudo service restart mongodb
sudo service restart opensearch
sudo service start graylog-server
sudo service nginx restart

Note: NGINX will return an HTTP 504 Bad Gateway error until Graylog fully starts.

Adding users

To add users to Graylog navigate to System> Users. User authentication via Active Directory or LDAP can be configured by navigating to System> Authentication. Check out the authentication documentation for more details.

Querying data

Log entries are called messages. Clicking on a result in a message table will show you all of the fields and values of that message. Clicking on any value in a dashboard or message will offer to temporarily include or exclude that value in the search filter. The time picker can be used to narrow the timeframe of search results and temporarily override the time window of dashboard widgets.

Graylog provides a powerful query syntax to provide more specific results.

One important difference between Graylog and other search engines is that Graylog does not search for substrings unless wildcards are explicitly used. For example, searching for google.com will only return results for connections to google.com, and not any google.com subdomains. To search for google.com and google .com subdomains, use *google.com. Likewise, to search for a substring, surround it with wildcards, such as *example*.

Commercial features

Graylog offers a commercial SIEM product called Graylog Security that includes features such as anomaly detection.

21 thoughts on “How to create a single-node Graylog instance and analyze FortiGate logs”

  1. Great write-up man.

    Couple of notes:
    * You included a part where you used letsencrypt/acmebot but never updated your Nginx to use it

    * You also generated a self signed cert for TLS inputs but you can use the one you generated with acmebot!

    * I love the dashboards, did you make them all? Lastly, you used a Stream Rule for directing the traffic, I strongly recommend a pipeline rule. Fire an email if you want a quick primer on how to do it.

    Great work, I love it.

    Abe
    Director PS & Training
    Graylog

    Reply
    • Wow! word got around quickly.

      > * You included a part where you used letsencrypt/acmebot but never updated your Nginx to use it
      The certbot –nginx command takes care of all of that for you

      > * You also generated a self signed cert for TLS inputs but you can use the one you generated with acmebot!

      Let’s Encrypt Certificates expire every few months and need to be renewed often. certbot takes care of the renewal process automatically, including reloading the web server configuration at the end. It does support adding renewal hooks for adding more services, but for Graylog I was concerned about the impact of reloading/restarting a listener while logs are streaming in. You don’t need to worry about any of that with a self-signed certificate.

      > * I love the dashboards, did you make them all? Lastly, you used a Stream Rule for directing the traffic, I strongly recommend a pipeline rule. Fire an email if you want a quick primer on how to do it.

      I just sent you an email.

      Reply
  2. hey man ,

    first i wanted to say , thank you for the great work you’ve done.

    im trying to follow step by step the guide here :
    but im ending with unreadable messages in Graylog :
    similar to this: �98��������Q=�5��+�/�����������\�`�V�R�#�’g@�r�v��� �32ED������P<�/A��

    my first though was the encoding was wrong , but im not sure.

    the only step im not doing is the TLS certificate , the communication happens in IP (so no FQDN is involved ), but i dont think this is problem , is it?

    looking forward to hear from you

    best regards
    Ervin

    Reply
    • Definitely looks like an encoding issue, not a TLS issue. I’m not sure what the cause could be. Try reaching out to Fortinet support and the Graylog community forums.

      Reply
      • it was an encryption problem. after fixing that it went straight forward.
        one thing im not getting on the dashboard is the IPS from the fortigate. any idea?

        Reply
  3. Great work here!

    One thing to be aware of: In Streams, I had to shorten the Fortigate Syslog stream rule from ^FGT to ^FG before it would catch logs from my FortiGate 101F hosts. For some reason those devid fields start with FGT101FT. A more inclusive regex expression might be ^FG([0-9]{2,3})[A-Z]T|^FGT

    Reply
  4. Hi Sean,

    Thank you for sharing this!

    We have some FortiGate’s that are virtual, and the serial number starts with FGVM, do we just edit the file where it says FGT to FGVM? because we are not seeing the logs come in on the server.

    Also, we are seeing on the graylog server, you have any idea what that means?

    ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/64403e99d67b386d1ca8a2fc] (channel [id: 0x75d1d980, L:/graylogserverIP:6514 ! R:/FTGVMFirewallIP:9540]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000070:SSL routines:OPENSSL_internal:BAD_PACKET_LENGTH)

    Reply
  5. Hi Sean, got an issue pretty early on with the procedure.

    After doing this part :
    sudo curl -sL -o /etc/apt/trusted.gpg.d/opensearch.gpg https://artifacts.opensearch.org/publickeys/opensearch.pgp
    echo “deb https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable main” | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list

    When I try to do the apt update, I get an error that the repository is unsigned along another error :
    GPG error: https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt/dists/stable/inRelease: The keys in the keyring /etc/apt/trusted.gpg.d/opensearch.gpg are ignored as the file has an unsupported filetype.

    Fresh install and followed without errors the guide up to that point.

    Any idea ?

    Reply
  6. Hi,
    I had the same issue as David, so I tried disabling TLS (on both sides), now I have the same issue Ervin:

    Did anyone find a solution?

    Reply
  7. no traffic arrives to graylog, if I configure the tcp port with cef format it does arrive, but follow the steps in the guide and I can’t get the traffic to arrive via tcp tls

    Reply
  8. Any idea how we get the DNS queries from our DNS filter into Graylog? We are seeing events on our fortigates but nothing is coming towards the Graylog.

    Thanks!

    Reply
  9. Hi, I’ve used your content pack for a while, but I noticed that all fields are string, so I can’t make any Sum in dashboard (for example a table with sentbyte, rcvdbyte for every policy). Have I make something wrong?

    Reply
  10. After importing the content pack, I don’t see an option in Editing the Stream forthe FortiGate Syslog index set; I only see the default index set. Is there a way to debug why the index set is missing?

    Reply
  11. Dear All,
    I recently had a graylog server on Linux and first input was my perimeter fortygate firewall . I was having issues installing the Fortigate content pack
    Thanks to sean for the new revision FG content pack
    So i create a Fresh Install of the below
    Rocky Linux 8
    graylog-enterprise-5.2.5-1.x86_64
    fortigate syslog content pack 1.6.5 rev 26
    graylog-fortigate-syslog-pipeline 1.0.5 rev 7
    I had no issues in the above install

    I have 2 fortigates 1500D HA active /active mode

    serial no of primary firewall is XXXX1477
    serial no of secondary firewall is XXXX1393

    Now what I see under streams tab is that no messages from the default stream get routed to the fortigate syslog Index set as shown in the attached picture .
    I also have only one input that is my fortygate syslog configured in my graylog server
    Attached is a sanpshot of my streams tab
    If any more information is required please do ask i really appreciate it

    Thanks and regards

    Simon

    Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.