Merge ~kwmonroe/graylog-snap:master into ~canonical-is-sa/graylog-snap:master

[Kevin W Monroe]

update to 3.0-latest

Graylog 3 is needed to facilitate rich Kubernetes logging (logs annotated with k8s pods/containers/labels/etc). I've tested this with CDK using graylog from the 2.5.1 snap and manually upgrading the system with the deb and config changes in this MP.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[Kevin W Monroe (kwmonroe)]

Bug 1824708 has more details about the caveats of this upgrade. We may need to work out snap build changes prior to merging this. That is, create and maintain tracks for v2 and v3 for users that do no wish to make the jump to this new major version.

[Stuart Bishop (stub)]

LGTM.

Does server.conf need to exist in this branch, or can it be pulled at build time from the graylog package?

review: Approve

[Junien F (axino)]

+1

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision e417d0c9e4378f984d6a641afbd80a6abebe3590

diff --git a/log4j2.xml b/log4j2.xml
index 743bce0..0c322f2 100644
--- a/log4j2.xml
+++ b/log4j2.xml
@@ -22,8 +22,6 @@
         <Logger name="org.elasticsearch.script" level="warn"/>
         <!-- Disable messages from the version check -->
         <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
-        <!-- Suppress crazy byte array dump of Drools -->
-        <Logger name="org.drools.compiler.kie.builder.impl.KieRepositoryImpl" level="warn"/>
         <!-- Silence chatty natty -->
         <Logger name="com.joestelmach.natty.Parser" level="warn"/>
         <!-- Silence Kafka log chatter -->
diff --git a/server.conf b/server.conf
index 02790fb..0c0bf78 100644
--- a/server.conf
+++ b/server.conf
@@ -73,102 +73,110 @@ root_password_sha2 = ROOT_PASSWORD
 # Default is UTC
 #root_timezone = UTC
 
+# Set the bin directory here (relative or absolute)
+# This directory contains binaries that are used by the Graylog server.
+# Default: bin
+bin_dir = /snap/graylog/current/usr/share/graylog-server/bin
+
+# Set the data directory here (relative or absolute)
+# This directory is used to store Graylog server state.
+# Default: data
+data_dir = /var/snap/graylog/common
+
 # Set plugin directory here (relative or absolute)
 plugin_dir = /snap/graylog/current/usr/share/graylog-server/plugin
 
-# REST API listen URI. Must be reachable by other Graylog server nodes if you run a cluster.
-# When using Graylog Collectors, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
-rest_listen_uri = http://0.0.0.0:9001/api/
+###############
+# HTTP settings
+###############
+
+#### HTTP bind address
+#
+# The network interface used by the Graylog HTTP interface.
+#
+# This network interface must be accessible by all Graylog nodes in the cluster and by all clients
+# using the Graylog web interface.
+#
+# If the port is omitted, Graylog will use port 9000 by default.
+#
+# Default: 127.0.0.1:9000
+http_bind_address = 127.0.0.1:9000
+#http_bind_address = [2001:db8::1]:9000
 
-# REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri
-# is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used.
-# If set, this will be promoted in the cluster discovery APIs, so other nodes may try to connect on
-# this address and it is used to generate URLs addressing entities in the REST API. (see rest_listen_uri)
-# You will need to define this, if your Graylog server is running behind a HTTP proxy that is rewriting
-# the scheme, host name or URI.
-# This must not contain a wildcard address (0.0.0.0).
-#rest_transport_uri = http://192.168.1.1:9000/api/
+#### HTTP publish URI
+#
+# The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all
+# clients using the Graylog web interface.
+#
+# The URI will be published in the cluster discovery APIs, so that other Graylog nodes will be able to find and connect to this Graylog node.
+#
+# This configuration setting has to be used if this Graylog node is available on another network interface than $http_bind_address,
+# for example if the machine has multiple network interfaces or is behind a NAT gateway.
+#
+# If $http_bind_address contains a wildcard IPv4 address (0.0.0.0), the first non-loopback IPv4 address of this machine will be used.
+# This configuration setting *must not* contain a wildcard address!
+#
+# Default: http://$http_bind_address/
+#http_publish_uri = http://192.168.1.1:9000/
 
-# Enable CORS headers for REST API. This is necessary for JS-clients accessing the server directly.
+#### External Graylog URI
+#
+# The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API.
+#
+# The external Graylog URI usually has to be specified, if Graylog is running behind a reverse proxy or load-balancer
+# and it will be used to generate URLs addressing entities in the Graylog REST API (see $http_bind_address).
+#
+# When using Graylog Collector, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
+#
+# This setting can be overriden on a per-request basis with the "X-Graylog-Server-URL" HTTP request header.
+#
+# Default: $http_publish_uri
+#http_external_uri =
+
+#### Enable CORS headers for HTTP interface
+#
+# This is necessary for JS-clients accessing the server directly.
 # If these are disabled, modern browsers will not be able to retrieve resources from the server.
 # This is enabled by default. Uncomment the next line to disable it.
-#rest_enable_cors = false
+#http_enable_cors = false
 
-# Enable GZIP support for REST API. This compresses API responses and therefore helps to reduce
+#### Enable GZIP support for HTTP interface
+#
+# This compresses API responses and therefore helps to reduce
 # overall round trip times. This is enabled by default. Uncomment the next line to disable it.
-#rest_enable_gzip = false
+#http_enable_gzip = false
 
-# Enable HTTPS support for the REST API. This secures the communication with the REST API with
-# TLS to prevent request forgery and eavesdropping. This is disabled by default. Uncomment the
-# next line to enable it.
-#rest_enable_tls = true
+# The maximum size of the HTTP request headers in bytes.
+#http_max_header_size = 8192
 
-# The X.509 certificate chain file in PEM format to use for securing the REST API.
-#rest_tls_cert_file = /path/to/graylog.crt
+# The size of the thread pool used exclusively for serving the HTTP interface.
+#http_thread_pool_size = 16
 
-# The PKCS#8 private key file in PEM format to use for securing the REST API.
-#rest_tls_key_file = /path/to/graylog.key
+################
+# HTTPS settings
+################
 
-# The password to unlock the private key used for securing the REST API.
-#rest_tls_key_password = secret
+#### Enable HTTPS support for the HTTP interface
+#
+# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
+#
+# Default: false
+#http_enable_tls = true
 
-# The maximum size of the HTTP request headers in bytes.
-#rest_max_header_size = 8192
+# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
+#http_tls_cert_file = /path/to/graylog.crt
+
+# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
+#http_tls_key_file = /path/to/graylog.key
 
-# The maximal length of the initial HTTP/1.1 line in bytes.
-#rest_max_initial_line_length = 4096
+# The password to unlock the private key used for securing the HTTP interface.
+#http_tls_key_password = secret
 
-# The size of the thread pool used exclusively for serving the REST API.
-#rest_thread_pool_size = 16
 
 # Comma separated list of trusted proxies that are allowed to set the client address with X-Forwarded-For
 # header. May be subnets, or hosts.
 #trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128
 
-# Enable the embedded Graylog web interface.
-# Default: true
-#web_enable = false
-
-# Web interface listen URI.
-# Configuring a path for the URI here effectively prefixes all URIs in the web interface. This is a replacement
-# for the application.context configuration parameter in pre-2.0 versions of the Graylog web interface.
-web_listen_uri = http://127.0.0.1:9000/
-
-# Web interface endpoint URI. This setting can be overriden on a per-request basis with the X-Graylog-Server-URL header.
-# Default: $rest_transport_uri
-#web_endpoint_uri =
-
-# Enable CORS headers for the web interface. This is necessary for JS-clients accessing the server directly.
-# If these are disabled, modern browsers will not be able to retrieve resources from the server.
-#web_enable_cors = false
-
-# Enable/disable GZIP support for the web interface. This compresses HTTP responses and therefore helps to reduce
-# overall round trip times. This is enabled by default. Uncomment the next line to disable it.
-#web_enable_gzip = false
-
-# Enable HTTPS support for the web interface. This secures the communication of the web browser with the web interface
-# using TLS to prevent request forgery and eavesdropping.
-# This is disabled by default. Uncomment the next line to enable it and see the other related configuration settings.
-#web_enable_tls = true
-
-# The X.509 certificate chain file in PEM format to use for securing the web interface.
-#web_tls_cert_file = /path/to/graylog-web.crt
-
-# The PKCS#8 private key file in PEM format to use for securing the web interface.
-#web_tls_key_file = /path/to/graylog-web.key
-
-# The password to unlock the private key used for securing the web interface.
-#web_tls_key_password = secret
-
-# The maximum size of the HTTP request headers in bytes.
-#web_max_header_size = 8192
-
-# The maximal length of the initial HTTP/1.1 line in bytes.
-#web_max_initial_line_length = 4096
-
-# The size of the thread pool used exclusively for serving the web interface.
-#web_thread_pool_size = 16
-
 # List of Elasticsearch hosts Graylog should connect to.
 # Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
 # If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that
@@ -242,6 +250,9 @@ web_listen_uri = http://127.0.0.1:9000/
 #
 # ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
 #            to your previous 1.x settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 rotation_strategy = count
 
 # (Approximate) maximum number of documents in an Elasticsearch index before a new index
@@ -250,6 +261,9 @@ rotation_strategy = count
 #
 # ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
 #            to your previous 1.x settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 elasticsearch_max_docs_per_index = 20000000
 
 # (Approximate) maximum size in bytes per Elasticsearch index on disk before a new index is being created, also see
@@ -258,6 +272,9 @@ elasticsearch_max_docs_per_index = 20000000
 #
 # ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
 #            to your previous 1.x settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 #elasticsearch_max_size_per_index = 1073741824
 
 # (Approximate) maximum time before a new Elasticsearch index is being created, also see
@@ -273,6 +290,9 @@ elasticsearch_max_docs_per_index = 20000000
 #
 # ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
 #            to your previous 1.x settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 #elasticsearch_max_time_per_index = 1d
 
 # Disable checking the version of Elasticsearch for being compatible with this Graylog release.
@@ -286,6 +306,9 @@ elasticsearch_max_docs_per_index = 20000000
 #
 # ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
 #            to your previous 1.x settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 elasticsearch_max_number_of_indices = 20
 
 # Decide what happens with the oldest indices when the maximum number of indices is reached.
@@ -295,11 +318,17 @@ elasticsearch_max_number_of_indices = 20
 #
 # ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
 #            to your previous 1.x settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 retention_strategy = delete
 
 # How many Elasticsearch shards and replicas should be used per index? Note that this only applies to newly created indices.
 # ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
 #            to your previous settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 elasticsearch_shards = 4
 elasticsearch_replicas = 0
 
@@ -307,6 +336,9 @@ elasticsearch_replicas = 0
 #
 # ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
 #            to your previous settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 elasticsearch_index_prefix = graylog
 
 # Name of the Elasticsearch index template used by Graylog to apply the mandatory index mapping.
@@ -314,6 +346,9 @@ elasticsearch_index_prefix = graylog
 #
 # ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
 #            to your previous settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 #elasticsearch_template_name = graylog-internal
 
 # Do you want to allow searches with leading wildcards? This can be extremely resource hungry and should only
@@ -331,6 +366,9 @@ allow_highlighting = false
 #
 # ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
 #            to your previous settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 elasticsearch_analyzer = standard
 
 # Global request timeout for Elasticsearch requests (e. g. during search, index creation, or index time-range
@@ -352,6 +390,11 @@ elasticsearch_analyzer = standard
 # Default: 1h
 #index_ranges_cleanup_interval = 1h
 
+# Time interval for the job that runs index field type maintenance tasks like cleaning up stale entries. This doesn't
+# need to run very often.
+# Default: 1h
+#index_field_type_periodical_interval = 1h
+
 # Batch size for the Elasticsearch output. This is the maximum (!) number of messages the Elasticsearch output
 # module will get at once and write to Elasticsearch in a batch call. If the configured batch size has not been
 # reached within output_flush_interval seconds, everything that is available will be flushed at once. Remember
@@ -375,8 +418,20 @@ output_fault_penalty_seconds = 30
 processbuffer_processors = 5
 outputbuffer_processors = 3
 
+# The following settings (outputbuffer_processor_*) configure the thread pools backing each output buffer processor.
+# See https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html for technical details
+
+# When the number of threads is greater than the core (see outputbuffer_processor_threads_core_pool_size),
+# this is the maximum time in milliseconds that excess idle threads will wait for new tasks before terminating.
+# Default: 5000
 #outputbuffer_processor_keep_alive_time = 5000
+
+# The number of threads to keep in the pool, even if they are idle, unless allowCoreThreadTimeOut is set
+# Default: 3
 #outputbuffer_processor_threads_core_pool_size = 3
+
+# The maximum number of threads to allow in the pool
+# Default: 30
 #outputbuffer_processor_threads_max_pool_size = 30
 
 # UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).
@@ -469,6 +524,7 @@ lb_recognition_period_seconds = 3
 mongodb_uri = mongodb://localhost/graylog
 
 # Authenticate against the MongoDB server
+# '+'-signs in the username or password need to be replaced by '%2B'
 #mongodb_uri = mongodb://grayloguser:secret@localhost:27017/graylog
 
 # Use a replica set instead of a single host
@@ -484,22 +540,30 @@ mongodb_max_connections = 1000
 # http://api.mongodb.com/java/current/com/mongodb/MongoOptions.html#threadsAllowedToBlockForConnectionMultiplier
 mongodb_threads_allowed_to_block_multiplier = 5
 
-# Drools Rule File (Use to rewrite incoming log messages)
-# See: http://docs.graylog.org/en/2.1/pages/drools.html
-#rules_file = /etc/graylog/server/rules.drl
 
 # Email transport
 #transport_email_enabled = false
 #transport_email_hostname = mail.example.com
 #transport_email_port = 587
 #transport_email_use_auth = true
-#transport_email_use_tls = true
-#transport_email_use_ssl = true
 #transport_email_auth_username = [e-mail]
 #transport_email_auth_password = secret
 #transport_email_subject_prefix = [graylog]
 #transport_email_from_email = [e-mail]
 
+# Encryption settings
+#
+# ATTENTION:
+#    Using SMTP with STARTTLS *and* SMTPS at the same time is *not* possible.
+
+# Use SMTP with STARTTLS, see https://en.wikipedia.org/wiki/Opportunistic_TLS
+#transport_email_use_tls = true
+
+# Use SMTP over SSL (SMTPS), see https://en.wikipedia.org/wiki/SMTPS
+# This is deprecated on most SMTP services!
+#transport_email_use_ssl = true
+
+
 # Specify and uncomment this if you want to include links to the stream in your stream alert mails.
 # This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.
 #transport_email_web_interface_url = https://graylog.example.com
@@ -520,14 +584,30 @@ mongodb_threads_allowed_to_block_multiplier = 5
 #http_write_timeout = 10s
 
 # HTTP proxy for outgoing HTTP connections
+# ATTENTION: If you configure a proxy, make sure to also configure the "http_non_proxy_hosts" option so internal
+#            HTTP connections with other nodes does not go through the proxy.
+# Examples:
+#   - http://proxy.example.com:8123
+#   - http://username:[e-mail]:8123
 #http_proxy_uri =
 
+# A list of hosts that should be reached directly, bypassing the configured proxy server.
+# This is a list of patterns separated by ",". The patterns may start or end with a "*" for wildcards.
+# Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.
+# Examples:
+#   - localhost,127.0.0.1
+#   - 10.0.*,*.example.com
+#http_non_proxy_hosts =
+
 # Disable the optimization of Elasticsearch indices after index cycling. This may take some load from Elasticsearch
 # on heavily used systems with large indices, but it will decrease search performance. The default is to optimize
 # cycled indices.
 #
 # ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
 #            to your previous settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 #disable_index_optimization = true
 
 # Optimize the index down to <= index_optimization_max_num_segments. A higher number may take some load from Elasticsearch
@@ -535,6 +615,9 @@ mongodb_threads_allowed_to_block_multiplier = 5
 #
 # ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
 #            to your previous settings so they will be migrated to the database!
+#            This configuration setting is only used on the first start of Graylog. After that,
+#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
+#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
 #index_optimization_max_num_segments = 1
 
 # The threshold of the garbage collection runs. If GC runs take longer than this threshold, a system notification
@@ -550,18 +633,7 @@ mongodb_threads_allowed_to_block_multiplier = 5
 # The default cache time for dashboard widgets. (Default: 10 seconds, minimum: 1 second)
 #dashboard_widget_default_cache_time = 10s
 
-# Automatically load content packs in "content_packs_dir" on the first start of Graylog.
-#content_packs_loader_enabled = true
-
-# The directory which contains content packs which should be loaded on the first start of Graylog.
-content_packs_dir = /snap/graylog/current/usr/share/graylog-server/contentpacks
-
-# A comma-separated list of content packs (files in "content_packs_dir") which should be applied on
-# the first start of Graylog.
-# Default: empty
-content_packs_auto_load = grok-patterns.json
-
 # For some cluster-related REST requests, the node must query all other nodes in the cluster. This is the maximum number
 # of threads available for this. Increase it, if '/cluster/*' requests take long to complete.
-# Should be rest_thread_pool_size * average_cluster_size if you have a high number of concurrent users.
+# Should be http_thread_pool_size * average_cluster_size if you have a high number of concurrent users.
 proxied_requests_thread_pool_size = 32
diff --git a/snapcraft.yaml b/snapcraft.yaml
index 23a5b23..f46a0af 100644
--- a/snapcraft.yaml
+++ b/snapcraft.yaml
@@ -1,5 +1,5 @@
 name: graylog
-version: 2.5.1
+version: 3.0.1
 summary: Graylog server snap, https://www.graylog.org/
 description: |
   Graylog is a full-featured log management system allowing for centralized
@@ -25,8 +25,8 @@ parts:
   graylog-repository:
     plugin: nil
     build: |
-      wget https://packages.graylog2.org/repo/packages/graylog-2.5-repository_latest.deb
-      dpkg -i graylog-2.5-repository_latest.deb
+      wget https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.deb
+      dpkg -i graylog-3.0-repository_latest.deb
       apt update
     build-packages:
       - wget