Aug 26
In collectd, what does the WriteQueueLimit do?
When you read this you probably already know what collectd is. An old-skool system daemon that collects various performance metrics on servers and logs them, usually to RRD statistics files. In combination with collectd-web, you get a nice monitoring solutions with a no-nonsense web GUI. collectd is written in C, collectd-web in wonderful Perl. Both are somewhat out of fashion yet still a popular choice among Arnold-type sysadmins and still occasionally maintained as of 2025.
As collectd with a number of plugins enabled can itself become an IO burden for a system, here are some important switches to toggle: the two WriteQueueLimit parameters - named WriteQueueLimitLow and WriteQueueLimitHigh - control how much data can be buffered in memory before being written by output plugins. They help prevent uncontrolled memory growth when output plugins (like write_graphite, write_http, etc.) fail to send data to their destination or have a lot data to write. Here's how they work:
- WriteQueueLimitHigh: This sets the upper threshold of the write queue. Once this limit is reached, collectd starts dropping metrics to avoid memory bloat.
- WriteQueueLimitLow: This is the lower threshold. Once the the write queue size drops below this value again, collectd resumes accepting new metrics.
Think of it these as buffer control parameters: If the queue grows too large (hits High), collectd stops accepting new data. Once the queue shrinks (below Low), it resumes normal operation.
Out of the box, these vars are not defined, meaning collectd (or better its queue) will grow indefinitely. This can be a problem with buggy plugins or on busy servers. In the past, this was notably seen in the plugin_dispatch_multivalue() function, which didn’t obey these limits and caused memory consumption to spiral. So make sure you think over these values and at least enable them to go with the defaults in collectd.conf.
Further reading:
Collectd on Wikipedia
collectd github repo
collectd-web github repo
List of available plugins and short descriptions