Event Flow Control in Agents: Buffering, Queueing and Data Loss Considerations – Guardsix Servicedesk

AgentX and Logpoint Agent Standalone include mechanisms to handle situations where event volume temporarily exceeds what the SIEM or network can handle. These mechanisms are primarily meant for flow control and protection against data loss.

Buffering (available in both agents) helps smooth out bursts of events while Queueing (available in Logpoint Agent Standalone) helps prevent data loss.

AgentX

Buffering in AgentX is a mechanism meant to control the flow of events and smooth out the bursts of events. It can be configured from the Configuration → AgentX → Global Settings page. The following parameters are available when buffering is enabled:

Max EPS: Maximum number of events per second forwarded to the SIEM
Buffer Size: Maximum number of events that can be held in memory

Internally, AgentX implements buffering using an in-memory leaky bucket algorithm. Events are forwarded at a controlled rate defined by Max EPS. If incoming event rate exceeds this limit, excess events are temporarily stored in the buffer. If the incoming rate remains high and the buffer reaches its capacity, new events might not be forwarded to the SIEM server. In such cases, the agent logs a message : Agent buffer flooded in the ossec.log file, indicating that the buffer has been exhausted.

It is important to note that this buffer is an in-memory construct and does not provide any form of persistence. Buffered events are lost if the agent is restarted or if the service crashes. Additionally, during a disconnection from the SIEM server, the buffering mechanism is not used as a fallback. Instead, log collection modules may be blocked from generating further events. For this reason, AgentX buffering should be viewed purely as a flow control mechanism rather than a reliability feature.

In practice, this mechanism is useful for controlling short-lived spikes in event volume and ensuring that the SIEM is not overwhelmed. However, it is not suitable in environments where data loss during outages is unacceptable.

Logpoint Agent (Standalone)

Logpoint Agent separates flow control and reliability into two distinct mechanisms: buffering and queueing. This provides more flexibility compared to AgentX, particularly in scenarios where event delivery guarantees are important.

1. Buffering

Buffering in Logpoint Agent is implemented through batch-based processing rather than a rate-limiting algorithm. Events are grouped into batches and forwarded together, with a default batch size of 50 events. This behavior can be adjusted using the BatchSize directive.

<Output out>
   ...
   BatchSize   100
   ...
</Output>

By controlling how many events are processed at a time, the agent regulates the flow of events to different internal modules and to the SIEM.

In addition to this general batching mechanism, certain modules implement their own internal buffers depending on their function. For example, UDP-based modules rely on socket buffer sizes. More information on this can be found here.

As with AgentX, buffering in Logpoint Agent Standalone does not provide data durability. It helps regulate throughput but does not guarantee that events will be preserved in the event of a failure or outage. However, Logpoint Agent Standalone also support Queueing which can be used for this purpose.

2. Queueing

Queueing in Logpoint Agent Standalone is designed to handle situations where events cannot be processed or forwarded immediately, such as during SIEM unavailability or when the incoming event rate exceeds processing capacity. Instead of dropping events, they are placed into a queue for later processing.

The queue size can be controlled using the LogQueuesize directive inside the module.

<Output out>
    ...
    LogQueueSize    <number_of_events>
    ...
</Output>

By default, the queue is maintained in memory. This provides fast performance but does not protect against data loss in the event of a restart or crash.

To enable persistence, the queue can be configured to use disk storage using the following global directives.

# To ensure that queued events are stored on disk
PersistLogqueue TRUE
# To force immediate synchronization of queued events to disk, reducing the risk of data loss.
SyncLogqueue    TRUE

Recommendations

In most environments, buffering is sufficient for handling short bursts of traffic. However, if the SIEM may become temporarily unavailable or if data loss is not acceptable, persistent queueing should be enabled.

When enabling disk-based queues, it is also important to ensure that sufficient disk space is available, as prolonged outages or sustained high event rates can cause the queue to grow significantly.

References: