Fabric proxy (ZooKeeper) on large deployments may crash due to insufficient heap space

If the Fabric proxy (ZooKeeper) service on a Fabric server exhausts its allocated heap memory, it crashes. Each crash writes a heap dump file to the server filesystem, /etc/service/zookeeper. Until Fleet v2.11.1, heap dumps were not rotated or removed automatically, and each dump is approximately the size of the configured heap, around 2 GB at the default setting. Repeated crashes accumulate dumps until the disk fills, at which point the Fabric server stops functioning.

In Director/Fleet v2.8.0, default heap memory was raised from 1 GB to 2 GB, but 2 GB can still be insufficient.

If your Fabric server is affected by this issue, you may see the following signs:

• Fabric servers repeatedly restart, become unreachable, or appear offline.

• Requests in Fleet management or Federated investigations fail or time out. This includes log source management, configuration changes, API calls, or investigations across connected SIEMs.

• If your Fabric server is already affected: Contact Guardsix Support to increase the heap size to an appropriate value for your environment. The heap size can only be adjusted by the Support team.

• To prevent this issue: Proactively contact Guardsix Support to evaluate your memory usage and increase the heap size if necessary.