Lua script optimization & Configure Logrotate

[astarinmymind]

Lua script:

• Groups the logs into hashtable by a second granularity, where keys are buckets that keep one second worth of logs
• Example: {1: [log_record1, log_record2], 2: [log_record3, log_record4]}
• On each new log & call to filter function:
  • checks if already ran the code for this second and exits if true
  • flushes all log buckets where bucket timestamp <= current timestamp - 120 seconds

Logrotate:

• Appends to logrotate added in meta-confidential-compute
• Rotates logs in /searcher_logs/bob.log and /delayed_logs daily, keeping 7 old copies in their folder
• Context: Searcher writes to /searcher_logs in container, which fluentbit forwards and delays to /delayed_logs.

root@tdx:~# logrotate -f /etc/logrotate.conf
root@tdx:~# ls /delayed_logs
output.log              output.log-20250128.gz
root@tdx:~# ls /searcher_logs
bob.log              bob.log-20250128.gz
root@tdx:~# 

Pins fluent-bit version to amd64-specific digest (not the multi-arch!)

podman manifest inspect docker.io/fluent/fluent-bit@sha256:a185ac0516e1f355
68ff0662f12c4ada0ea38c4300ed223d0fde485599dff5b5

"digest": "sha256:d2e5b2fe876ca343ff68e431a62249caba2f5300bd340145b1f30bdacedf3a6a",
"platform": {
    "architecture": "amd64",
    "os": "linux"
}

[bakhtin]

Copying records one by one into a new table sounds like a lot of overhead. Esp. on large volumes.

Not sure how to properly do it in Lua but I suggest instead of copying each record one by one, consider storing only indices of the buckets that you want to emit, concatenate the tables, and emit them all together later in the code.