Skip to content

fix(logrotate): rotate by rename and reopen instead of copytruncate - #1960

Draft
mrrobot47 wants to merge 7 commits into
EasyEngine:developfrom
mrrobot47:fix/logrotate-reopen-instead-of-copytruncate
Draft

mrrobot47 wants to merge 7 commits into
EasyEngine:developfrom
mrrobot47:fix/logrotate-reopen-instead-of-copytruncate

Conversation

@mrrobot47

Copy link
Copy Markdown
Member

Stops log rotation from failing for good once a log is bigger than the free disk space. Stacked on #1958, so merge that one first. Until then this PR's diff also shows #1958's commits, and the change here is the last commit only.

Problem

All three rules EasyEngine installs (ee_nginx_proxy, ee_sites, ee_cli) use copytruncate. logrotate copies the whole live log to .1 and only then truncates it, so a rotation needs as much free space as the log itself. On a busy server the nginx-proxy access.log reached about 20 GB. One night the copy didn't fit, logrotate logged error copying … No space left on device and left the log in place, and the same thing happened every night after that while the log kept growing. Each failed copy also filled the disk for several minutes around midnight, so backups and databases running at that moment failed. Nothing recovers from that state without a manual truncate.

#1958 adds maxsize and an hourly run, which keeps logs small in normal operation. It doesn't fix this failure: if free space ever drops below the log size, copytruncate fails, and the log grows past maxsize from then on.

What changes

The three rules rename the log and have the writer reopen it, which needs no extra space:

  • ee_nginx_proxy runs nginx -s reopen in the container whose compose service is global-nginx-proxy, found with docker ps -f label=com.docker.compose.service=global-nginx-proxy instead of a hard-coded container name. It can't use docker kill -s USR1, because PID 1 in the nginx-proxy image is forego, not nginx.
  • ee_sites sends USR1 to each site's nginx and php containers (io.easyengine.site label plus the compose service name). In both images the master process is PID 1 (openresty, and php-fpm through an exec entrypoint), and both reopen their logs on USR1.
  • ee_cli just drops copytruncate. Each ee run opens ee.log again, so nothing needs a signal.

delaycompress stays, so the renamed file isn't compressed until the next rotation, after the writer has moved to the new file. The postrotate scripts end in || true: a stopped proxy or site has nothing writing to its log, so a failed signal there is harmless.

Existing installs pick up the new rules on the next ee cli update, because the setup step rewrites any EasyEngine rule whose content differs. No migration is needed.

Not in this PR

  • Lower retention or dateext. rotate 30 with compressed archives isn't what fills disks. fix(logrotate): cap host syslog and mail.log growth #1958 already sets dateformat for hosts that enable dateext.
  • Reporting a failed rotation. logrotate still exits non-zero and logs to the journal, as before.

Testing

Throwaway containers only: a docker:dind container acted as the EasyEngine host, so the postrotate label filters only ever matched the test containers. Images are the ones img-versions.json pins: easyengine/nginx-proxy:v4.13.0, easyengine/nginx:v4.11.0 and easyengine/php8.3:v4.13.3. The rules were rendered from Utils.php before (#1958 head) and after this change, and run with the logrotate from the docker:dind image (Alpine).

  • php -l is clean on PHP 7.4 and 8.5, and logrotate -d accepts all three new rules (Ubuntu 24.04).
  • Proxy logs on a 64 MB tmpfs holding a 40 MB access.log (24 MB free):
    • Old rule: error copying …/access.log to …/access.log.1: No space left on device, exit 1, log left in place.
    • New rule: exit 0, access.log renamed to access.log.1. nginx's open files point at the new access.log, and a request made after the rotation is logged there and not in .1.
  • maxsize with a non-forced run (what the hourly job from fix(logrotate): cap host syslog and mail.log growth #1958 does): right after the forced rotation, the run reports log does not need rotating (log has already been rotated). Once access.log is 1.1 GB, the next run reports log needs rotating and renames it, and new requests go to the new file.
  • Site rule: a forced rotation renames the site's nginx and php-fpm logs, and after the USR1 both containers' PID 1 hold the new files open.

Proxy rule, old and new:

proxy.png

Site rule:

sites.png

Postfix containers log through the host's /dev/log, so a logging loop lands in /var/log/syslog and /var/log/mail.log. Distros rotate those weekly with no size limit, and logrotate runs once a day, so a flood can fill the disk between runs and take the global DB down with it.

A one-time migration now adds `maxsize 2G` to every stanza of the rsyslog logrotate rule that has no size limit yet, keeping the rest of the file as it is (a stanza or file that already sets maxsize or size is left alone, an unparseable or symlinked file is left alone with a warning, and the result must pass `logrotate -d` with no new errors). The original is saved under the EasyEngine backup dir.

It also makes logrotate run hourly so the limit is checked in time: a logrotate.timer drop-in on systemd hosts (a stopped timer stays stopped), or an hourly cron job that runs the daily logrotate script on hosts that use cron. An existing drop-in or job is left alone.
The ee_cli, ee_sites and ee_nginx_proxy rules rotate daily with no size limit. With logrotate now running hourly, `maxsize 1G` rotates a runaway CLI, site or proxy log within the hour instead of letting it grow until the next day.
With dateext and the default day-only suffix, a second rotation on the same day fails with "destination already exists", so maxsize stopped capping a flood after the first rotation of the day (RHEL-family hosts enable dateext in logrotate.conf by default).

The host stanzas that get maxsize also get `dateformat -%Y%m%d-%s` unless the stanza or file sets its own dateformat, and EasyEngine's own rules get the same. It changes nothing when dateext is off, and keeps dated archive names when it is on.
…p cron job

The inserted lines shift the line numbers of errors the original rule already has, so an unrelated pre-existing error made the edit look like it added a new one and the cap was refused. Line numbers are now masked along with the path.

On a systemd host without a usable logrotate.timer (e.g. masked), the packaged cron.daily script exits early because systemd runs, so the hourly job would never rotate anything. EasyEngine now warns that maxsize is only checked when logrotate runs instead of installing that job.
copy() created the backup with the default umask, so a rule restricted to root (which may hold credentials in its scripts) was saved world-readable. The backup dir is now 0700 and the copy 0600.
copytruncate copies the whole log before truncating it, so a rotation needs as much free space as the log. Once a log is bigger than the free space, every run fails with "No space left on device", leaves the log in place, and the log grows until the disk is full. Rotation never recovers on its own.

All three EasyEngine rules now rename the log and tell the writer to reopen it, which needs no extra space:

- ee_nginx_proxy: `nginx -s reopen` inside the container whose compose service is global-nginx-proxy. A signal to the container would go to forego (PID 1), not nginx.
- ee_sites: USR1 to every site's nginx and php container (io.easyengine.site label), where nginx and php-fpm are PID 1 and reopen their logs on USR1.
- ee_cli: no signal needed, since each ee run opens ee.log again.

Existing installs get the new rules on the next update, as the setup step rewrites any rule that differs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant