Repository navigation
Conversation
Postfix containers log through the host's /dev/log, so a logging loop lands in /var/log/syslog and /var/log/mail.log. Distros rotate those weekly with no size limit, and logrotate runs once a day, so a flood can fill the disk between runs and take the global DB down with it. A one-time migration now adds `maxsize 2G` to every stanza of the rsyslog logrotate rule that has no size limit yet, keeping the rest of the file as it is (a stanza or file that already sets maxsize or size is left alone, an unparseable or symlinked file is left alone with a warning, and the result must pass `logrotate -d` with no new errors). The original is saved under the EasyEngine backup dir. It also makes logrotate run hourly so the limit is checked in time: a logrotate.timer drop-in on systemd hosts (a stopped timer stays stopped), or an hourly cron job that runs the daily logrotate script on hosts that use cron. An existing drop-in or job is left alone.
The ee_cli, ee_sites and ee_nginx_proxy rules rotate daily with no size limit. With logrotate now running hourly, `maxsize 1G` rotates a runaway CLI, site or proxy log within the hour instead of letting it grow until the next day.
With dateext and the default day-only suffix, a second rotation on the same day fails with "destination already exists", so maxsize stopped capping a flood after the first rotation of the day (RHEL-family hosts enable dateext in logrotate.conf by default). The host stanzas that get maxsize also get `dateformat -%Y%m%d-%s` unless the stanza or file sets its own dateformat, and EasyEngine's own rules get the same. It changes nothing when dateext is off, and keeps dated archive names when it is on.
…p cron job The inserted lines shift the line numbers of errors the original rule already has, so an unrelated pre-existing error made the edit look like it added a new one and the cap was refused. Line numbers are now masked along with the path. On a systemd host without a usable logrotate.timer (e.g. masked), the packaged cron.daily script exits early because systemd runs, so the hourly job would never rotate anything. EasyEngine now warns that maxsize is only checked when logrotate runs instead of installing that job.
copy() created the backup with the default umask, so a rule restricted to root (which may hold credentials in its scripts) was saved world-readable. The backup dir is now 0700 and the copy 0600.
copytruncate copies the whole log before truncating it, so a rotation needs as much free space as the log. Once a log is bigger than the free space, every run fails with "No space left on device", leaves the log in place, and the log grows until the disk is full. Rotation never recovers on its own. All three EasyEngine rules now rename the log and tell the writer to reopen it, which needs no extra space: - ee_nginx_proxy: `nginx -s reopen` inside the container whose compose service is global-nginx-proxy. A signal to the container would go to forego (PID 1), not nginx. - ee_sites: USR1 to every site's nginx and php container (io.easyengine.site label), where nginx and php-fpm are PID 1 and reopen their logs on USR1. - ee_cli: no signal needed, since each ee run opens ee.log again. Existing installs get the new rules on the next update, as the setup step rewrites any rule that differs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stops log rotation from failing for good once a log is bigger than the free disk space. Stacked on #1958, so merge that one first. Until then this PR's diff also shows #1958's commits, and the change here is the last commit only.
Problem
All three rules EasyEngine installs (
ee_nginx_proxy,ee_sites,ee_cli) usecopytruncate. logrotate copies the whole live log to.1and only then truncates it, so a rotation needs as much free space as the log itself. On a busy server the nginx-proxyaccess.logreached about 20 GB. One night the copy didn't fit, logrotate loggederror copying … No space left on deviceand left the log in place, and the same thing happened every night after that while the log kept growing. Each failed copy also filled the disk for several minutes around midnight, so backups and databases running at that moment failed. Nothing recovers from that state without a manual truncate.#1958 adds
maxsizeand an hourly run, which keeps logs small in normal operation. It doesn't fix this failure: if free space ever drops below the log size,copytruncatefails, and the log grows pastmaxsizefrom then on.What changes
The three rules rename the log and have the writer reopen it, which needs no extra space:
ee_nginx_proxyrunsnginx -s reopenin the container whose compose service isglobal-nginx-proxy, found withdocker ps -f label=com.docker.compose.service=global-nginx-proxyinstead of a hard-coded container name. It can't usedocker kill -s USR1, because PID 1 in the nginx-proxy image is forego, not nginx.ee_sitessendsUSR1to each site'snginxandphpcontainers (io.easyengine.sitelabel plus the compose service name). In both images the master process is PID 1 (openresty, and php-fpm through anexecentrypoint), and both reopen their logs onUSR1.ee_clijust dropscopytruncate. Eacheerun opensee.logagain, so nothing needs a signal.delaycompressstays, so the renamed file isn't compressed until the next rotation, after the writer has moved to the new file. The postrotate scripts end in|| true: a stopped proxy or site has nothing writing to its log, so a failed signal there is harmless.Existing installs pick up the new rules on the next
ee cli update, because the setup step rewrites any EasyEngine rule whose content differs. No migration is needed.Not in this PR
dateext.rotate 30with compressed archives isn't what fills disks. fix(logrotate): cap host syslog and mail.log growth #1958 already setsdateformatfor hosts that enabledateext.Testing
Throwaway containers only: a
docker:dindcontainer acted as the EasyEngine host, so the postrotate label filters only ever matched the test containers. Images are the onesimg-versions.jsonpins:easyengine/nginx-proxy:v4.13.0,easyengine/nginx:v4.11.0andeasyengine/php8.3:v4.13.3. The rules were rendered fromUtils.phpbefore (#1958 head) and after this change, and run with the logrotate from thedocker:dindimage (Alpine).php -lis clean on PHP 7.4 and 8.5, andlogrotate -daccepts all three new rules (Ubuntu 24.04).access.log(24 MB free):error copying …/access.log to …/access.log.1: No space left on device, exit 1, log left in place.access.logrenamed toaccess.log.1. nginx's open files point at the newaccess.log, and a request made after the rotation is logged there and not in.1.maxsizewith a non-forced run (what the hourly job from fix(logrotate): cap host syslog and mail.log growth #1958 does): right after the forced rotation, the run reportslog does not need rotating (log has already been rotated). Onceaccess.logis 1.1 GB, the next run reportslog needs rotatingand renames it, and new requests go to the new file.USR1both containers' PID 1 hold the new files open.Proxy rule, old and new:
Site rule: