Branch: refs/heads/metascheduler-pool-hardening Home: https://github.com/ehb54/us3lims_gridctl Commit: d37d51a98d00609badf2602273db2211d339d080 https://github.com/ehb54/us3lims_gridctl/commit/d37d51a98d00609badf2602273db... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-06 (Tue, 06 Oct 2026) Changed paths: M job_state_machine.php M jobmonitor/cleanup.php M jobmonitor/cleanup_job.php M jobmonitor/gridctl.php M jobmonitor/jobmonitor.php M services.php Log Message: ----------- Retry permanent DB failures in cleanup only when they are actually connection-class, dedupe the 'fail' admin mail per job, retry the boot restart for its full budget in a detached background child with a final mail/syslog on failure, and fix the should-fix nits (post-ping scancel retries, missing-dbutils stderr spam, double timestamps, two stale comments, the startup connect exiting on a blip) Commit: 6f08d03ecaccd8327c814f80e91e90ba47d46bab https://github.com/ehb54/us3lims_gridctl/commit/6f08d03ecaccd8327c814f80e91e... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-06 (Tue, 06 Oct 2026) Changed paths: M job_state_machine.php M jobmonitor/jobmonitor.php Log Message: ----------- Use remote_exec's new breaker_gate option for the post-ping scancel so a real failure still counts against the breaker, and add +/-20% jitter to the steady-state poll (ultrascan-tickets#1120) Commit: bfefa2acc362a44ee4d5192d54f76a768905937d https://github.com/ehb54/us3lims_gridctl/commit/bfefa2acc362a44ee4d5192d54f7... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-06 (Tue, 06 Oct 2026) Changed paths: M services.php Log Message: ----------- Remove services.php's own monitor restart: it duplicated us3-jobmonitors.service, a separate dedicated unit, and put restarted monitors in us3-listen's own cgroup where a later systemctl restart us3-listen would kill them Commit: a9539fb68121c1f95d3e664aef8137298192b4aa https://github.com/ehb54/us3lims_gridctl/commit/a9539fb68121c1f95d3e664aef81... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-06 (Tue, 06 Oct 2026) Changed paths: M jobmonitor/gridctl.php Log Message: ----------- Keep a SUBMIT_TIMEOUT/RUN_TIMEOUT row from being reset to its first stall window by a still-alive cluster probe answer Commit: 46a3ffd91d87ff29bb60634c8c327c59b2c08410 https://github.com/ehb54/us3lims_gridctl/commit/46a3ffd91d87ff29bb60634c8c32... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-07 (Wed, 07 Oct 2026) Changed paths: M services.php Log Message: ----------- Revert "Remove services.php's own monitor restart: it duplicated us3-jobmonitors.service, a separate dedicated unit, and put restarted monitors in us3-listen's own cgroup where a later systemctl restart us3-listen would kill them" This reverts commit bfefa2acc362a44ee4d5192d54f76a768905937d. Commit: eb1414165ce95a94e274b8082aa6d64a3e836f71 https://github.com/ehb54/us3lims_gridctl/commit/eb1414165ce95a94e274b8082aa6... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-07 (Wed, 07 Oct 2026) Changed paths: M job_state_machine.php M jobmonitor/gridctl.php Log Message: ----------- Treat errno 1053 as connection-class, and keep a fire_stall()-written FAILED from being reset by a still-alive probe answer Commit: 5d73bcc6f92efd608cc9f1ac6474225b6972d099 https://github.com/ehb54/us3lims_gridctl/commit/5d73bcc6f92efd608cc9f1ac6474... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-07 (Wed, 07 Oct 2026) Changed paths: M jobmonitor/cleanup_job.php M jobmonitor/gridctl.php M listen_functions.php Log Message: ----------- Dedupe 'fail' admin mail on the message instead of a single flag, log the connect errno/error on every startup-connect retry, and fix two stale comments misdescribing joblinkjson.php Commit: c4e5355307199b66054f51a50b9198f24fdef365 https://github.com/ehb54/us3lims_gridctl/commit/c4e5355307199b66054f51a50b91... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-07 (Wed, 07 Oct 2026) Changed paths: M jobmonitor/cleanup.php Log Message: ----------- Remove the dead $aira_details reference in the admin-mail template (undefined on PHP 8.2, Airavata is already gone) Commit: 899d19396996d4d8a15caa920ac266260fbe6ddb https://github.com/ehb54/us3lims_gridctl/commit/899d19396996d4d8a15caa920ac2... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-07 (Wed, 07 Oct 2026) Changed paths: M job_state_machine.php M jobmonitor/cleanup.php M jobmonitor/gridctl.php M services.php Log Message: ----------- Retry a restart through MariaDB's HY000/2002 and gone-away/shutdown texts after a clean stop, let stop() kill a pending restart retry, validate _restart-retry-background's arguments, fix the backwards tar-location comment, and drop the remaining round-N references Commit: 3f56987b0f10e30ac0b1f5b85909b28767255ebf https://github.com/ehb54/us3lims_gridctl/commit/3f56987b0f10e30ac0b1f5b85909... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-08 (Thu, 08 Oct 2026) Changed paths: M job_state_machine.php M jobmonitor/cleanup.php M jobmonitor/cleanup_job.php M jobmonitor/gridctl.php M jobmonitor/jobmonitor.php M listen_functions.php M services.php Log Message: ----------- Poll local clusters without ssh again, kill the retry child by command line instead of a stale pidfile, let the child compute its own restart command, break retry on any non-connection-class failure, and clean up stale round references Commit: 4594b9790b73bf5b1d44ea54e1be48c4d8cd0b2a https://github.com/ehb54/us3lims_gridctl/commit/4594b9790b73bf5b1d44ea54e1be... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-08 (Thu, 08 Oct 2026) Changed paths: M jobmonitor/cleanup.php Log Message: ----------- Try the output/ directory before the top-level fallback when fetching a finished job's results tar, since that is where us_mpi_analysis actually writes it Commit: 18d1f503c62f742e6cee1c1f4b247d2c485c903b https://github.com/ehb54/us3lims_gridctl/commit/18d1f503c62f742e6cee1c1f4b24... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-08 (Thu, 08 Oct 2026) Changed paths: M jobmonitor/cleanup.php M jobmonitor/cleanup_job.php Log Message: ----------- Leave the finalizing marker in place when a connection-class failure interrupts the stage write, so --restart can still close the job out Commit: 0172ba8565633b49cfe4c8e9f5dcc67778c3c3e6 https://github.com/ehb54/us3lims_gridctl/commit/0172ba8565633b49cfe4c8e9f5dc... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-08 (Thu, 08 Oct 2026) Changed paths: M status/slurm Log Message: ----------- Decide cluster status from usable node counts instead of the partition's blanket admin flag, so one down node no longer reports the whole queue down Commit: 1f7eaa40513f37529c4c8eadb817675af101197e https://github.com/ehb54/us3lims_gridctl/commit/1f7eaa40513f37529c4c8eadb817... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-09 (Fri, 09 Oct 2026) Changed paths: M job_state_machine.php M jobmonitor/cleanup.php M jobmonitor/cleanup_job.php M jobmonitor/gridctl.php M status/slurm Log Message: ----------- Retry a connection-class failure writing the finalizing span before leaving the marker for --restart, check the span's last status write too, and stop status/slurm reporting a confident down when sinfo cannot reach the controller at all Commit: aa1f64d522f682d7c16cb6f0259c5cfa238801a6 https://github.com/ehb54/us3lims_gridctl/commit/aa1f64d522f682d7c16cb6f0259c... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-09 (Fri, 09 Oct 2026) Changed paths: M jobmonitor/jobmonitor.php M services.php Log Message: ----------- Anchor the pending-retry-child pkill pattern to its exact command line, escape it for POSIX ERE instead of PCRE, fix the stale and undercounted comments, and drop review-round labels Commit: 463271d253166391a310a4ef7b196764d2ca58f2 https://github.com/ehb54/us3lims_gridctl/commit/463271d253166391a310a4ef7b19... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-10 (Sat, 10 Oct 2026) Changed paths: M jobmonitor/queuepos.php Log Message: ----------- Ask squeue for explicit delimited fields instead of parsing its version-dependent default table, so a cluster whose row shape differs no longer loses every job's queue position Commit: f5d747746083f829dc389b98636d146c8823ac96 https://github.com/ehb54/us3lims_gridctl/commit/f5d747746083f829dc389b98636d... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-10 (Sat, 10 Oct 2026) Changed paths: M job_state_machine.php M jobmonitor/cleanup_job.php Log Message: ----------- Reopen the global db_handle before retrying the finalizing status write after a MariaDB restart, and fold update_hpc_analysis_result_status()'s own result into update_autoflow_status()'s return value so a non-autoflow job's status failure is no longer silently reported as success Commit: 8379fc323156a05f9b7e010edf10212dd2487276 https://github.com/ehb54/us3lims_gridctl/commit/8379fc323156a05f9b7e010edf10... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-10 (Sat, 10 Oct 2026) Changed paths: M jobmonitor/cleanup_job.php Log Message: ----------- Name HPCAnalysisResult alongside autoflowAnalysis in the finalizing-write admin mail and log, since a non-autoflow job's only status write can now be the one that failed Commit: 035fa3b489f08bad41bff2e8a108fae658b473ce https://github.com/ehb54/us3lims_gridctl/commit/035fa3b489f08bad41bff2e8a108... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-10 (Sat, 10 Oct 2026) Changed paths: M jobmonitor/cleanup_job.php M jobmonitor/jobmonitor.php M services.php M status/slurm Log Message: ----------- Fix status/slurm's and the PR description's backwards up/down claim, correct services.php's wrong reason for avoiding preg_quote(), match the retry child's spawning interpreter as any token instead of this invocation's own PHP_BINARY, pin fd 0 to /dev/null when daemonizing so fd 2 is never left for the next mysqli reconnect to claim, fix two run-on comments left from round-label removal, and log the first write's own reconnect-and-retry Commit: ba9207d792bcd9f36779b02cc1ec18a775fe6ef7 https://github.com/ehb54/us3lims_gridctl/commit/ba9207d792bcd9f36779b02cc1ec... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-10 (Sat, 10 Oct 2026) Changed paths: M jobmonitor/queuepos.php Log Message: ----------- Count PD jobs' position as ascending rank in listing order instead of a descending total-count offset, so the next job to start reports position 1 instead of the queue size Commit: 9d209b114ce29191d3e163a5b1e02e35415d8360 https://github.com/ehb54/us3lims_gridctl/commit/9d209b114ce29191d3e163a5b1e0... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-11 (Sun, 11 Oct 2026) Changed paths: M services.php Log Message: ----------- Refuse to start or restart the gridctl daemons as root, with a message naming the account, since root has no ssh key or known_hosts entry for any cluster us3 does and every submission would otherwise just fail at staging Commit: 67054cfeec9e710d8713eb8a266c9e9398175c85 https://github.com/ehb54/us3lims_gridctl/commit/67054cfeec9e710d8713eb8a266c... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-11 (Sun, 11 Oct 2026) Changed paths: M jobmonitor/cleanup_job.php M jobmonitor/queuepos.php M services.php Log Message: ----------- Retry the finalizing status write with doubling backoff over ~2 minutes instead of 2 retries 2s apart, require a php-like first word in the pending-restart pkill pattern instead of matching any word, and count a PD job's queue position per partition instead of across the whole queue Commit: 04b04ac89e1eccd49dc5f5b245f90263c88e6f6a https://github.com/ehb54/us3lims_gridctl/commit/04b04ac89e1eccd49dc5f5b245f9... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-11 (Sun, 11 Oct 2026) Changed paths: M esign.php Log Message: ----------- Fix esign.php's SME-notified timestamp format string, which wrote the month into the seconds field instead of the seconds Commit: b8f3f689b013918e878d1d2bdd5490c91b3ed87f https://github.com/ehb54/us3lims_gridctl/commit/b8f3f689b013918e878d1d2bdd54... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-11 (Sun, 11 Oct 2026) Changed paths: M esign.php Log Message: ----------- Stop esign.php overriding the site's admin_email and mailto with a personal address Commit: 9047727ee1ce82322330ff0eb6d718420d553e54 https://github.com/ehb54/us3lims_gridctl/commit/9047727ee1ce82322330ff0eb6d7... Author: aaron-auc <aaron@aucsolutions.com> Date: 2026-10-11 (Sun, 11 Oct 2026) Changed paths: M esign.php M job_state_machine.php M jobmonitor/cleanup.php M jobmonitor/cleanup_job.php M jobmonitor/gridctl.php M jobmonitor/jobmonitor.php M jobmonitor/queuepos.php M listen_functions.php M services.php M status/slurm Log Message: ----------- Merge updated branch '780-request-pbs-gfac-cleanup' (gfac-cleanup fixes, including the esign.php admin_email fix) into metascheduler-pool-hardening Compare: https://github.com/ehb54/us3lims_gridctl/compare/78c5bbec489a...9047727ee1ce To unsubscribe from these emails, change your notification settings at https://github.com/ehb54/us3lims_gridctl/settings/notifications