# VPS Hosting Server This document descibes the configuration of the VPS Apache server required for the hosting of the kulibri App. ## Required Folders ### Folder: export You need folder called "export" in the public root. This filder is used to temporary store the exported exvel files Use the following command to create the folder ``` mkdir /var/www//export ``` ## Storage link When create new server just create a storage:link under that server. to create link use the below route: /storage_link for ex (for test server): ========= https://test.kulibri.com/storage_link This is a shortcut for ``` php artisan storage:link ``` ## User-uploaded images (`storage/app/public`) Poll background images and logos (`UserController::imageUpload`/`logoUpload`, tracked via `UserImage`/`UserLogo`) are stored on local disk at `storage/app/public/`, served through the `storage:link` symlink above. This is real user data, not code or config — it's not covered by deployment or the Ansible cron setup, and has to be copied by hand when moving to a new server. Since the old server typically stays live and accepting uploads during testing of the new one, do this as **two passes**, run from the old server: 1. Bulk copy, well ahead of cutover: ``` rsync -avz --omit-dir-times -e "ssh -i /home/mitko/.ssh/id_ed25519_backup -o IdentitiesOnly=yes" \ /var/www/app.kulibri.com/storage/app/public/ \ mitko@:/var/www/app.kulibri.com/storage/app/public/ ``` 2. Final delta sync right before flipping traffic to the new server — adds `--delete` to pick up anything uploaded (or removed) since the bulk copy: ``` rsync -avz --delete --omit-dir-times -e "ssh -i /home/mitko/.ssh/id_ed25519_backup -o IdentitiesOnly=yes" \ /var/www/app.kulibri.com/storage/app/public/ \ mitko@:/var/www/app.kulibri.com/storage/app/public/ rsync -avz --delete --omit-dir-times -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" /var/www/app.kulibri.com/storage/app/public/ mitko@46.62.167.243:/var/www/app.kulibri.com/storage/app/public/ ``` After each sync, on the new server: ``` chown -R www-data:www-data /var/www/app.kulibri.com/storage/app/public ``` (files land owned by `mitko`, the SSH user — PHP won't be able to read/write them until this runs). Prerequisites: - The app must already be deployed at that path on the new server (destination directory exists) before syncing. - `id_ed25519_backup`'s public key must be added to `mitko@`'s `authorized_keys`. That key was originally set up for the offsite DB backup host (`65.21.184.46`) — a different machine — so don't assume it already has access to the new server; add it explicitly (or use a separate key). - `storage:link` (above) is independent of this — run it before or after, doesn't matter. - If the syncing user is only a member of the `www-data` group (not the owner) and the destination directory already exists from deployment, expect `rsync: failed to set times on ".../public/.": Operation not permitted` — group write access lets you create/modify files inside, but setting an explicit mtime on a pre-existing directory you don't own requires ownership or root. `--omit-dir-times` (included above) skips this; it doesn't matter here since only the files' own timestamps are relevant, not the directory's. ## Cron Jobs New servers need three separate crontabs configured (found by inventorying the prod box on 2026-08-27). These are **not** set up by the app deployment. > ### Provisioning lives in `ansible-thinkpi`, not in this repo > > The cron lines are installed by **`playbooks/setup-kulibri-backup.yml`** in the > separate `ansible-thinkpi` repo (`~/ansible-thinkpi` in WSL), which is the > source of truth for every server-side concern: > > ```bash > cd ~/ansible-thinkpi # ansible.cfg uses relative paths — run from the root > ansible-playbook playbooks/setup-kulibri-backup.yml --ask-vault-pass -e target_group=prod > ansible-playbook playbooks/authorize-kulibri-backup-key.yml --ask-vault-pass -e target_group=prod > ``` > > The inventory is **`inventory/hosts.ini`**, and `ansible.cfg` already points at > it (`inventory=inventory/hosts.ini`, `roles_path=./roles`) — so no `-i` flag is > needed, but both paths are relative, which is why the play must be run from the > repo root. > > | Group | Host | SSH user | Notes | > |---|---|---|---| > | `prod` | 195.201.32.252 | `ansible` | `kulibri-prod` — **this is the Kulibri production server** | > | `test` | 65.21.184.46 | `ansible` | also the offsite backup destination; the same machine, worth remembering | > | `flex` | 37.27.254.83 | `ansible` | **SSH port 2222**, not 22 | > | `new` | 46.62.167.243 | `root` | connects as root, so no privilege escalation | > > All four use `~/.ssh/id_rsa`. `prod`, `test` and `flex` set > `ansible_become_password` from the vault (`group_vars//vars.yml` → > `vault_become_password`), so **`--ask-vault-pass` supplies sudo as well** — > `--ask-become-pass` is not needed and will just prompt for something nothing > reads. The vault password is required for any play targeting those groups even > when the play itself uses no secrets, because group_vars load before the first > task runs. > > There is no `kulibri_servers` group and never was; `prod` is the group for > everything Kulibri. > > This repo once carried its own `ansible/cron.yml`; it was deleted on > 2026-09-19 because it had drifted from `setup-cron.yml` and named its cron > entry differently, so applying both would have installed *two* `schedule:run` > lines and fired every scheduled job twice a minute. Do not reintroduce a copy > here — change `ansible-thinkpi` instead. ### After a server rebuild, cron is empty Production was rebuilt in place on **2026-09-16** for the Ubuntu 24.04 upgrade, keeping the same IP. A rebuilt disk has no crontabs, and `crontab -l` exits 1 rather than printing nothing — so a check that greps its output looks identical to "installed but no match". Verify positively: ```bash sudo crontab -u www-data -l | grep schedule:run sudo -u www-data php /var/www/app.kulibri.com/artisan schedule:list ``` `UBUNTU-UPGRADE.md`'s Phase 3 exists for exactly this: for 48 hours after a rebuild, every scheduled job is one that has never run once on that disk. ### `www-data` crontab — Laravel scheduler ``` * * * * * php /var/www/app.kulibri.com/artisan schedule:run >> /dev/null 2>&1 ``` This drives the scheduled commands defined in `app/Console/Kernel.php` — currently `process:task` (daily 21:30), `auto:reminder` (hourly at :15), `process:subscriptions` (daily 07:30), `backup:database` (daily 00:10), `backup:cleanup` (daily 03:00). Without this single line, none of those jobs run even though they're in code. The new server needs the `mysqldump` binary installed and on `www-data`'s `PATH` — previously the dump ran under `root`, now it runs as whatever user PHP-FPM/Apache runs as. ### `root` crontab — SSL renewal only ``` 30 2 * * 1 /usr/bin/certbot renew >> /var/log/le-renew.log 35 2 * * 1 systemctl reload apache2 ``` `certbot renew` requires certbot installed and DNS already pointed at the new server before it will succeed — do this after cutover, not during initial testing on the new box's temporary IP. **DB backup creation and cleanup have been moved out of the crontab and into the Laravel scheduler** (`backup:database` and `backup:cleanup` in `app/Console/Kernel.php`, run via the `www-data` `schedule:run` entry above) — no separate cron entries or root DB credentials needed for them anymore. This also fixes the old setup putting the MySQL root password in plaintext in root's crontab. Details: - `backup:database` runs `mysqldump` using the app's own DB connection credentials from `.env` (or `DB_BACKUP_USERNAME`/`DB_BACKUP_PASSWORD` if you want a dedicated, more restricted backup DB user — recommended over reusing the app's write credentials or root). The password is passed via the `MYSQL_PWD` environment variable, not a command-line flag, so it doesn't show up in `ps` output. - Dumps are written to `DB_BACKUP_PATH` (defaults to `storage/app/db-backups`) as `db__.sql.gz`. - **Dumps are gzipped** (`DB_BACKUP_COMPRESS`, default on). SQL compresses 6–8×, so a night goes from ~816 MB to ~120 MB — fourteen days is ~1.7 GB instead of ~11 GB, here *and* again on the offsite host. `backup:cleanup` and the health check's freshness probe match `.sql` and `.sql.gz` alike, so switching the setting later does not orphan existing dumps. To restore: `zcat db_kulibri_20260919-08.sql.gz | mysql -u kulibri -p kulibri`. - The dump runs under `bash -c` with `set -o pipefail`, not `/bin/sh`. On Debian and Ubuntu `/bin/sh` is dash, which has no `pipefail`, and a pipeline reports only its *last* command's status — so gzip would compress a half-finished dump and the backup would be recorded as successful. Keep the `bash` invocation if you touch this. - `backup:cleanup` deletes files matching `db_*.sql` in that directory older than 14 days (`--days=N` to override). - Both are scheduled daily in `Kernel.php` at the same times the old cron jobs ran (00:10 and 03:00). `backup:database` runs with `--alert`. - The dump is redirected to its file by the shell, never read into PHP. It used to be buffered through `$process->getOutput()`, which holds the whole dump in memory as a string — survivable at 50 MB, not at the ~800 MB production is at now. - **Exit code 0 is not trusted on its own.** The dump is checked for non-zero size and for mysqldump's `-- Dump completed` trailer, and a dump that fails either is deleted rather than left in the directory. Before this, `db_kulibri_20260827-14.sql` sat in production at 0 bytes for three weeks looking exactly like a real backup. - `backup:cleanup` refuses to delete the newest dump when *every* dump has expired — that state means the backup job has not run for longer than the retention window, and pruning then would destroy the last restore point. Fix the job first, then `--force` if you still want it gone. - A failed dump emails the support team (`BACKUP_ALERT_EMAIL`, falling back to `HEALTH_ALERT_EMAIL` and then the mail support address). `health:check` additionally reports the age of the newest dump and mails once when it passes `DB_BACKUP_MAX_AGE_HOURS` (default 26) and once when a fresh one appears. > **Backup freshness is reported on `GET /api/health`, not just from cron.** > It is deliberately a `warn`, so it never flips the endpoint or the exit code > to failing — an uptime monitor that pages for a backup gap is one people > learn to ignore. Point the external monitor at `checks.backup.age_hours` if > you want it paged separately. This matters because a check that only runs > from the scheduler cannot detect a scheduler that is not running — which is > precisely what happened after the 2026-09-16 rebuild, found by hand three days > later only because someone happened to look in the directory. ### Backup-account crontab — offsite sync ``` 1 1 * * * rsync -avz -e "ssh -i ~/.ssh/id_ed25519_backup -o IdentitiesOnly=yes" /var/www/app.kulibri.com/storage/app/db-backups/ mitko@65.21.184.46:/home/mitko/db-backups/ --delete >> ~/logs/cron.log 2>&1 ``` The source path changed from `/home/mitko/db-backups/` to the app's `DB_BACKUP_PATH` (`storage/app/db-backups`) now that `backup:database` writes dumps there directly. The old `~/scripts/cleanup_db_backups.sh` + its cron line are no longer needed — `backup:cleanup` (see above) replaces it. > **This rsync runs with `--delete`, so the offsite copy is a mirror, not an > archive.** It does not protect against a gap on the app server — it reproduces > one. On a freshly rebuilt disk the local directory starts empty, so the first > run would erase the entire offsite history. Seed the directory *before* this > cron exists, and preserve the offsite side with hardlinks first (cheap — it > pins inodes rather than copying ~800 MB per dump): > > ```bash > ssh mitko@65.21.184.46 'cp -al ~/db-backups ~/db-backups-precutover' > ``` > > Remove that directory a week or two later, once the new server has proven > itself. See `ansible-thinkpi/UBUNTU-UPGRADE.md` → Gotchas. **All of this is provisioned by the two plays above — none of it is manual.** `tasks/setup-backup-user.yml` creates the `kulibri-backup` user (nologin, in the `www-data` group so it can read the dumps), its `~/.ssh` and `~/logs` directories, and **generates `id_ed25519_backup` itself** with `community.crypto.openssh_keypair`, which is idempotent and will not overwrite an existing key. `authorize-kulibri-backup-key.yml` then slurps that public key off the app server and adds it to `mitko@65.21.184.46`'s `authorized_keys`. The one genuinely manual step is accepting the offsite host's SSH host key once, since the first unattended rsync would otherwise hang on the prompt: ```bash sudo -u kulibri-backup ssh -i /home/kulibri-backup/.ssh/id_ed25519_backup -o IdentitiesOnly=yes mitko@65.21.184.46 true ``` --- ## Historical / scratch — NOT current configuration Everything below this heading is kept for reference only: one-off commands from the 2026-09 server migration, and the pre-scheduler backup setup it replaced. The paths (`/home/mitko/db-backups/`), the host (`alpha.kulibri.com`) and the root `mysqldump` cron line are all obsolete — dumps now live in the app's `DB_BACKUP_PATH` and are taken by `backup:database`. For the current setup see the sections above. dry run rsync -avzn --omit-dir-times -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" \ /var/www/app.kulibri.com/public/images/ \ mitko@alpha.kulibri.com:/var/www/app.kulibri.com/public/images/ real run rsync -avz --omit-dir-times -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" \ /var/www/app.kulibri.com/public/images/ \ mitko@alpha.kulibri.com:/var/www/app.kulibri.com/public/images/ real run rsync -avz --omit-dir-times -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" \ /var/www/app.kulibri.com/storage/app/public/ \ mitko@alpha.kulibri.com:/var/www/app.kulibri.com/storage/app/public/ rsync -avz --no-perms --omit-dir-times -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" \ /var/www/app.kulibri.com/storage/app/public/ \ mitko@alpha.kulibri.com:/var/www/app.kulibri.com/storage/app/public/ rsync -avz --no-perms --omit-dir-times -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" \ /home/mitko/db-backup \ mitko@alpha.kulibri.com:/home/mitko/db-backup/ rsync -avz -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" /home/mitko/db-backups/ mitko@alpha.kulibri.com:/home/mitko/db-backups/ --delete rsync -avz -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" /home/mitko/db-backups/ mitko@65.21.184.46:/home/mitko/db-backups/ --delete rsync -avz -e "ssh -i /home/mitko/.ssh/id_rsa -o IdentitiesOnly=yes" /home/mitko/db-backups/ mitko@65.21.184.46:/home/mitko/db-backups/ --delete mysqldump -u root -pxxxxxx kulibri > /home/mitko/db-backups/db_kulibri_$(date +\%Y\%m\%d-\%H).sql --- ## Alive ping / uptime monitoring Two endpoints, deliberately different in what they prove: | Endpoint | What it proves | Cost | |---|---|---| | `GET /ping` | Apache/PHP-FPM answers and Laravel boots. No DB, cache or disk touched. | ~nothing | | `GET /api/health` | A real user journey works end to end: **login → fetch that user's polls → logout**, over real HTTP against this server's own public URL. | 3 internal requests | `/api/health` is the one to point an uptime monitor at. Because it goes over the wire instead of calling controllers in-process, it also exercises the web server, TLS certificate, routing, the Sanctum guard, the `verified` gate and the hashids middleware — the things that break for infrastructure reasons rather than code reasons. **Responses** - `200` + `{"status":"ok"}` — everything worked. - `503` + `{"status":"fail"}` — any step failed. Alert on this. The public body is intentionally just the status. Per-step detail (timings, HTTP codes, poll count, error text) needs the shared secret: ``` curl -s -H "X-Health-Token: $HEALTH_CHECK_TOKEN" https://app.kulibri.com/api/health | jq { "status": "ok", "checked_at": "2026-09-05T08:14:22+00:00", "duration_ms": 214, "checks": { "login": {"status": "ok", "duration_ms": 96}, "polls": {"status": "ok", "duration_ms": 84, "polls": 3}, "logout": {"status": "ok", "duration_ms": 34} } } ``` `?token=...` works too, but the header is preferred — query strings end up in Apache access logs. ### Server setup 1. **Create a dedicated monitoring user** in the app (a normal registered account). It must be **email-verified** — `/api/polls` sits behind the `verified` middleware — and must own **at least one poll**. 2. **Set the env vars** (see `.env.example` for the full list): ``` HEALTH_CHECK_EMAIL=monitoring@kulibri.com HEALTH_CHECK_PASSWORD= HEALTH_CHECK_TOKEN= ``` `APP_URL` must be the real public URL, since the check calls back into it. Override with `HEALTH_CHECK_BASE_URL` only if the server has to reach itself on a different address. After changing these, run `php artisan config:cache` if the config is cached. 3. **Point the monitor** (UptimeRobot / Pingdom / Better Uptime) at `https://app.kulibri.com/api/health`, alerting on any non-200. **Every 10 minutes** is the chosen interval: the polls step runs a heavy dashboard query (see below), and at that cadence its cost is negligible while an outage still surfaces within ten minutes. The check writes nothing to the database on the happy path. ### Email alerts `php artisan health:check --alert` runs the same check from cron and mails the support team. It is scheduled every 10 minutes in `app/Console/Kernel.php`, so it rides the existing `schedule:run` crontab entry (installed by `ansible-thinkpi/playbooks/tasks/setup-cron.yml`) — no extra cron line to add. ``` HEALTH_ALERT_EMAIL=support@kulibri.com # optional; falls back to MAIL_SUPPORT_ADDRESS ``` **One mail when the check starts failing, one when it recovers** — not one per failed run. A three-hour outage sends two emails, not eighteen. The last seen status lives in the cache store under `health-check:last-status`; clearing the cache mid-outage can cost you one duplicate alert, which is the right way round. Alerting from cron rather than from the endpoint is deliberate: - **It can still alert when the web server is down.** An email sent from inside `/api/health` only goes out when the app is healthy enough to serve the request — silent in exactly the outages that matter most. CLI and SMTP do not need Apache. - **It keeps a public URL from being a lever that sends mail.** `/api/health` is unauthenticated by necessity; it must not email on demand. - **It adds no latency to the endpoint.** `QUEUE_CONNECTION=sync`, so mail sends inline; a slow SMTP server would otherwise stall the health response and make the check fail for reasons unrelated to the app. `HealthCheckAlertMail` is deliberately **not** `ShouldQueue`, unlike the customer-facing mailables next to it: an outage alert handed to a queue only goes out if the queue is being worked, which is not safe to assume mid-outage. If the send itself throws, the stored state is left unchanged so the next run retries rather than going quiet about an outage nobody was told about. Run it by hand any time — it prints the same JSON and exits non-zero on failure: ``` php artisan health:check # check only php artisan health:check --alert # check and mail on a state change ``` The external uptime monitor stays the authoritative up/down signal. This is the channel that tells you *what* broke. ### Why the poll count matters `HEALTH_CHECK_MIN_POLLS` (default 1) makes an empty poll list a failure. A database that is *up but wrong* — blank schema after a bad restore, credentials pointing at the wrong host — otherwise answers `200 []` and looks perfectly healthy. Raise the floor if the monitoring user reliably owns more. ### Things to know - **The check needs at least two free workers.** It is one request that makes further requests to itself. Apache/PHP-FPM in production handles this fine, but PHP's built-in dev server is single-process, so locally the self-call deadlocks. Verified behaviour: it fails cleanly rather than hanging — the request blocks for exactly `HEALTH_CHECK_TIMEOUT` seconds, then answers 503 with `cURL error 28: Operation timed out`. Use `/ping` for local smoke tests, or run the dev server with `PHP_CLI_SERVER_WORKERS=4` (Linux/macOS only; PHP on Windows ignores it). - **`php artisan serve` drops these env vars.** `ServeCommand` forwards only an allowlist of variables to the PHP server process it spawns, so `HEALTH_CHECK_EMAIL=... php artisan serve` leaves the check unconfigured and reporting `checks.config` failed. Put the values in `.env` — which is how production runs anyway — or, to test without touching `.env`, start the server directly: `HEALTH_CHECK_EMAIL=... php -S 127.0.0.1:8000 -t public public/index.php`. - **Each ping creates and revokes one Sanctum token** named `health-check`. If the logout step ever fails, stale tokens older than an hour are swept on the next ping, so `personal_access_tokens` cannot grow unbounded. - **Each ping writes ~2 lines to `storage/logs/laravel.log`** — one from the `logger` middleware on `/api/polls`, one from `PollController::getMyPolls`. At one ping per 10 minutes that is roughly 300 lines a day, but make sure log rotation is in place regardless. - **`storage/` and `bootstrap/cache/` must be owned by `www-data`.** This is the single most common failure on this app, and it is worth understanding rather than just fixing. `config/logging.php` sets `'permission' => 0664` on the daily channel, so Monolog calls `chmod()` every time it opens the log file — and `chmod` on a file owned by another user is denied even when the file is group-writable. Combined with `'ignore_exceptions' => false` on the stack, a failed log write becomes an exception, so **every endpoint that logs returns 500**: `LoggerMiddleware` logs on every authenticated route, and `PollController::getMyPolls` logs at the end. `/api/login` and `/api/logout` do not log, which is why an outage of this kind shows up as login OK, polls 500 — a signature worth recognising. Fix: `chown -R www-data:www-data storage bootstrap/cache`. - **Always run artisan as `www-data`** (`sudo -u www-data php artisan ...`). Running it as root or as your SSH user leaves root-owned compiled views and cache files behind, which breaks the web user later in exactly the way above. The scheduler cron already runs as `www-data` (see `setup-cron.yml`). - **The alert survives broken logging.** `App\Listeners\LogSendingEmail` writes a line on every `MessageSending` event, so with an unwritable log the alert mail would throw on its way out — silent in precisely the outage it exists to report. `health:check` therefore switches the log channel to `null` around the send, and its own error handling never logs unguarded. There is also a `logs` step in the check itself, which writes one line through the real log stack and reports the exception verbatim, so the alert names the cause instead of just showing an opaque `HTTP 500` on the polls step. - **The polls step is not cheap.** `getMyPolls` left-joins `participants` with an `orWhere` across both tables, which stops the index on `polls.user_id` from being used, so it scans the join. Measured at ~4s against a production-sized dataset on the test server. That is affordable at a 10 minute interval; it would not be at one minute. - **TLS verification stays on** (`HEALTH_CHECK_VERIFY_TLS=true`). An expired certificate is exactly the kind of failure this ping should surface. Only turn it off on a staging box with a self-signed cert.