Typesense droplet setup

How to stand up one DigitalOcean Typesense host. QA and staging share smarthealth-typesense-qa (159.223.218.151, Ubuntu 26.04, ams3). Production is a second droplet later (SHAPP-3724). Local search stays on OrbStack at 127.0.0.1:8108.

The image tag must match root docker-compose.yml (typesense and typesense-testing) and scripts/typesense/docker-compose.yml. That pin is typesense/typesense:30.2.

Times on the droplet are UTC. Night work is 03:00 to 04:00 UTC.

Replace $HOST with smarthealth-typesense-qa or smarthealth-typesense-production. $HOST-root is the same address with User root. The agent uses ssh $HOST as smarthealth and ssh $HOST-root as root. Never ssh root@…, ssh smarthealth@…, or -o User=root after those host blocks exist.

Do not add update helpers or compose copies to this repo. The only committed Typesense file for the droplet is scripts/typesense/docker-compose.yml. The agent writes the rest on the box with the heredocs in this file.

Agent rules

Work top to bottom. Run each numbered command as written. Do not invent a shorter path.

At the start of a section, run the check. If the check already matches the expected output, skip that section.

✋ Manual means the operator does it in a UI, at a password prompt, or in 1Password.

On every ✋ Manual step:

Stop. Do not run the next command.
Tell the operator the exact command or click.
Ask them to say when it is done.
Wait for that answer. Then continue.

Ask for the sudo password from 1Password when sudo on $HOST prompts. Do not generate, print, or store that password.

Until smarthealth exists, use ssh $HOST-root only. After that user exists, use ssh $HOST for operator checks and ssh $HOST-root for privileged writes. Do not flip User on the $HOST block.

Create the droplet

✋ Manual. DigitalOcean → Create → Droplets.

Pick Basic, 512 MB RAM, 1 vCPU, 10 GB SSD. The size slug is often s-1vcpu-512mb-10gb.
Pick Amsterdam (ams3) or Frankfurt (fra1).
Pick Ubuntu 26.04.
Name it smarthealth-typesense-qa or smarthealth-typesense-production.
Attach the team SSH keys. The droplet starts with those keys on root. You can log in as root over SSH as soon as the droplet is active.
Wait until the droplet is active. Copy the public IPv4.

Resize to 1 GB if /health fails or RSS stays above about 350 MB after import. Do not load a Typesense embedding model.

✋ Manual. Add both host blocks to ~/.ssh/config on the laptop you will hand to the agent. $HOST is User smarthealth. $HOST-root is User root. On a new droplet, ssh $HOST fails until the operator user exists.

Host smarthealth-typesense-qa
	HostName 159.223.218.151
	User smarthealth
	IdentitiesOnly yes

Host smarthealth-typesense-qa-root
	HostName 159.223.218.151
	User root
	IdentitiesOnly yes

Host smarthealth-typesense-production
	HostName <public-ipv4>
	User smarthealth
	IdentitiesOnly yes

Host smarthealth-typesense-production-root
	HostName <public-ipv4>
	User root
	IdentitiesOnly yes

Prove root SSH from that laptop:

bash

ssh $HOST-root 'whoami'

Expect root. Tell the agent $HOST and that both host blocks are in place.

Agent: ssh $HOST-root and continue.

Bootstrap

Check:

bash

ssh $HOST-root 'command -v docker && command -v fail2ban-client && swapon --show && systemctl is-enabled docker droplet-agent'

Skip this section if Docker, Fail2ban, swap, docker, and droplet-agent are already on.

Install packages:

bash

ssh $HOST-root 'apt-get update && apt-get install -y docker.io docker-compose-v2 ufw fail2ban byobu'

Enable services:

bash

ssh $HOST-root 'byobu-enable; systemctl enable --now docker fail2ban droplet-agent'

Add 1 GB swap and keep it after reboot. Skip if swapon --show already lists /swapfile.

bash

ssh $HOST-root 'fallocate -l 1G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile && grep -qxF "/swapfile none swap sw 0 0" /etc/fstab || echo "/swapfile none swap sw 0 0" >> /etc/fstab'

Copy aliases onto root. The file is in the sibling docs repo. Ubuntu sources ~/.bash_aliases from ~/.bashrc. The operator copy is in Operator user and Bash aliases.

bash

scp ../smarthealth-docs/app/infrastructure/bash-aliases.sh $HOST-root:/root/.bash_aliases

Operator user

Check:

bash

ssh $HOST-root 'id smarthealth && groups smarthealth'

Skip user creation if smarthealth exists and is in sudo only. Not in docker.

Create the operator user and copy root keys:

bash

ssh $HOST-root "adduser --disabled-password --gecos 'Smart Health operator' smarthealth
mkdir -p /home/smarthealth/.ssh
chmod 700 /home/smarthealth/.ssh
cp /root/.ssh/authorized_keys /home/smarthealth/.ssh/authorized_keys
chown -R smarthealth:smarthealth /home/smarthealth/.ssh
chmod 600 /home/smarthealth/.ssh/authorized_keys
usermod -aG sudo smarthealth
install -o smarthealth -g smarthealth -m 644 /root/.bash_aliases /home/smarthealth/.bash_aliases"

✋ Manual. Set the sudo password. Stop here.

bash

ssh $HOST-root 'passwd smarthealth'

Store it in the team 1Password vault before the agent touches sshd.

Title Typesense QA droplet sudo or Typesense production droplet sudo
Username smarthealth

Do not put that password in the repo or in chat. Ask the operator if this step is done. Wait.

Bash aliases

Install the same file for root and smarthealth. ssh $HOST-root loads /root/.bash_aliases. ssh $HOST loads /home/smarthealth/.bash_aliases. Do not leave this on one user.

Check:

bash

ssh $HOST-root 'md5sum /root/.bash_aliases /home/smarthealth/.bash_aliases; grep -c "install --only-upgrade" /root/.bash_aliases /home/smarthealth/.bash_aliases'

Skip if both checksums match and both counts are 1.

Copy from the sibling docs repo onto root:

bash

scp ../smarthealth-docs/app/infrastructure/bash-aliases.sh $HOST-root:/root/.bash_aliases

Copy that file onto the operator user:

bash

ssh $HOST-root 'install -o smarthealth -g smarthealth -m 644 /root/.bash_aliases /home/smarthealth/.bash_aliases'

SSH and Web Console

Check:

bash

ssh $HOST-root 'sshd -T | grep -Ei "permitrootlogin|passwordauthentication|allowusers|x11forwarding|kbdinteractiveauthentication"'

Expected values: permitrootlogin prohibit-password, passwordauthentication no, allowusers smarthealth root, x11forwarding no, kbdinteractiveauthentication no.

Write /etc/ssh/sshd_config.d/70-smarthealth.conf. The Droplet Console is SSH through droplet-agent. It injects a short-lived key and logs in as root unless you pick another user. prohibit-password keeps that working. Password SSH stays off.

bash

ssh $HOST-root 'tee /etc/ssh/sshd_config.d/70-smarthealth.conf' <<'EOF'
AllowUsers smarthealth root
PermitRootLogin prohibit-password
PasswordAuthentication no
KbdInteractiveAuthentication no
X11Forwarding no
MaxAuthTries 3
EOF

Check the config, then reload SSH. Keep this SSH session open until the next ✋ Manual step passes.

bash

ssh $HOST-root 'sshd -t && systemctl reload ssh && systemctl enable --now droplet-agent && systemctl is-active droplet-agent'

Expect active.

✋ Manual. In a second terminal, prove both logins. Stop here.

bash

ssh $HOST
whoami
sudo -v

Expect smarthealth and a sudo password prompt. Then:

bash

ssh $HOST-root 'whoami'

Expect root. On the droplet page, click Web Console and confirm a shell. Then tell the agent it is done.

Typesense

Check:

bash

ssh $HOST 'curl -sS http://127.0.0.1:8108/health && grep image /opt/typesense/docker-compose.yml'

Expect {"ok":true} and image: typesense/typesense:30.2.

✋ Manual. Create an API key. Stop here.

bash

openssl rand -hex 32

Store it in 1Password as Typesense QA API key or Typesense production API key. Paste it to the agent once, or write /opt/typesense/.env yourself. Do not reuse the QA key on production. Ask if this step is done. Wait.

Create the data directory:

bash

ssh $HOST 'sudo mkdir -p /opt/typesense/data'

Copy the committed compose file from this repo. Do not rewrite it by hand on the box.

bash

scp scripts/typesense/docker-compose.yml $HOST:/tmp/typesense-docker-compose.yml
ssh $HOST 'sudo mv /tmp/typesense-docker-compose.yml /opt/typesense/docker-compose.yml'

Write the env file if the operator did not. Mode 600. Replace <the-key> with the value from 1Password.

bash

ssh $HOST "printf 'TYPESENSE_API_KEY=%s\n' '<the-key>' | sudo tee /opt/typesense/.env >/dev/null && sudo chmod 600 /opt/typesense/.env"

Start Typesense:

bash

ssh $HOST 'cd /opt/typesense && sudo docker compose up -d'

Check health:

bash

ssh $HOST 'curl -sS http://127.0.0.1:8108/health'

Expect {"ok":true}.

Firewall

This application is eu-central-1. Fetch the live list. Do not reuse a remembered IP list.

Check:

bash

ssh $HOST 'sudo ufw status && sudo iptables -S DOCKER-USER | head'

22/tcp should allow anywhere. Every 8108 line should name a Cloud IP or the Cloud IPv6 CIDR. There should be no 8108/tcp ALLOW Anywhere. DOCKER-USER should RETURN Cloud sources on tcp 8108, then DROP other tcp 8108.

On the laptop, print the region:

bash

curl -sS https://cloud.laravel.com/api/ip

Allow SSH:

bash

ssh $HOST 'sudo ufw allow OpenSSH'

Allow port 8108 from each eu-central-1 IPv4 address and from the IPv6 CIDR. Repeat the IPv4 line once per address from the JSON you just fetched.

bash

ssh $HOST 'sudo ufw allow from <cloud-ipv4> to any port 8108 proto tcp comment "Laravel Cloud eu-central-1"'
ssh $HOST 'sudo ufw allow from <cloud-ipv6-cidr> to any port 8108 proto tcp comment "Laravel Cloud eu-central-1"'

Do not add a laptop IP. Inspect Typesense through a tunnel:

bash

ssh -L 8108:127.0.0.1:8108 $HOST

bash

curl -sS -H "X-TYPESENSE-API-KEY: $TYPESENSE_API_KEY" http://127.0.0.1:8108/health

Docker publishes 8108:8108, so inbound packets skip UFW INPUT. Write the same sources on DOCKER-USER. Replace the block between the markers on each run so a second pass does not stack rules.

Insert this into /etc/ufw/after.rules inside *filter, before COMMIT. One -A DOCKER-USER -s … RETURN line per Cloud IPv4.

# BEGIN TYPESENSE CLOUD
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
-A DOCKER-USER -s <cloud-ipv4>/32 -p tcp -m tcp --dport 8108 -j RETURN
-A DOCKER-USER -p tcp -m tcp --dport 8108 -j DROP
-A DOCKER-USER -j RETURN
# END TYPESENSE CLOUD

Insert the IPv6 CIDR the same way in /etc/ufw/after6.rules:

# BEGIN TYPESENSE CLOUD
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
-A DOCKER-USER -s <cloud-ipv6-cidr> -p tcp -m tcp --dport 8108 -j RETURN
-A DOCKER-USER -p tcp -m tcp --dport 8108 -j DROP
-A DOCKER-USER -j RETURN
# END TYPESENSE CLOUD

Reload UFW so DOCKER-USER picks up the file:

bash

ssh $HOST 'sudo ufw reload'

✋ Manual. Confirm sudo ufw enable if the prompt asks. Stop here if the command waits for y. Ask if it is done. Wait.

bash

ssh $HOST 'sudo ufw enable && sudo ufw status && sudo iptables -S DOCKER-USER'

HTTP is enough while the allow list holds.

Nightly security updates

Check:

bash

ssh $HOST 'grep -E "Automatic-Reboot" /etc/apt/apt.conf.d/52smarthealth-reboot /etc/apt/apt.conf.d/50unattended-upgrades; systemctl list-timers apt-daily-upgrade.timer'

Expect reboot at 03:30 and the next apt-daily-upgrade.timer near 03:00 UTC, not 06:00.

Write /etc/apt/apt.conf.d/52smarthealth-reboot. Leave /etc/apt/apt.conf.d/50unattended-upgrades with -updates commented out. Daily runs stay on -security. Leave -proposed and -backports off.

bash

ssh $HOST 'sudo tee /etc/apt/apt.conf.d/52smarthealth-reboot' <<'EOF'
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:30";
EOF

Pin the daily upgrade timer to 03:00 UTC:

bash

ssh $HOST 'sudo mkdir -p /etc/systemd/system/apt-daily-upgrade.timer.d'
ssh $HOST 'sudo tee /etc/systemd/system/apt-daily-upgrade.timer.d/override.conf' <<'EOF'
[Timer]
OnCalendar=
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=10m
Persistent=true
EOF
ssh $HOST 'sudo systemctl daemon-reload && sudo systemctl restart apt-daily-upgrade.timer && systemctl list-timers apt-daily-upgrade.timer'

Dry-run security upgrades. Do not pipe to less.

bash

ssh $HOST 'sudo unattended-upgrade --dry-run --debug 2>&1 | grep -Ei "allowed.origin|resolute-security|resolute-updates" | head'

Allowed origins should include Ubuntu:resolute-security and should not include Ubuntu:resolute-updates.

Weekly host upgrades

Check:

bash

ssh $HOST 'systemctl list-timers typesense-weekly-upgrades.timer && test -x /usr/local/sbin/typesense-weekly-upgrades'

Write /usr/local/sbin/typesense-weekly-upgrades mode 755. Daily unattended-upgrades already applied -security. This command installs whatever is left, including -updates.

bash

ssh $HOST 'sudo tee /usr/local/sbin/typesense-weekly-upgrades' <<'EOF'
#!/bin/bash
set -euo pipefail

export DEBIAN_FRONTEND=noninteractive

apt-get update -qq
apt-get -y -o Dpkg::Options::=--force-confold upgrade

if [[ -f /var/run/reboot-required ]]; then
	shutdown -r +5 "weekly Typesense host upgrades"
fi
EOF
ssh $HOST 'sudo chmod 755 /usr/local/sbin/typesense-weekly-upgrades'

Write the oneshot unit:

bash

ssh $HOST 'sudo tee /etc/systemd/system/typesense-weekly-upgrades.service' <<'EOF'
[Unit]
Description=Weekly Ubuntu -updates for the Typesense host
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/typesense-weekly-upgrades
EOF

Write the Sunday timer:

bash

ssh $HOST 'sudo tee /etc/systemd/system/typesense-weekly-upgrades.timer' <<'EOF'
[Unit]
Description=Weekly Ubuntu -updates for the Typesense host

[Timer]
OnCalendar=Sun *-*-* 03:00:00
Persistent=true
RandomizedDelaySec=5m

[Install]
WantedBy=timers.target
EOF

Enable the timer:

bash

ssh $HOST 'sudo systemctl daemon-reload && sudo systemctl enable --now typesense-weekly-upgrades.timer && systemctl list-timers typesense-weekly-upgrades.timer'

Do not run /usr/local/sbin/typesense-weekly-upgrades during the day. systemd and containerd upgrades restart Docker. The first apply is Sunday 03:00 UTC.

Nightly image pin

This recreates the container only when typesense/typesense:30.2 has a new digest. It does not move you to 30.3 or 31.0. Version bumps are SHAPP-3723, every 3 months, on this repo and every droplet together.

Check:

bash

ssh $HOST 'grep image /opt/typesense/docker-compose.yml && systemctl list-timers typesense-image-refresh.timer'

Confirm the compose pin matches this repo:

bash

grep image scripts/typesense/docker-compose.yml docker-compose.yml
ssh $HOST 'grep image /opt/typesense/docker-compose.yml'

Write /usr/local/sbin/typesense-image-refresh mode 755:

bash

ssh $HOST 'sudo tee /usr/local/sbin/typesense-image-refresh' <<'EOF'
#!/bin/bash
set -euo pipefail

cd /opt/typesense

avail_kb=$(df --output=avail / | tail -1 | tr -d ' ')
if (( avail_kb < 2000000 )); then
	echo "Need at least 2 GB free before docker compose pull" >&2
	exit 1
fi

docker compose pull
docker compose up -d
docker image prune -f

curl -fsS --retry 5 --retry-delay 2 http://127.0.0.1:8108/health | grep -q '"ok":true'
EOF
ssh $HOST 'sudo chmod 755 /usr/local/sbin/typesense-image-refresh'

Write the oneshot unit:

bash

ssh $HOST 'sudo tee /etc/systemd/system/typesense-image-refresh.service' <<'EOF'
[Unit]
Description=Pull the pinned Typesense image and recreate if the digest changed
After=docker.service network-online.target
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/typesense-image-refresh
EOF

Write the 04:00 UTC timer:

bash

ssh $HOST 'sudo tee /etc/systemd/system/typesense-image-refresh.timer' <<'EOF'
[Unit]
Description=Nightly pinned Typesense image refresh

[Timer]
OnCalendar=*-*-* 04:00:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable the timer and run it once:

bash

ssh $HOST 'sudo systemctl daemon-reload && sudo systemctl enable --now typesense-image-refresh.timer && sudo /usr/local/sbin/typesense-image-refresh && curl -sS http://127.0.0.1:8108/health && sudo docker ps --format "{{.Image}} {{.Status}}"'

Expect {"ok":true} and typesense/typesense:30.2. Do not add smarthealth to the docker group. Use sudo docker for a one-off. Do not install Watchtower.

Do not run php artisan search:index --fresh after a reboot or a same-tag recreate.

Cloud and import

✋ Manual. In Laravel Cloud, set these on the right environment. QA droplet: this branch's QA, preview automation, and staging. Production droplet: production only. Do not use the Cloud CLI. Stop after each of the three blocks below and ask if it is done.

SCOUT_DRIVER=typesense
SCOUT_QUEUE=true
TYPESENSE_API_KEY=<droplet key>
TYPESENSE_HOST=<droplet public IPv4>
TYPESENSE_PORT=8108
TYPESENSE_PROTOCOL=http

Do not set SCOUT_PREFIX. Redeploy. On Commands, php artisan config:show scout.driver prints typesense.

✋ Manual. On a new droplet or after a schema change, run php artisan search:index --fresh from Commands. Wait for the low queue. Never scout:flush or scout:import --fresh. Do not add search:index to the deploy command. Do not run --fresh on every new QA preview.

✋ Manual. Sign in as employee@example.com / secretsecret. Walk Offer Search on that host per qa.md.

Production

Replay this file on a new host when SHAPP-3008 is done. New name, new API key, new Cloud IP list. Same pin as scripts/typesense/docker-compose.yml. Point only production at it. Staging stays on smarthealth-typesense-qa.

Check

bash

ssh $HOST 'whoami
sudo sshd -T | grep -Ei "permitrootlogin|passwordauthentication|allowusers"
systemctl is-enabled unattended-upgrades
systemctl list-timers apt-daily-upgrade.timer typesense-weekly-upgrades.timer typesense-image-refresh.timer
curl -sS http://127.0.0.1:8108/health
sudo iptables -S DOCKER-USER | head'
ssh $HOST-root 'md5sum /root/.bash_aliases /home/smarthealth/.bash_aliases; grep -c "install --only-upgrade" /root/.bash_aliases /home/smarthealth/.bash_aliases'

whoami prints smarthealth. sshd -T shows prohibit-password and allowusers smarthealth root. The three timers exist. Health is {"ok":true}. DOCKER-USER lists only Cloud sources on 8108. Both alias files share one checksum. Both grep counts are 1.

Typesense droplet setup ​

Agent rules ​

Create the droplet ​

Bootstrap ​

Operator user ​

Bash aliases ​

SSH and Web Console ​

Typesense ​

Firewall ​

Nightly security updates ​

Weekly host upgrades ​

Nightly image pin ​

Cloud and import ​

Production ​

Check ​

Typesense droplet setup

Agent rules

Create the droplet

Bootstrap

Operator user

Bash aliases

SSH and Web Console

Typesense

Firewall

Nightly security updates

Weekly host upgrades

Nightly image pin

Cloud and import

Production

Check