# Mails.so Transport Validation

## Context

AWS SES bounce rate is increasing. The mails.so API integration (`MailSoValidator`) already exists in the codebase but only runs in `SendEmailToContact::setContact()` — **before** emails enter the Symfony Messenger queue. When `messenger:consume email` picks up a message, Symfony's default `MessageHandler` calls `$transport->send()` directly, **bypassing all Mautic validation**. This creates a gap where retried or re-queued messages skip mails.so validation.

Additionally, the existing `QueueEmailValidationSubscriber` (listens to `EMAIL_RESEND`) was dead code — that event is never dispatched.

**Goal:** Validate every email right before the actual SMTP/SES send, with no double API calls.

---

## Implementation (completed 2026-03-18)

### 1. File-based cache added to MailSoValidator

**File:** `docroot/app/bundles/EmailBundle/Helper/MailSoValidator.php`

- Added 3 private methods: `getCachePath()`, `getFromCache()`, `writeToCache()`
- Cache dir: `{projectDir}/var/cache/mailso/`
- Cache key: `md5(strtolower(email)).json`
- Cache TTL: 24 hours (86400 seconds)
- Atomic writes: write to `.{pid}.tmp` then `rename()` (safe for concurrent workers)
- `validateEmail()` modified:
  - After whitelist/API-key checks, checks cache first
  - On cache miss, calls API as before
  - After successful API call, writes to cache
  - Never caches API failures (fail open)
- All cache I/O uses `@` suppression — degrades gracefully to always-call-API

### 2. Created MailSoTransportValidationSubscriber

**New file:** `docroot/app/bundles/EmailBundle/EventListener/MailSoTransportValidationSubscriber.php`

- Listens to `Symfony\Component\Mailer\Event\MessageEvent` at priority 255
- Only acts when `isQueued() === false` (the actual transport send during messenger:consume)
- Only acts on `MauticMessage` instances (skips password resets, test emails, etc.)
- For each envelope recipient:
  - Calls `MailSoValidator->validateEmail()` (hits cache — no double API call)
  - If not deliverable: logs it, adds DNC with `'Mail So validation: not deliverable (transport)'`, removes from recipient list
- If ALL recipients invalid: throws `UnrecoverableMessageHandlingException` (no retries)
- If some invalid: creates new `Envelope` with valid recipients only, updates `To`/`Cc`/`Bcc` on the message
- Dependencies: `MailSoValidator`, `DoNotContact`, `LoggerInterface` — all autowired

### 3. No manual service registration needed

The bundle's `services.php` already uses `autowire()` + `autoconfigure()` and loads the entire bundle directory. The new subscriber is auto-discovered and registered at priority 255 (#1 listener on `MessageEvent`).

### 4. Deleted dead code QueueEmailValidationSubscriber

**Deleted:** `docroot/app/bundles/EmailBundle/EventListener/QueueEmailValidationSubscriber.php`

Listened to `EMAIL_RESEND` which was never dispatched. Replaced by the new transport-level subscriber.

### 5. SendEmailToContact validation unchanged

The existing validation at `SendEmailToContact::setContact()` stays. It catches bad emails **early** (prevents stat creation, saves queue overhead). The cache ensures the transport-level check is free (cache hit, no API call).

---

## Config Changes

| Setting | Old Value | New Value |
|---------|-----------|-----------|
| `mailso_api_key` | `null` | `040b1b82-3d0e-4c23-93be-344a891fdaea` |
| `messenger_dsn_email` | `doctrine://default` | `doctrine://default?queue_name=default` |

The `?queue_name=default` was required because existing messages in the DB use `queue_name = 'default'` (the Doctrine transport default), not `email` (the Symfony transport name).

---

## Files Modified/Created

| File | Action |
|------|--------|
| `docroot/app/bundles/EmailBundle/Helper/MailSoValidator.php` | Added caching layer |
| `docroot/app/bundles/EmailBundle/EventListener/MailSoTransportValidationSubscriber.php` | **Created** — transport-level validation |
| `docroot/app/bundles/EmailBundle/EventListener/QueueEmailValidationSubscriber.php` | **Deleted** — dead code |
| `config/local.php` | Set `mailso_api_key` and fixed `messenger_dsn_email` |

---

## How Double API Calls Are Prevented

```
SendEmailToContact::setContact()
  -> MailSoValidator::validateEmail("user@test.com")
    -> Cache miss -> API call -> cached 24h
    -> Returns deliverable/not

messenger:consume email
  -> AbstractTransport::send()
    -> MessageEvent dispatched
      -> MailSoTransportValidationSubscriber::onMessage()
        -> MailSoValidator::validateEmail("user@test.com")
          -> Cache HIT -> returns cached result (no API call)
```

---

## Key Findings During Implementation

- **Messenger queue_name mismatch:** Messages in DB use `queue_name=default` (Doctrine transport default), but the Symfony transport name is `email`. Without `?queue_name=default` in the DSN, the consumer polls for `queue_name=email` and finds nothing.
- **Cron had wrong PHP path:** `usr/local/bin/ea-php82` (missing leading `/`) — consumer never ran, 27k+ messages accumulated in the queue.
- **Consumer output is silent:** `messenger:consume email` shows no per-message output even with `-vvv`, but IS processing. Use DB stats to verify (check `email_stats` and `messenger_messages` counts).
- **Safe to run 2 concurrent workers:** Doctrine transport uses `SELECT ... FOR UPDATE` row locking — no duplicate sends.
- **mails.so API limit is 10 calls/sec:** With cache, concurrent workers stay well under the limit. Most lookups are cache hits (free file reads).

---

## Concurrent Workers

Two `messenger:consume email` cron jobs can run safely:
- Doctrine transport uses `SELECT ... FOR UPDATE` row locking — each worker picks a different message
- mails.so cache prevents API rate limit issues (most lookups are cache hits)
- No duplicate email sends possible

---

## Live Test Results (2026-03-18)

| Metric | Value |
|--------|-------|
| Emails sent successfully | 1,137 |
| Failed sends | 0 |
| Transport DNC entries created | 57 |
| Cache files created | 1,261 |
| Processing speed | ~1.9s per message (with API calls for uncached emails) |

---

## Verification Commands

```bash
# Cache file count
ls var/cache/mailso/ | wc -l

# Undeliverable email log
cat var/logs/undeliverable_emails_$(date +%Y-%m-%d).json

# Transport DNC count
SELECT COUNT(*) FROM lead_donotcontact WHERE comments LIKE '%transport%';

# Queue count
SELECT COUNT(*) FROM messenger_messages WHERE queue_name = 'default';

# Emails sent today
SELECT COUNT(*) FROM email_stats WHERE date_sent >= CURDATE();

# Clear cache after changes
/opt/cpanel/ea-php82/root/usr/bin/php bin/console cache:clear --env=prod

# Run consumer manually
/usr/local/bin/ea-php82 /home/whizzmailer/public_html/swiss-belhotel-v2/bin/console messenger:consume email --limit=50 --time-limit=55 --env=prod --no-interaction
```