# Always Be Selling — Technical Changelog

All changes in chronological order.

---

## [2026-05-26] — MILESTONE 2: Three-Layer Dedup, Sequence Eligibility, Renewals, Import

### Three-Layer Dedup Architecture

Every lead entry point (harvest, CSV import, renewal scan, manual create) now passes
through three dedup layers before entering the backlog.

#### Layer 1: Suppression Registry (`state/suppressions.py`)

- **NEW**: `state/suppressions.py` — append-mostly store at `data/suppressions.json`
- **ADDED**: `is_suppressed(email)` — checks exact email then wildcard domain match
- **ADDED**: `suppress_email(email, reason, source_lead_id)` — suppress specific address
- **ADDED**: `suppress_domain(domain, reason, source_lead_id)` — suppress entire domain
- **ADDED**: `remove_suppression(email_or_domain)`, `list_suppressions()`
- Auto-populated when `burn_lead()` is called with `"opt_out"` in the reason
- **CLI**: `absel suppress list | add | add-domain | remove | check`

#### Layer 2: Zoho Dedup (`state/zoho_dedup.py`)

- **NEW**: `state/zoho_dedup.py` — Zoho CRM dedup using paginated API
- **ADDED**: `list_all_records()` in `shared/zoho/client.py` — paginated fetcher for
  all Zoho records (supports `Leads` and `Contacts` modules)
- **ADDED**: `build_zoho_email_sets()` — single API call fetches all Lead + Contact
  emails into O(1) lookup sets
- **ADDED**: `check_email_in_sets(email, lead_set, contact_set)` — O(1) set membership
- **ADDED**: `is_in_zoho(email)` — per-lead mode for single operations (renewal create)
- **Batch mode**: Used by harvest and CSV import — pre-fetch once, check many
- **Per-lead mode**: Used by `create_renewal_lead()` and `create_renewal_lead_manual()`
- **Fail open**: API/network errors treated as "not found" — harvests don't fail
- **ADDED**: `--skip-zoho-check` flag on all 4 harvest commands and import command

#### Layer 3: Backlog + In-File Dedup

- Within-file dedup (CSV import: same email appearing twice in one file)
- Cross-file dedup (existing backlog emails)
- Renewal dedup by `zoho_deal_id` (must be unique)

### Sequence Eligibility Guard (`capabilities/sequencing.py`)

- **ADDED**: `_get_sequence_state(lead)` — classifies leads into 6 states:
  `eligible`, `active`, `paused`, `completed`, `burned`, `preview_only`
- **ADDED**: `check_sequence_eligibility(lead_id)` — safety guard called before
  `preview_sequence()`, `approve_sequence()`, and `sequence_batch()`
- **ADDED**: `_get_cooling_days(preset)` — per-preset cooling duration lookup
- **ADDED**: `_cooling_period_active(last_sent_date)` — checks if cooling in effect
- **ADDED**: `_cooling_days_remaining(lead)` — days left in current cooling period
- **ADDED**: `_find_substitutes(blocked_specs, proposed_ids, backlog)` — 4-pass
  matching strategy for auto-substitution in batch mode
- **ADDED**: `_best_eligible_lead(candidates)` — picks highest-scoring substitute

#### Cooling Period Config (`config/sequences.yaml`)

```yaml
cooling:
  default_days: 120
  by_preset:
    light_45: 90
    standard_60: 120
    deep_90: 180
    extended_120: 240
    urgent_21: 60
    renewal_90: null    # no cooling — event-driven
```

#### Wire-in Points

- `_build_prompt()` — cooling context injected into all prompts
- `absel sequence due` — blocked leads hidden from due list
- `sequence_batch()` — blocked leads replaced by substitutes
- `approve_sequence()` — final safety check before storage
- All CLI output updated with dedup-aware icons and counts

### Global Operator Intel (`config/intel.yaml`)

- **ADDED**: `config/intel.yaml` — global operator intel store
- **ADDED**: `absel intel global "message"` — set global intel text
- **ADDED**: `absel intel global --off` — disable global intel
- **ADDED**: `absel intel global --status` — show current intel
- Injected into all drafts and sequence touches when active

### CSV Import (`capabilities/import_leads.py`)

- **NEW**: `capabilities/import_leads.py` — bulk CSV import for trade show lists,
  purchased lists, partner referrals
- **ADDED**: Auto-column detection from 13 known aliases (email, name, phone, city,
  state, zip, naics, equipment fields, contract expiration, notes)
- **ADDED**: Auto-encoding detection (UTF-8 → Latin-1 → cp1252)
- **ADDED**: Per-row: suppression → Zoho dedup → in-file dup → backlog dup →
  build lead → schema validate
- **ADDED**: `--mapping` JSON override for explicit column mapping
- **ADDED**: Renewal support: requires `equipment_model` + `contract_expiration`,
  auto-runs `recommend_upgrade()`
- **CLI**: `absel import leads <csv_path> [--type cold|renewal] [--dry-run]
  [--intel] [--mapping] [--skip-zoho-check]`

### Sequence Trace in Zoho (`integrations/zoho.py`)

- **ADDED**: `_push_sequence_trace_to_zoho(zoho_id, sequence)` — writes sequence
  spec and touch content to Zoho custom fields on Leads module
- **Sequence field**: `"4x75 — approved 2026-05-26"` (touches × days + approval date)
- **Touch1–Touch6 fields**: Per-touch content with purpose, differentiator, subject,
  and full message body (capped at 2000 chars each)
- **Deferred push**: If lead has no `zoho_id` at approval time, sets
  `sequence_trace_pending` flag — pushed when lead next gets a `zoho_id`
- **Wired into**: `approve_sequence()`, `sequence_batch()` (approve=True),
  `push_lead_to_zoho()` (deferred)
- Zoho field API names: `Sequence`, `Touch1`–`Touch6`

### Renewal Manual Creation

- **ADDED**: `create_renewal_lead_manual()` in `capabilities/renewals.py` — operator
  supplies email, name, equipment model, expiration date
- Fills defaults: company from email domain, contact name from email prefix
- Runs full dedup: suppression → Zoho check → backlog duplicate
- **CLI**: `absel renewals create --email --name --model --expiration [--contact]
  [--serial] [--mono] [--color] [--phone] [--notes] [--intel] [--force]`
- **ADDED**: `cancel_renewal_lead()` — sets lead_status to `"cancelled"`
- **CLI**: `absel renewals cancel <lead_id>`

### Bug Fixes

**OpenClaw integration fix**:
- Removed stale plugin entries (`index`, `foundry-openclaw`) from OpenClaw config
- Switched context engine from missing `lossless-claw` to `legacy`
- Removed lossless-claw npm package + extension dirs

**Renewal NoneType fix** in `recommend_sequence()`:
- `lead.get("research_note", {}).get("findings")` failed with AttributeError when
  `research_note` was explicitly `None` (Python `dict.get` only returns default
  for *missing* keys, not `None` values)
- **Fixed**: Changed to `(lead.get("research_note") or {}).get("findings")`

**Envelope clamping fix** in `recommend_sequence()`:
- `recommend_sequence()` could select `extended_120` (8 touches) after competitor
  bump, but `_max_touches_for_lead()` returned 7 — `preview_sequence()` then
  crashed with "insufficient data"
- **Fixed**: Added clamping loop that walks down tier order until
  `preset_touches <= max_touches`

**Harvest summary formatting**:
- Zoho dedup count now always visible in harvest output (not conditionally hidden)

**CLI import output**:
- Added Zoho dedup count to import CLI summary display
- Added `--skip-zoho-check` forwarding to `import_from_csv()`

### Files Changed

| File | Change |
|---|---|
| `state/suppressions.py` | NEW — suppression registry |
| `state/zoho_dedup.py` | NEW — Zoho CRM dedup module |
| `capabilities/import_leads.py` | NEW — CSV import |
| `config/intel.yaml` | NEW — global intel store |
| `config/sequences.yaml` | ADDED — cooling period config |
| `capabilities/sequencing.py` | ADDED — eligibility guard, substitute finder, cooling helpers |
| `capabilities/renewals.py` | ADDED — manual create, cancel, skip_zoho_check forwarding |
| `integrations/zoho.py` | ADDED — sequence trace push, deferred push, send-person fix |
| `shared/zoho/client.py` | ADDED — `list_all_records()` paginated fetcher |
| `cli/main.py` | ADDED — import, suppress, create, cancel, skip_zoho_check flags, dedup display |
| `capabilities/harvest.py` | ADDED — Zoho dedup integration, skip_zoho_check, summary format |
| `.openclaw/` | FIXED — removed stale plugins, legacy engine |

### Data State After Milestone 2

- 23,243 leads in backlog
- 2,041 fresh, 11,422 recent, 9,772 historical
- 8 renewal leads (7 active, 1 cancelled)
- 53 drafted (all categories)
- 21 sent to Zoho
- 77 leads with sequences
- 68 researched
- 0 suppressions (clean slate)
- Gate: OPEN

---

## [2026-05-21] — Zoho CRM Integration (Bidirectional)

## [2026-05-21] — Zoho CRM Integration (Bidirectional)

### Part 1: OAuth2 Authentication (`integrations/zoho.py`)

- **ADDED**: `ZohoAuth` class — manages OAuth2 tokens with automatic refresh
- Ported from `gardener-fork/lib/zoho.py` — working refresh-token flow
- Credentials from env vars: `ZOHO_CLIENT_ID`, `ZOHO_CLIENT_SECRET`, `ZOHO_REFRESH_TOKEN`
- Access tokens cached with 60s expiry buffer
- `_zoho_api()` helper for authenticated API calls

### Part 2: Push to Zoho

- **ADDED**: `push_lead_to_zoho()` — creates or updates Zoho Lead record
- **ADDED**: `push_touch_to_zoho()` — pushes sequence touch to Zoho
- **ADDED**: `push_batch()` — bulk push with counts
- **ADDED**: `_enforce_push_prerequisites()` — validates approval gate, filing-service filter, gate state
- **ADDED**: `absel push <id>` — push single lead
- **ADDED**: `absel push --batch` — push all approved leads
- **ADDED**: `absel push --dry-run` — preview without pushing
- Field mapping: Last_Name, Company, Email, Phone, City, State, Lead_Source, Email_Draft_Subject, Email_Draft_Body2, Lead_Score, Lead_Category
- Upsert logic: searches by email, then company name — no duplicates

### Part 3: Reply Polling + Opt-Out Detection

- **ADDED**: `detect_opt_out()` — scans reply for 21 hard + 12 soft opt-out phrases
- **ADDED**: `process_reply()` — central reply handler (burn/pause/surface)
- **ADDED**: `check_for_replies()` — polls Zoho Activities API for email replies
- **ADDED**: `get_due_touches_safe()` — reply check BEFORE listing due touches
- **ADDED**: `get_replies_summary()` — list all leads with replies
- **ADDED**: `absel replies check` — poll and process new replies
- **ADDED**: `absel replies list [--unhandled]` — show replied leads
- Reply state stored in `data/reply_check_state.json`

### Opt-Out Actions

- **Hard opt-out**: Lead burned (permanent), sequence cancelled, no response drafted
- **Soft opt-out**: Lead paused, sequence paused, operator decides next
- **Engagement**: Sequence paused, operator drafts response

### Pre-Delivery Safety

- `absel sequence due` calls `get_due_touches_safe()` — polls replies first
- If Zoho poll fails, delivery is blocked (fail safe)
- Burned/paused leads excluded from due-touches list

### Required Zoho Custom Fields

Documented in README: Email_Draft_Subject, Email_Draft_Body2, Lead_Score, Lead_Category

---

## [2026-05-21] — Socrata Integration, Competitive Intelligence, Reply Handling, Filing History

### Phase 1: Agent & Principal Integration

**New Socrata datasets** (`integrations/ctsos.py`):
- **ADDED**: `AGENTS_API_BASE` (`qh2m-n44y`), `PRINCIPALS_API_BASE` (`ka36-64k6`), `FILING_HISTORY_API_BASE` (`ah3s-bes7`)
- **ADDED**: `fetch_agent_for_business()` — returns agent_name, agent_type, agent_is_self
- **ADDED**: `fetch_principals_for_business()` — returns list with name, first_name, last_name, designation, type. Sorted by designation priority (President > CEO > Managing Member > Member > Director > Officer)
- **ADDED**: `fetch_agents_bulk()`, `fetch_principals_bulk()` — batch size 100, uses `$where=key IN (...)`
- **ADDED**: `fetch_filing_history()`, `fetch_filing_history_bulk()` — filing history signals
- **ADDED**: `classify_agent_type()` — classifies as individual/business/secretary_of_state
- **ADDED**: `_designation_priority()` — sorts principals by role importance

**Intentional exclusions**:
- Agent phone/email NOT stored (ethical/legal review pending)
- Principal residence addresses NOT stored (personal information)

**Schema updates** (`state/schema.py`):
- **ADDED**: `agent_type`, `agent_is_self`, `business_id` (join key)
- **ADDED**: `principals`, `principal_names`, `primary_principal_name`, `principal_designation`
- **ADDED**: `greeting_name`, `greeting_name_source`
- **ADDED**: `last_filing_date`, `last_filing_type`, `filing_count`, `has_recent_amendment`, `uses_paper_filing`, `annual_report_current`
- **ADDED**: `reply_received`, `reply_received_at`, `reply_text`, `response_drafted`, `response_body`
- **ADDED**: `principal_competitor_overlap`, `principal_competitor_detail`, `address_competitor_overlap`, `address_competitor_detail`

**Enrichment integration** (`capabilities/enrichment.py`):
- Agent/principal data fetched during enrichment (bulk via `enrich_batch()`)
- Filing history gated on historical leads or score ≥ 70
- `primary_principal_name` extracted from first principal with type == "Individual"
- `agent_is_self` flag set when agent_type == "individual"

**Greeting resolution** (`capabilities/drafting.py`):
- **NEW**: `_looks_like_person_name()` — rejects company/entity names
- **UPDATED**: `_determine_greeting()` priority chain:
  1. Principal name (person-like only)
  2. Agent name (individual type only)
  3. First name from personal-name PLLC
  4. Short company name
  5. None

### Phase 2: Filing History Signals

**Scoring integration** (`capabilities/scoring.py`):
- **ADDED**: `has_recent_amendment` → +0.08 priority weight
- **ADDED**: `uses_paper_filing` → +0.05 priority weight
- **ADDED**: `annual_report_current` → +0.03 priority weight
- Small supplementary weights — phone and custom email remain dominant

### Phase 3: Competitive Intelligence

**New capability** (`capabilities/competitive_intel.py`):
- **ADDED**: `agent_displacement_report()` — groups by agent, calculates displacement rate vs baseline, flags HIGH at >1.5x
- **ADDED**: `principal_overlap_report()` — cross-references principals of competitor-flagged leads, writes flags back
- **ADDED**: `address_clustering_report()` — normalizes addresses, finds multi-business locations with competitor presence, writes flags back
- **ADDED**: `win_rate_report()` — stub for future Zoho integration
- **ADDED**: `_normalize_address()` — lowercase, strip punctuation, expand 20 common abbreviations

**New CLI** (`cli/main.py`):
- **NEW**: `absel intel agents [--min-filings N]`
- **NEW**: `absel intel principals [--competitor-only]`
- **NEW**: `absel intel addresses [--min-businesses N]`
- **NEW**: `absel intel wins`

### Phase 4: Reply Handling

**New capability** (`capabilities/respond.py`):
- **NEW**: `draft_response(lead_id, reply_text)` — contextual response drafting
- Reads original touch, enrichment data, equipment context
- Principles: acknowledge reply, answer questions, stay on product, one next step, 50-80 words
- Response printed for review — NOT sent automatically
- State tracking: `reply_received`, `reply_received_at`, `reply_text`, `response_drafted`, `response_body`

**CLI** (`cli/main.py`):
- **NEW**: `absel respond <lead_id> "<reply text>"`
- **NEW**: `absel respond <lead_id> --reply-file reply.txt`

### Phase 5: Drafting Refinements

**Services knowledge** (`config/services-knowledge.yaml`):
- **UPDATED**: `month_to_month` fact → "We rent, we don't lease. No multi-year contract, cancel anytime."
- **ADDED**: `alt_phrasing` support for differentiators
- **ADDED**: `speed_of_setup` differentiator — "Most businesses printing within days"

**Prompt updates** (`capabilities/drafting.py`):
- **ADDED**: Empathy-first opening principle for historical leads
- **ADDED**: Alt phrasing instruction for close differentiators

### Phase 6: Outside Sales Export

**New module** (`cli/export_outside_sales.py`):
- **NEW**: `get_outside_sales_leads(county_filter)` — qualification: no usable email, no phone, has address
- **NEW**: `format_txt()` — county-grouped, plain ASCII, New Haven first
- **NEW**: `format_csv()` — county,name,address,city,state,zip,filed,naics,score,reason
- **NEW**: `--limit` flag on export command

**CLI** (`cli/main.py`):
- **NEW**: `absel export outside-sales [-c COUNTY] [-l LIMIT] [-f txt|csv] [-o FILE]`

### Tests
- All 49 tests passing
- Fixed 5 drafting test failures (mock responses missing phone in signature)

---

## [2026-05-20] — Data Quality Fixes: Filing-Service Emails, Greetings, Signal Tiers

### Filing-Service Email Detection (`integrations/ctsos.py`, `capabilities/enrichment.py`)

- **ADDED**: `FILING_SERVICE_DOMAINS` list (20 domains: incfile.com, legalzoom.com, northwestregisteredagent.com, etc.)
- Detection runs during CT SoS field mapping AND as second line of defense in enrichment
- Sets `email_is_filing_service: true` on lead — email kept for record-keeping but treated as unusable
- Review queue shows `⚠ OUTSIDE SALES` tag for filing-service leads with no phone
- `approve` command rejects filing-service leads without real email: "Route to outside sales or enrich first"

### Greeting Fallback Rules (`capabilities/drafting.py`)

- **NEW**: `_determine_greeting()` with strict priority chain:
  1. Agent name from CT SoS (if looks like person, not company/government/filing service)
  2. First name from personal-name PLLC ("Jane Doe PLLC" → "Jane")
  3. Short company name (≤35 chars, entity suffix stripped)
  4. None (email starts without salutation)
- **NEVER** extracts from email prefix
- **NEVER** uses both person name AND company name
- Added `GREETING` and `NAMES` rules to prompt

### Robert Hallucination Fix

- **Root cause**: `brave_summary` contained "Name: ROBERT A. COSTANZO" from Brave search — LLM extracted it as greeting
- **Fix**: `_determine_greeting()` never looks at `brave_summary` for names
- **Prompt rules**: "Never invent, assume, or guess a recipient's name. Use ONLY names that appear in the greeting_name field."

### Signal Tier Display (`capabilities/drafting.py`, `capabilities/orchestration.py`, `cli/review.py`)

- **NEW**: `_compute_signal_tier()` — rates leads by contact signal quality:
  - ★★★ Strong: real email + phone + live website
  - ★★ Good: real email + phone OR real email + live website
  - ★ Basic: real email only OR phone only
  - ⚠ Weak: filing-service email, no phone, no website
- Display in review queue and brief
- **NEW**: `--strong-only` flag on `absel review` and `absel brief` — shows only ★★+ leads

---

## [2026-05-20] — Draft Review Queue

### Review Command (`cli/review.py`, `cli/main.py`)

- **NEW**: `absel review` — Interactive paginated review of drafted emails
- Displays numbered list with tags (`[FRESH]`, `[RECENT]`, `[2-YR]`, `[5-YR]`, `[EST]`), subjects, and 120-char body previews
- Commands: `burn #N [reason]`, `pause #N [reason]`, `delete #N`, `regen #N`, `approve #N`, `next`, `prev`, `quit`
- Page-relative numbering (always #1-#25 regardless of page)
- `--category fresh|recent|historical` filter before pagination
- `--page-size N` (default: 25)
- `approve #N` sets `reviewed_at` timestamp — mandatory gate for future Zoho push
- `regen #N` clears draft, calls LLM, displays new subject/preview inline
- Bad input produces usage hint, loop stays active until `quit`

### Schema (`state/schema.py`)

- **ADDED**: `reviewed_at` — timestamp when operator approves draft for sending
- **ADDED**: `regen_count` — tracks how many times a lead has been regenerated

### Backlog Helpers (`state/backlog.py`)

- **NEW**: `get_drafted_leads(category)` — returns drafted leads sorted by drafted_at desc
- **NEW**: `clear_draft(lead_id)` — removes draft, returns lead to undrafted state
- **NEW**: `approve_lead(lead_id)` — sets reviewed_at timestamp
- **NEW**: `increment_regen_count(lead_id)` — increments regen_count, clears reviewed_at

---

## [2026-05-20] — Age-Aware Close Principles + ABS Differentiators

### Close Principles (`capabilities/drafting.py`)

- **ADDED** CLOSE section to drafting rules with age-aware behavior:
  - Fresh/recent leads: one-sentence seed close, no pitch, no meeting asks
  - Historical/milestone leads: ONE specific differentiator fact, stated plainly, no CTA after
  - All leads: banned hard-drop closes ("I can outline", "I can brief you", "let's discuss", "when you're ready", "you'll know who to call", "happy to chat")
  - Strongest close = last substantive sentence + signature, nothing else

### Close Differentiators (`config/services-knowledge.yaml`)

- **ADDED** `close_differentiators` section with 6 facts for historical leads:
  - `month_to_month`: ABS offers month-to-month service, no locked lease required
  - `toner_included`: Parts, labor, toner, staples all included in service agreements
  - `heavy_media`: Xerox machines handle 110lb cardstock without bypass tray
  - `scan_to_email`: One-touch scan-to-email and scan-to-folder, no multi-step process
  - `no_credit_rule`: No 2-year credit history requirement
  - `total_satisfaction`: Replace equipment at no charge if not satisfied, 3-year guarantee
- LLM picks ONE differentiator most relevant to lead's industry
- Differentiators are facts, not pitches. No question format ("Did you know...?"). No CTA after.

### Historical Framing Refinement (`capabilities/drafting.py`)

- **UPDATED** temporal framing principle: fresh leads can say "saw your filing" (honest, natural). Historical leads must not narrate filing discovery — reason for outreach is business age + equipment lifecycle.

---

## [2026-05-20] — Brief Filters, Product Anchor, Phone Validation, Em Dash Ban

### Brief Filters (`capabilities/orchestration.py`, `cli/main.py`)

- **REMOVED**: Stage-priority tier ordering (fresh_pending → recent_pending → ...)
- **Priority score** is now the sole sort key — always
- **ADDED**: `--category fresh|recent|historical` flag to filter by temporal category
- **ADDED**: `--drafted-only` flag to show only leads with `draft_status == "drafted"`
- **ADDED**: `--undrafted-only` flag to show only leads with `draft_status == "none"`
- Flags combine: `absel brief --limit 30 --category historical --drafted-only`
- Brief header shows active filter label when filters applied
- Extracted `_select_leads()` helper for clean filter + sort logic

### Product Anchor (`capabilities/drafting.py`)

- **ADDED** to DRAFTING RULES: "You sell copiers, printers, and multifunction devices. Physical machines that print, copy, scan, and fax... Every problem you describe must be solved by a physical machine in their office. If the sentence would still make sense without a copier involved, cut it."
- Explicitly excludes: document management software, workflow consulting, digital transformation, IT services

### Phone Validation (`capabilities/drafting.py`)

- Signature block in prompt includes `CT 203-777-0011 x743` as required text
- `_parse_llm_response` raises `ValueError` if `203-777-0011` is not found in draft body
- Drafts without the phone number are rejected entirely

### Em Dash Ban (`capabilities/drafting.py`)

- **Prompt rule**: "Never use em dashes (—). Use two hyphens (--) instead. Non-negotiable."
- **Post-generation**: `_parse_llm_response` replaces all `—` with `--` in subject, body, and anchor
- Double enforcement: prompt instruction reduces them, code replacement guarantees zero in output

---

## [2026-05-20] — Outreach Control: Burn, Pause, Unpause

### Schema (`state/schema.py`)

- **ADDED**: `burned` to `lead_status` enum
- **ADDED**: `burn_reason`, `burned_at`, `paused_at`, `paused_reason` optional fields

### Backlog Helpers (`state/backlog.py`)

- **NEW**: `burn_lead(lead_id, reason)` — Sets status to `burned`, records reason + timestamp
- **NEW**: `pause_lead(lead_id, reason)` — Sets status to `paused`, records reason + timestamp
- **NEW**: `unpause_lead(lead_id)` — Returns to `active` status, validates lead is paused
- **NEW**: `count_by_status()` — Counts leads by status

### CLI (`cli/main.py`)

- **NEW**: `absel burn <lead_id> [reason]` — Permanent removal from all outreach
- **NEW**: `absel pause <lead_id> [reason]` — Stops touch clock until unpaused
- **NEW**: `absel unpause <lead_id>` — Resumes outreach
- **NEW**: `absel counts` — Shows lead counts by status and stage

### Orchestration (`capabilities/orchestration.py`)

- **CHANGED**: Pipeline excludes `burned`, `paused`, `imported_legacy`, and `dead` leads from selection
- **CHANGED**: Brief header shows pending/drafted counts per temporal category
- **CHANGED**: Each lead shows `✓ drafted` or `○ no draft` indicator

### Drafting (`capabilities/drafting.py`)

- **ADDED**: `draft_status: "drafted"` set on successful draft generation

---

## [2026-05-20] — Drafting: Principles Over Branches

### Drafting Refactor (`capabilities/drafting.py`)

- **REMOVED**: `SCENARIO_FRAMING` dict (8 scenario-specific prompt branches)
  - `new_filing`, `recent_operating`, `milestone_2yr`/`3yr`/`5yr`/`7yr`/`10yr`, `established_business`
  - Each had hardcoded framing instructions that forced the LLM into a specific tone
  - **Replacement**: Single `TEMPORAL FRAMING PRINCIPLE` instruction:
    "Let the business's age inform your framing naturally. A recent filing means you noticed it.
    A business that's been operating for years means equipment decisions come around again at this
    stage. If they're near a milestone anniversary, it's worth acknowledging. Don't force a
    celebration."

- **CHANGED**: Temporal data moved into `WHO THEY ARE` lead profile
  - `filing_date`, `business_age_months`, `temporal_category`, `milestone_year` (if applicable)
  - LLM sees the data and decides how to use it — no branching logic

- **KEPT**: `_select_scenario()` function — now only stores scenario tag as `draft_scenario`
  metadata on the lead for tracking purposes. Not used for prompt construction.

- **CHANGED**: `_build_prompt()` signature — no longer takes `scenario` argument
  - Was: `_build_prompt(scenario, ctx)`
  - Now: `_build_prompt(ctx)`

### Substance Source (`capabilities/drafting.py`, `config/services-knowledge.yaml`)

- **REMOVED**: Keyword-matching of pain points into prompt based on NAICS
  - Was matching NAICS keywords to `services-knowledge.yaml` pain points and injecting
    them as "LIKELY WORKFLOW CONTEXT" — created a second competing substance source
  - **Now**: Equipment talk track from enrichment is the sole evidence-based substance source

- **CHANGED**: Pain points in `services-knowledge.yaml` labeled as category-level reference
  - "CATEGORY-LEVEL PAIN POINTS (reference only — these are common for these verticals,
    NOT necessarily true for this specific business)"
  - LLM sees them as general context, not as facts about this business

- **KEPT**: Company name and never-say rules in `services-knowledge.yaml`

### Subject Line Constraints (`capabilities/drafting.py`)

- **REPLACED**: Old instruction ("use their name, city, or business type")
  with tight constraints:
  - 4-8 words
  - Must reference one specific detail about THIS business's likely workflow or situation
  - Could this subject have been sent to any other business in the same city and industry?
    If yes, rewrite
  - Never use 'For [Name] in [City]'
  - Never start with the business name
  - Never start with 'A note' or 'Setting up'

### Breaking Changes
- **NONE**: Prompt structure changed but output format (SUBJECT/body/ANCHOR) unchanged
- `draft_scenario` field added to lead metadata (new field, not breaking)

---

## [2026-05-20] — Baseline 1.0 Parity Release

### Core Architecture
- **NEW**: Complete state layer implementation with atomic file operations
- **NEW**: Cross-process locking using `fcntl` for `backlog.json`
- **NEW**: Full schema validation on all save/update operations

### Integrations
- **NEW**: `integrations/llm.py` — Featherless API client with reasoning-field fallback
- **NEW**: `integrations/brave.py` — Brave Search API with county caching
- **NEW**: `integrations/domain.py` — Website checks, phone scraping, competitor detection
- **FIXED**: Critical GLM-5.1 bug fix: empty content field falls back to reasoning field

### Capabilities
- **NEW**: `capabilities/scoring.py` — Complete scoring engine ported from gardener
  - PLLC fast-track detection (primary/secondary tiers)
  - NAICS tier scoring with keyword fallback
  - Email domain tiering (professional TLD, custom business, free provider)
  - Location quality, recency bonuses, formation signals
  - Priority calculation with readiness weights
- **NEW**: `capabilities/enrichment.py` — Enrichment orchestration
  - Pattern A (standard): domain + Brave + competitor + equipment
  - Equipment mapping from NAICS codes
  - Sequential batch processing
- **NEW**: `capabilities/drafting.py` — LLM email drafter
  - Full prompt assembly matching gardener structure
  - "Use don't list" instruction for substance context
  - THREE-BLOCK lead profile (WHO/HOW/CONTACT)
  - Historical vs. fresh prompt variants
  - ANCHOR line extraction
- **NEW**: `capabilities/orchestration.py` — End-to-end pipeline
  - Coordinates harvest → score → enrich → draft → brief
  - Markdown brief generation
  - JSON output option

### CLI
- **NEW**: `brief` command added to `cli/main.py`
  - `--days` — Lookback days for harvest
  - `--limit` — Max leads to display
  - `--no-draft` — Skip LLM drafting
  - `--output` — Save brief to file

### Testing
- **NEW**: `tests/test_backlog.py` — 9 tests for state layer
- **NEW**: `tests/test_scoring.py` — 5 tests for scoring logic
- **NEW**: `tests/test_drafting.py` — 5 tests for drafting
- **NEW**: `tests/test_llm_parsing.py` — 5 tests for reasoning-field fallback
- **NEW**: `tests/test_orchestration.py` — 2 tests for end-to-end pipeline
- **TOTAL**: 49 tests passing

### Configuration
- **COPIED**: `config/gardener.json` — Full scoring configuration
- **COPIED**: `config/enhanced_shell_patterns.json` — Name pattern rules
- **COPIED**: `config/equipment-mapping.json` — NAICS to equipment mapping

### Bug Fixes
- **FIXED**: `capabilities/harvest.py` import bug — `get_lead` was referenced but not defined
- **FIXED**: All deprecation warnings for `datetime.utcnow()` (preserved for now)
- **FIXED**: Proper handling of 429 rate limits in LLM client

### Dependencies
- **ADDED**: Virtual environment with `pytest`, `requests`, `typer`, `pyyaml`, `python-dotenv`

### Breaking Changes
- **NONE**: This is a baseline parity release. No breaking changes from gardener's behavior.

### Known Issues
- `datetime.utcnow()` deprecation warnings — will fix in future cleanup
- Parallel enrichment not implemented (sequential only for baseline)
- County cache file path uses default location

---

## [2026-05-20] — Audit Fix Batches 1 & 2

### Critical Fixes (Batch 1)

- **FIX #2 — CLI draft command signature mismatch** (`cli/main.py`)
  - `draft` command called `draft_touch(lead_id, prompt_id, position)` with 3 args
  - `draft_touch` only accepts `(lead_id, prompt_variant=None)` — 2 args
  - **Fix**: Updated CLI to match actual signature, added `--variant` flag
  - **Impact**: Running `absel draft` would crash with TypeError before this fix

- **FIX #7 — Priority not recalculated after enrichment** (`capabilities/orchestration.py`)
  - Priority was calculated BEFORE enrichment, so readiness signals (phone, website,
    brave data, competitor displacement) were missing from the priority score
  - Brief sorted by stale priority — leads with good enrichment data ranked the same
    as leads without
  - **Fix**: Added priority recalculation step AFTER enrichment, BEFORE sorting
  - **New flow**: harvest → score → enrich → recalculate priority → sort → draft → brief
  - **Impact**: Brief ranking is now meaningful — enriched leads rank higher

- **FIX #4/#5 — Email tier classification hardcoded and incomplete**
  - Professional TLD list was 5 entries vs 18+ in gardener config
  - Free provider list was 3 entries vs 8+ in gardener config
  - Leads with `.legal`, `.health`, `.outlook.com` etc. were misclassified
  - **Fix**: Created `classify_email_domain()` in `capabilities/scoring.py` as single
    source of truth, reads from `gardener.json` config
  - **Also fixed**: `integrations/domain.py` now reads free providers from same config
  - **Impact**: Email tier scores and LLM prompt context now accurate for all TLDs

### Scale Fixes (Batch 2)

- **FIX #6 — PLLC credential false positives** (`capabilities/scoring.py`)
  - Secondary credential matching used `\b{suffix}\b` which matched "MD" inside
    "MD Development Group" — incorrectly fast-tracking non-medical businesses
  - **Fix**: Tightened regex to require credentials appear after comma/period,
    before entity type, or at end of name — NOT as standalone word in middle
  - **Impact**: "MD Development Group LLC" no longer scores 92 as medical practice

- **FIX #8 — Pipeline enriches 2x more leads than displayed** (`capabilities/orchestration.py`)
  - `top_leads = backlog[:limit * 2]` enriched/drafted 2x the requested limit
  - Wasted Brave API calls ($5/2000) and LLM calls on leads not shown in brief
  - **Fix**: Changed to `top_leads = backlog[:limit]`
  - **Impact**: API costs cut in half per pipeline run

- **FIX #16 — Historical detection uses 89-day threshold** (`capabilities/drafting.py`)
  - Used `days_since_filing > 89` which missed leads filed exactly 90 days ago
  - **Fix**: Now uses `lead.get("temporal_category") == "historical"` — the canonical
    field computed at harvest
  - **Impact**: 90-day leads correctly get historical prompt (equipment lifecycle framing)

- **FIX #15 — Brave called on nameless leads** (`capabilities/enrichment.py`)
  - `brave_enrich_lead()` called even when lead had no name — wasted API call
  - **Fix**: Added guard `if lead.get("name", "").strip()` before Brave call
  - **Impact**: No wasted Brave calls on leads that can't be searched

### Summary

| Fix | Severity | File | Impact |
|-----|----------|------|--------|
| #2 | HIGH | `cli/main.py` | Draft command no longer crashes |
| #7 | HIGH | `orchestration.py` | Brief ranking now reflects enrichment data |
| #4/#5 | MEDIUM | `scoring.py`, `drafting.py`, `domain.py` | Email tiers accurate for all TLDs |
| #6 | MEDIUM | `scoring.py` | No false-positive PLLC fast-tracking |
| #8 | MEDIUM | `orchestration.py` | API costs cut in half |
| #15 | MEDIUM | `enrichment.py` | No wasted Brave calls on nameless leads |
| #16 | MEDIUM | `drafting.py` | Historical leads get correct prompt |

### Tests
- All 49 tests pass after fixes
- No new test failures introduced

### Breaking Changes
- **CLI `draft` command**: Signature changed from `draft <lead_id> <prompt_id> <position>`
  to `draft <lead_id> [--variant <variant>]`. Old positional args no longer accepted.

---

## [2026-05-22] — Business ID Backfill + Zoho Push Enrichment + Placeholder Guard

### Business ID Backfill (`capabilities/backfill.py`, `integrations/ctsos.py`)

Top 100 leads had only 30/100 with `business_id` set — blocking principal/agent
enrichment. Built a 3-pass backfill system using the CT SoS Socrata API.

- **ADDED**: `search_filings_by_address()` and `search_filings_by_name()` in `ctsos.py`
- **ADDED**: `backfill_business_ids()` — 3-pass strategy: address match → normalized name → token fuzzy
- **ADDED**: `absel backfill-ids --top N --dry-run --enrich` CLI command
- **RESULT**: 70/70 missing backfilled (100/100 top leads now have `business_id`)

### Greeting Resolution Fix (`capabilities/enrichment.py`)

The Principals dataset returns `type=""` (empty), not `"Individual"`. The old check
`if primary.get("type") == "Individual"` never matched. Fixed by treating empty/missing
type as individual — all Principals entries are natural persons.

- **ADDED**: `_set_principal_greeting()` helper — uses `firstname`/`lastname` when available
- **RESULT**: 99/100 top leads now have `primary_principal_name` (greeting) populated (was 3/100)

### Filing History API Fix (`integrations/ctsos.py`)

The Filing History dataset field is `account`, not `business_id`. Old queries returned
400 errors for every enrichment call. Fixed in `fetch_filing_history()` and bulk variant.

### Zoho Auth Auto-Bootstrap (`shared/zoho/auth.py`)

Zoho credentials only loaded when running through `cli/main.py` entry point.
Direct imports of `integrations/zoho` failed with "credentials not configured."

- **ADDED**: `load_dotenv()` bootstrap in shared auth module — searches from CWD upward for `.env`
- **ADDED**: `python-dotenv` dependency to shared package

### Zoho Push Enrichment (`integrations/zoho.py`, `capabilities/description.py`)

Major refactor of the Zoho push path. Previous push was a single API call with draft
body written to the Description field.

#### Two-Call Split (Load-Bearing)

Push is now **two sequential API calls**:
- **Call 1**: `Email_Draft_Subject`, `Description`, `Business_Vertical`, `Vertical_Segment`,
  `Filing_Date`, `Filing_Agent`, `LLManchor`, plus all standard lead fields
- **Call 2** (after Call 1 completes): `Email_Draft_Body2` only

The split creates a gap between subject and body arrival — required for Zoho email
automation to fire reliably.

#### Description Brief (`capabilities/description.py`)

- **ADDED**: `build_description_brief(lead)` — structured 8-line plain-text brief from
  enriched lead data
- Format: header (entity type, tier, score, temporal), filing, NAICS, principal,
  agent (with formation service name), domain/phone, competitor, filing signals, window
- **Write-once**: Description set on first push, skipped on subsequent pushes
  (preserves operator manual edits). Guarded by `description_pushed` flag on lead record.

#### NAICS Vertical Mapping (`config/naics-verticals.yaml`)

- **ADDED**: `get_vertical_segment(naics)` — derives `Business_Vertical` and `Vertical_Segment`
  from NAICS code prefix. Match on longest prefix first.
- 100+ entries: Legal, Healthcare, Construction, Real Estate, Hospitality, Automotive,
  Retail, Education, Nonprofit, Government, Services
- Catch-all: `"Other"` / NAICS code. No NAICS: `"Unknown"` / `"Unknown"`

#### New Zoho Push Fields

- `Description` — structured brief (replaces old draft_body write)
- `Business_Vertical`, `Vertical_Segment` — derived from NAICS
- `Filing_Date` — date-only format (e.g., `2021-04-19`)
- `Filing_Agent` — formation service name (e.g., `DRURY, PATZ & CITRIN, LLP`)
- `LLManchor` — draft anchor text
- `Last_Name` — now derived from principal last name, not greeting

### Bracket Placeholder Defense-in-Depth

Vascular CT, PLLC was pushed with `Dr. [Last Name],` in the draft body — an LLM-generated
placeholder that went live as an actual email. Three-layer defense built:

- **Layer 1 — Prompt rules** (`drafting.py`): "NO BRACKET PLACEHOLDERS" section in base
  drafting rules tells LLM to never output `[LastName]`, `[First Name]`, `[Company]`, etc.
- **Layer 2 — Sequence auto-regen** (`sequencing.py`): `validate_no_placeholders()` added
  to validation loop alongside competitor/restricted-model checks. Retries with corrective
  prompt on detection.
- **Layer 3 — Push gate** (`integrations/zoho.py`): `_enforce_push_prerequisites()` scans
  draft body for bracket placeholders. Hard stop — push blocked with error if found.
- **ADDED**: `validate_no_placeholders()` — 15-pattern regex list covering spaces,
  abbreviations, case insensitivity, common variations

### Files Changed

| File | Change |
|---|---|
| `capabilities/backfill.py` | NEW — 3-pass business ID backfill |
| `capabilities/description.py` | NEW — Description brief + NAICS vertical lookup |
| `config/naics-verticals.yaml` | NEW — NAICS prefix mapping table |
| `capabilities/enrichment.py` | FIXED — greeting resolution with empty principal type |
| `integrations/ctsos.py` | ADDED — address/name search functions; FIXED — filing history field name |
| `integrations/zoho.py` | REFACTOR — two-call split, new push fields, placeholder guard |
| `capabilities/drafting.py` | ADDED — placeholder validation, prompt hardening |
| `capabilities/sequencing.py` | ADDED — placeholder validation in auto-regen loop |
| `cli/main.py` | ADDED — `absel backfill-ids` command |
| `state/schema.py` | ADDED — `description_pushed`, `description_pushed_at` fields |
| `shared/zoho/auth.py` (external) | ADDED — auto-bootstrap `.env` loading |

---

## [2026-05-24] — Contract Renewal Sequences

### Part 1: Zohar Renewal Queries (`zohar/queries/renewals.py`)

- **ADDED**: `get_upcoming_renewals(days_ahead=120)` — queries Deals for contracts
  expiring within N days, enriched with equipment data from Products module
- **ADDED**: `get_renewal_by_account(account_name)` — lookup for a specific account
- **ADDED**: `get_renewal_summary()` — aggregated counts by 30/60/90/120 day buckets
- Field auto-resolution: tries 4 candidate field names for contract expiration
  (Closing_Date, Contract_End_Date, Contract_Expiration, Expiration_Date)
- Equipment enrichment via Deal/Products API
- Contact email lookup via Deal/Contacts API
- Contract length derivation from Description keywords
- **ADDED**: `zohar renewals upcoming`, `zohar renewals summary`, `zohar renewals account`

### Part 2: Renewal Lead Creation (`capabilities/renewals.py`)

- **ADDED**: `UPGRADE_PATHS` dict — 20+ config-driven model mappings (e.g.,
  AltaLink C8145 → C8245, WorkCentre 6655 → AltaLink C8245)
- **ADDED**: `recommend_upgrade(current_model, meters)` — config + meter urgency tier
  (recommended/suggested/optional)
- **ADDED**: `create_renewal_lead(renewal)` — converts Zohar renewal record into
  backlog lead with lead_type="renewal". Skips duplicates (same zoho_deal_id).
- **ADDED**: `create_renewal_leads(renewals)` — bulk create with counts
- **ADDED**: `get_renewal_leads(urgent_only=False)` — filterable by urgency
- **ADDED**: `calculate_renewal_schedule()` — deadline-anchored evenly distributed
  schedule (not front-loaded like cold outreach). Last touch lands buffer_days
  before expiration.

### Part 3: Renewal Sequence Preset (`config/sequences.yaml`)

- **ADDED**: `renewal_90` preset — 4 touches over 75 days, deadline-anchored
  evenly distributed schedule
- **ADDED**: 4 renewal-specific purposes:
  - `renewal_check_in` — service-oriented check-in, reference equipment model + meters
  - `renewal_value_add` — upgrade recommendation as substance
  - `renewal_meeting` — soft meeting ask
  - `renewal_final` — graceful close (same as cold final)

### Part 4: Renewal Prompt Mode (`capabilities/drafting.py`)

- **ADDED**: `_build_renewal_prompt(ctx, lead)` — warm, professional, service-oriented
  framing. Does NOT introduce ABS, explain copiers, or plant-the-seed. References
  specific equipment model, meter counts, expiration date naturally.
- **ADDED**: `validate_renewal_draft(text, equipment_model)` — renewal-specific
  validators: rejects urgency language ("act now", "about to expire"), requires
  equipment model mention, rejects cold-outreach intro. All standard validators
  (competitor brands, em dashes, placeholders) still apply.
- **ADDED**: `RENEWAL_URGENCY_PHRASES` — 9 banned urgency patterns
- **ADDED**: `RENEWAL_COLD_INTRO_PHRASES` — 10 cold-intro patterns to reject

### Part 5: Renewal Sequence Engine (`capabilities/sequencing.py`)

- **UPDATED**: `recommend_sequence()` — detects `lead_type="renewal"` → returns
  `renewal_90` preset
- **ADDED**: `_build_renewal_sequence_prompt()` — renewal-specific sequence prompt
  builder using `_build_renewal_prompt()` as foundation
- **UPDATED**: `draft_sequence_touch()` — branches on lead_type, uses renewal
  prompt + `validate_renewal_draft()` for renewal leads
- **UPDATED**: `preview_sequence()` — uses `calculate_renewal_schedule()` for
  evenly distributed deadline-anchored schedule; uses preset's purpose_progression
  directly instead of cold-outreach `select_purposes()`
- Differentiator check skipped for renewal sequences (they don't use
  close_differentiators from services-knowledge.yaml)

### Part 6: Renewal Reply Handling (`capabilities/respond.py`)

- **UPDATED**: `_build_response_prompt()` — detects `lead_type="renewal"` and
  generates warm, service-oriented response prompt. Includes existing equipment
  model, contract expiration, and upgrade recommendation in context.

### Part 7: Renewal CLI Commands

**absel CLI** (`cli/main.py`):
- `absel renewals scan [--days N] [--dry-run]` — scan Zoho, create renewal leads
- `absel renewals list [--urgent] [--days N]` — list renewal leads from backlog
- `absel renewals sequence [--limit N] [--dry-run] [--approve]` — generate sequences
- `absel renewals due [--push]` — due touches for renewal leads only

**zohar CLI** (`zohar/cli/main.py`):
- `zohar renewals upcoming [--days N]` — show upcoming renewals
- `zohar renewals summary` — aggregated counts by window
- `zohar renewals account <name>` — lookup specific account

### Schema (`state/schema.py`)

- **ADDED**: Renewal-specific fields: `equipment_model`, `equipment_serial`,
  `equipment_mono_meter`, `equipment_color_meter`, `equipment_age_months`,
  `contract_expiration`, `days_until_expiration`, `contract_value`,
  `contract_length_months`, `recommended_upgrade`, `upgrade_reason`,
  `meter_insight`, `upgrade_urgency`, `zoho_deal_id`
- **ADDED**: `harvest_source` enum value: `"zoho_renewal"`
- **ADDED**: `temporal_category` enum values: `"renewal_30"`, `"renewal_60"`,
  `"renewal_90"`, `"renewal_120"`

### Key Design Decisions

1. **Renewal is a mode, not a separate system**: `lead_type` field drives prompt
   selection, schedule calculation, and validators. Same sequence engine, same
   LLM pipeline, same approval gate.
2. **Upgrade recommendations are config, not AI**: `UPGRADE_PATHS` dict maintained
   by operator. Product knowledge stays editable without prompts.
3. **Meter counts are the strongest evidence**: Used in every renewal touch as the
   natural substance source. Replace cold-outreach talk track.
4. **Expiration date is informative, not a threat**: Banned urgency language ensures
   renewal emails read as service-oriented, not pressure-based.
5. **Deadline-anchored scheduling**: Renewal touches are evenly distributed toward
   expiration (buffer 15 days), unlike cold outreach's front-loaded exponential
   curve.

---

## [2026-05-24] — Reply Classification + Autoresponder Protection + Clamping Fixes

### Reply Classification (`capabilities/respond.py`)

Replaced binary opt-out detection (`detect_opt_out`) with 5-category
`classify_reply()`. Autoresponders no longer pause sequences. Bounces
are separated from opt-outs. The sequence continues undisturbed unless
a human actually engaged or explicitly opted out.

- **ADDED**: `classify_reply()` — 5-category classifier with priority order:
  1. Bounce (delivery failure — flag email, pause sequence, do NOT burn)
  2. Autoresponder (OOO/vacation/auto-reply — log and continue, do NOT pause)
     - Opt-out INSIDE autoresponder → opt-out wins (compliance)
  3. Hard opt-out (burn lead, cancel sequence)
  4. Soft opt-out (pause lead + sequence, surface to operator)
  5. Engagement (pause sequence, operator drafts response with `absel respond`)
- **ADDED**: `AUTORESPONDER_PATTERNS` — 30+ patterns covering OOO variants,
  auto-reply markers, acknowledgment templates, and redirect instructions
- **ADDED**: `BOUNCE_PATTERNS` — 10+ patterns for mailer-daemon, postmaster,
  undeliverable, SMTP permanent failure codes (550, 552, 553)
- **ADDED**: `_extract_return_date()` — best-effort date extraction from
  autoresponders ("returning on June 5" → "2026-06-05"). Unparseable dates
  return None without failing. Covers: month-name + day, M/D, M/D/Y, ISO
- **MOVED**: `HARD_OPT_OUT_PHRASES` (21), `SOFT_OPT_OUT_PHRASES` (12) from
  `integrations/zoho.py` to `capabilities/respond.py` — canonical source
- **UPDATED**: `draft_response()` — still calls `pause_sequence_on_reply()`
  as safety net (already-safe double-call)

### Zoho Reply Pipeline (`integrations/zoho.py`)

- **REPLACED**: `detect_opt_out()` removed — `classify_reply()` imported from
  `capabilities.respond` is the single classifier
- **REWRITTEN**: `process_reply()` — 5-branch handler matching classifier:
  - `"continue"` — autoresponder logged, sequence continues
  - `"burn"` — hard opt-out, burn + cancel (unchanged behavior)
  - `"pause"` — soft opt-out, pause + surface (unchanged)
  - `"flag_bounce"` — `email_bounced=True` set, sequence paused, do NOT burn
  - `"engage"` — sequence paused, operator drafts response
- **UPDATED**: `get_due_touches_safe()` — autoresponders do NOT remove lead
  from due-touches list. Logging distinguishes each action.

### Sequence Engine Pause (`capabilities/sequencing.py`)

- **UPDATED**: `pause_sequence_on_reply()` now accepts optional `classification`
  parameter. If `"autoresponder"`, no-op (sequence continues). Caller
  (`process_reply`) decides when to pause.

### Schema (`state/schema.py`)

- **ADDED**: `last_autoresponder` — dict with `received_at`, `text`, `return_date`
- **ADDED**: `email_bounced`, `email_bounce_reason`, `email_bounced_at`
- **ADDED**: `reply_action` — stores classification category string

### Bug Fixes

**Bug 1 — NoneType on research_note access** (`sequencing.py:193,226`)

`lead.get("research_note", {}).get("findings")` fails with AttributeError
when `research_note` is explicitly `None` (Python's `dict.get` only returns
the default when the key is *missing*, not when the value is `None`). Hit
~60% of leads in batch runs because most leads have `research_note: null` in
the backlog JSON.

- **FIXED**: Changed to `(lead.get("research_note") or {}).get("findings")`
  in both `_max_touches_for_lead()` and `recommend_sequence()`

**Bug 2 — Envelope constraint rejection from competitor bump** (`sequencing.py:258`)

`recommend_sequence()` selected `extended_120` (8 touches) after competitor
displacement bumped `deep_90` (6) up the tier list, but `_max_touches_for_lead()`
returned 7 (6 base + 1 competitor). `preview_sequence()` then crashed with
"insufficient data for N touches." Hit 6 leads in batch run.

- **FIXED**: Added clamping loop in `recommend_sequence()`. After selecting
  preset, loads sequences config and walks down tier order until
  `preset_touches <= _max_touches_for_lead(lead)`. `extended_120` (8 touches)
  currently unreachable with max ceiling of 7 — lands on `deep_90` (6 touches).

### Acceptance Criteria

All 12 classification criteria pass:
- Autoresponder → sequence continues (was: pauses)
- Autoresponder + opt-out → opt-out wins (compliance)
- Bounce → email flagged, do NOT burn
- Hard opt-out → burned (unchanged)
- Soft opt-out → paused (unchanged)
- Engagement → paused (unchanged)
- Short replies ("Ok", "Thanks") → engagement (better safe)
- Return date extraction: 7/7 patterns pass (month+day, M/D, M/D/Y, ISO)

### Files Changed

| File | Change |
|---|---|
| `capabilities/respond.py` | REWRITTEN — classifier added, patterns moved here |
| `integrations/zoho.py` | REFACTOR — uses `classify_reply`, 5-branch process_reply |
| `capabilities/sequencing.py` | FIXED — NoneType guard, clamping loop in recommend_sequence |
| `state/schema.py` | ADDED — autoresponder/bounce/reply_action fields |

---

## [2026-05-23] — MILESTONE 1

First complete end-to-end pipeline: harvest → score → enrich → research → draft →
approve → push with rich metadata. The system can now generate research-grounded,
personally-addressed, compliance-validated emails and push them to Zoho with full
operational context.

### Research Note Capability (`capabilities/research.py`)

Generates per-lead research notes from website content and Google Business Profile
data. The note surfaces 3-5 specific, concrete details about the business that the
LLM drafter can reference in personalized outreach. Research is gated — only runs on
leads with a live website or substantive Brave data.

- **ADDED**: `research_lead(lead_id, force=False)` — full pipeline: gate check →
  website content fetch → Brave Google Business extraction → LLM structured
  extraction → storage → Zoho push
- **ADDED**: `research_batch(lead_ids, force=False)` — sequential batch with
  per-lead error handling
- **ADDED**: `fetch_website_content()` in `integrations/domain.py` — fetches
  homepage + about pages, strips HTML via stdlib `html.parser`, 12k chars per
  page, 24k total, 5s/page timeout
- **ADDED**: `_fetch_google_business()` — one Brave API call extracts rating,
  review count, snippets from Google Business card in search results
- **ADDED**: `_extract_research_note()` — LLM extraction with JSON parse retry,
  falls back to empty findings on double failure (non-fatal — lead can still
  draft from NAICS context)
- **ADDED**: `absel research <id>` — single lead
- **ADDED**: `absel research --batch --limit N --category C --force --dry-run`
- **RESULT**: Research on Sas Strategic extracted 3 findings from website +
  Brave, 5 pages fetched, pushed to Zoho `LLMResearch_Note` field

### Website Content Fetcher (`integrations/domain.py`)

- **ADDED**: `_ContentStripper(HTMLParser)` — drops script, style, nav, footer,
  preserves block structure from p, li, h1-h3
- **ADDED**: `_strip_html(html)` — strips tags, collapses whitespace
- **ADDED**: `fetch_website_content(url, max_chars_per_page, max_total_chars,
  timeout)` — tries `/`, `/about`, `/about-us`, `/our-team`, `/services`
- Tolerates SSL cert issues (common on small business sites)
- Returns `{pages_fetched, content, fetch_errors}`

### LLM Resilience (`integrations/llm.py`) — Complete Rewrite

The old single-model retry loop was incapable of recovering from Featherless
capacity issues, timeouts, or gateway errors. Replaced with a 4-model fallback
chain across 3 full passes (12 total attempts before giving up).

- **REPLACED**: `call_llm()` — now iterates through model chain on failure
- **ADDED**: `LLMError` exception — `error_type`, `detail`, `status_code`,
  `model`, `attempts` list
- **ADDED**: `_ModelFailure` internal exception — single attempt error
- **ADDED**: `_try_model(model, prompt, ...)` — per-model attempt with retry
  logic for 429, 502/503/504, timeout, empty response
- **ADDED**: `health_check()` — 5-token ping to all models, returns status + latency
- **ADDED**: `absel health` CLI command
- **Model chain**: DeepSeek V3.1 → Qwen3.6-27B → Gemma-4-31B-it → Mistral-7B
- **Chain config**: `FEATHERLESS_MODELS_FALLBACK` env var for custom ordering
- **Capacity handling**: "at capacity" in 502/503 body → skip model immediately
  (no retry on same model when capacity is the issue)
- **Chain retry**: 30s wait between full passes, up to 3 passes
- **Logging**: Every attempt logs to stderr with timestamp and model name
- **Caller handling**: Drafters set `draft_status: "failed"` + `draft_error`;
  Research returns empty-findings note (non-fatal); Sequences continue to next
  touch
- **Metadata**: `model_used`, `model_position`, `chain_pass` recorded on every
  successful response

### Zoho Push Enrichment — v2 (`integrations/zoho.py`, `capabilities/description.py`)

The Zoho push now captures operational context alongside the email draft,
making the lead record self-documenting in Zoho.

#### Two-Call Split (Load-Bearing)

Push is two sequential API calls — do NOT consolidate.
- **Call 1**: All metadata fields + `Email_Draft_Subject`
- **Call 2** (after Call 1 completes): `Email_Draft_Body2` only
- Creates a split-second gap between subject arrival and body arrival that
  Zoho email automation requires to fire reliably

#### Description Brief (`capabilities/description.py`)

- **ADDED**: `build_description_brief(lead)` — 8-line structured plain-text
  brief with entity type, tier, score, temporal category, filing date, county,
  NAICS with label, principal, agent (with formation service name), domain/phone
  with source, competitor status, filing signals, outreach window
- Write-once: set on first push, skipped on subsequent pushes (operator edits
  preserved). Guarded by `description_pushed` flag.
- Description was previously filled with raw `draft_body` — now holds the
  structured brief instead

#### NAICS Vertical Mapping

- **ADDED**: `get_vertical_segment(naics_raw)` — derives `Business_Vertical` and
  `Vertical_Segment` from NAICS code prefix, stored in
  `config/naics-verticals.yaml` (100+ entries across Legal, Healthcare,
  Construction, Real Estate, Hospitality, Automotive, Retail, Education,
  Nonprofit, Government, Services)
- Match on longest prefix first; catch-all: `"Other"` / NAICS code; no NAICS:
  `"Unknown"` / `"Unknown"`

#### New Zoho Push Fields

| Field | Source | Format |
|---|---|---|
| `LLMResearch_Note` | Research note | Plain text |
| `Business_Vertical` | NAICS derivation | Text |
| `Vertical_Segment` | NAICS derivation | Text |
| `Filing_Date` | `filing_date` | `YYYY-MM-DD` |
| `Filing_Agent` | `agent_name` (if business type) | Text |
| `LLManchor` | `draft_anchor` | Text |
| `Business_Type` | `entity_type` | Text |
| `Lead_Age_Years` | Computed from filing_date | Integer string |

#### Last_Name Fix

- **ADDED**: `_extract_last_name(lead)` — derives proper last name from
  principal names (`"ELIZABETH WING"` → `"Wing"`) or agent name if individual,
  falling back to company name. Replaces old logic that used greeting (first
  name) as last name.

### Bracket Placeholder Defense-in-Depth (`capabilities/drafting.py`)

After a live email was sent with `Dr. [Last Name],` as the salutation, three
layers of defense were built — prompt, auto-regen, and push gate.

- **ADDED**: `validate_no_placeholders(text)` — 15-pattern regex covering
  `[LastName]`, `[First Name]`, `[Company]`, `[Name]`, `[Title]`, and
  spacer-padded variants (`[ Name ]`)
- **Prompt hardening**: "NO BRACKET PLACEHOLDERS" section in base drafting
  rules — LLM told to never output bracket placeholders
- **Sequence auto-regen**: `validate_no_placeholders()` added to validation
  loop alongside competitor/restricted-model/fabricated-conversation checks
- **Push gate**: `_enforce_push_prerequisites()` scans draft body for
  placeholders — hard stop with error if found

### Greeting & Opener Rules Update (`capabilities/drafting.py`)

Greeting instruction replaced with comprehensive rules covering time-of-day
awareness, formality calibration (Dr./Esq. vs casual), milestone acknowledgment
(without congratulations), and research-informed openers.

- **ADDED**: `send_time` / `send_period` to `_build_context_object()` —
  injected into prompt as `"This email is being sent in the afternoon (2:30 PM)"`
- **REPLACED**: Old greeting rules with: TIME OF DAY, FORMALITY (professional
  title → "Dr. Riccio," — never "Hey Dr."), MILESTONE ACKNOWLEDGMENT
  ("Five years in Fairfield -- that's no small thing." — never
  "Congratulations"), RESEARCH-INFORMED OPENER, SIMPLE OPENERS
- **ADDED**: Post-generation validations for "congratulations"/"congrats",
  "Hey Dr.", "I hope this finds you well", "I wanted to reach out",
  "I'm reaching out", "Just wanted to"
- **Research integration**: `has_research`, `research_findings`,
  `research_suggested_angle` passed to drafter; "Use ONE or TWO" instruction
  ensures findings aren't listed as a dossier

### Double-Dash Spacing Fix

- Em dashes (`—`) replaced with spaced double dashes (` -- `) via regex:
  `(?<! )--(?! )` — ensures readability without double-spacing or breaking
  already-correct dashes. Applied in both `_parse_llm_response` (single
  touch) and `_parse_sequence_response` (sequence touches).

### Sequence Prompt Inheritance (`capabilities/sequencing.py`)

- Confirmed: `_build_sequence_prompt()` calls `_build_base_drafting_rules(ctx)`
  as its first step — all base rules (greeting, close bans, competitor bans,
  historical framing, bracket placeholder ban) automatically apply to sequence
  touches
- **ADDED**: LLMError handling in `draft_sequence_touch()`
- **ADDED**: `model_used` / `model_position` metadata on sequence touches
- **FIXED**: Double-dash regex applied to sequence response parsing

### Business ID Backfill (`capabilities/backfill.py`, `integrations/ctsos.py`)

- **ADDED**: `search_filings_by_address()` / `search_filings_by_name()` in
  `ctsos.py` — queries CT SoS Socrata API by billingstreet/city or name
- **ADDED**: `backfill_business_ids(lead_ids, dry_run)` — 3-pass strategy:
  address match → normalized name → token fuzzy
- **ADDED**: `absel backfill-ids --top 100 --dry-run --enrich`
- **RESULT**: 70/70 top-100 leads backfilled (was 30/100 business_id coverage)

### Greeting Resolution Fix (`capabilities/enrichment.py`)

- Principals dataset returns `type=""` (empty), not `"Individual"`. Old check
  never matched. Fixed: `_set_principal_greeting()` treats empty type as
  individual. Also uses `firstname`/`lastname` from dataset when available.
- **RESULT**: 99/100 top leads now have `primary_principal_name` populated
  (was 3/100)

### Filing History API Fix (`integrations/ctsos.py`)

- Filing History dataset field is `account`, not `business_id`. Old queries
  returned 400 errors for every enrichment call. Fixed in single + bulk fetch.

### Zoho Auth Auto-Bootstrap (`shared/zoho/auth.py`)

- Credentials only loaded when running through `cli/main.py` entry point.
  Direct imports of `integrations/zoho` failed. Added `load_dotenv()` bootstrap
  that searches from CWD upward for `.env`.

### CLI Commands Added

| Command | Purpose |
|---|---|
| `absel health` | Check all 4 LLM models (latency, availability) |
| `absel research <id>` | Research single lead |
| `absel research --batch --limit 20` | Batch research |
| `absel backfill-ids --top 100 --enrich` | Backfill business IDs |
| `absel push <id> --dry-run` | Preview push payloads |

### Files Changed in Milestone 1

| File | Change |
|---|---|
| `integrations/llm.py` | REWRITTEN — 4-model fallback chain, 3 passes, LLMError, health check |
| `capabilities/research.py` | IMPLEMENTED — full research pipeline (was a stub) |
| `capabilities/description.py` | NEW — Description brief builder, NAICS vertical lookup |
| `config/naics-verticals.yaml` | NEW — 100+ entry NAICS prefix mapping |
| `capabilities/backfill.py` | NEW — 3-pass business ID backfill |
| `integrations/domain.py` | ADDED — `fetch_website_content()`, `_ContentStripper` |
| `integrations/zoho.py` | REFACTOR — two-call split, 8 new push fields, last_name fix |
| `capabilities/drafting.py` | ADDED — placeholder validation, greeting rules, post-gen validations, research context, double-dash regex, LLMError handling |
| `capabilities/sequencing.py` | ADDED — placeholder validation, LLMError handling, model metadata, double-dash regex |
| `capabilities/enrichment.py` | FIXED — greeting resolution for empty principal type |
| `integrations/ctsos.py` | ADDED — address/name search; FIXED — filing history field name |
| `cli/main.py` | ADDED — `health`, `research --batch`, `backfill-ids` |
| `state/schema.py` | ADDED — `has_research`, `researched_at`, `description_pushed`, `draft_model_position`, `draft_error` |
| `shared/zoho/auth.py` | ADDED — auto-bootstrap `.env` loading |

### Data State After Milestone 1

- 23,233 leads in backlog
- 100/100 top leads have `business_id` (was 30)
- 99/100 top leads have `primary_principal_name` (was 3)
- 99/100 top leads have `principal_names`
- 99/100 top leads have `agent_name`
- 1 lead researched (Sas Strategic Enterprises PLLC)
- 192 leads eligible for research (Gate 3 pass)
- 4/4 LLM models healthy at last check
