Weekly-Partner-Pipeline-SOP.md
Language Markdown23.5 KB338 lines
---type: workflow-sopscope: 07-Automationtags: [claude, outreach, automation, workflow]created: 2026-07-22last_modified: 2026-09-21---# Weekly Partner Client Pipeline: SOPThe end-to-end weekly refresh that turns live Plinth order data into a partner clientclassification, a spreadsheet, updated stubs + XML map, enrichment, and (on approval)Bitrix deals. This note is both the human procedure and the script the scheduled taskruns. Config lives in [[partner-thresholds]]; the deterministic work lives in`weekly_partner_pipeline.py`; the scheduler definition is in [[scheduled-task]].> **Current autonomy: `through_vault`** (enabled 2026-07-23). The unattended Monday 06:00> America/Chicago run goes all the way through the vault on its own (delta pull, classify,> verify, **write and upgrade stubs, and refresh the XML census**), then reads Bitrix for> the reconcile, route and outreach-audit stages. It does **not** enrich, write to Bitrix,> or route anyone into an outreach pipeline; those stay human-gated. Change `autonomy` in> [[partner-thresholds]] to widen or narrow this.> **The pipeline is ARMED.** `min_orders_to_qualify` was the arming switch and is now set> (`1`), and `mode: ltv_bands` carries the live program bands, so the write stages fire. If> any threshold in the *active* mode is ever set back to `TBD`, stages A to D still run> read-only and everything from E onward refuses.> **The scheduler runs from a file outside this vault.** [[scheduled-task]] is a *copy*.> The prompt that actually fires lives at> `~/Documents/Claude/Scheduled/weekly-partner-client-pipeline/SKILL.md`, which no session> can read or verify, and which has silently drifted from the vault copy before. Any change> to paths or stage order here must be pasted into that file by hand, or Monday's run and> this SOP will disagree. Tell: if the Monday summary reports one spreadsheet instead of> two, or the audit Summary sheet says "INCOMPLETE RUN", the external file has drifted.> **Two spreadsheets per run (2026-08-10).** The weekly run now produces> **(1)** `Complete-Partner-Clients-List_updated-<date>.xlsx`, the Drive-swap partner> roster (Stage F2), and **(2)** `Bitrix_Pipeline_Outreach_Audit_<date>.xlsx`, one sheet> per outreach funnel showing who is in it, how to find them in Plinth, and when they were> last actually contacted (Stage F5). (2) resolves Plinth identity out of (1), so **F5 must> run after F2.** They are otherwise independent; neither overwrites the other.---## The chain at a glance```A0 Baseline seed ONE TIME: Ines's roster workbook -> order_history.json (seed-baseline)A Delta pull orders.fetch, trailing pull_window_days, DAY-sized slices,offset+dedupe until a page adds 0 new ids -> ingest ─┐B Compute lifetime = baseline + post-baseline history (local) │ deterministicC Classify apply partner-thresholds.md │ (weekly_partner_pipeline.py)D Export xlsx 05-Outputs/partner-runs/… + verify (no-shrink guardrail) ─┘│ ── autonomy=through_vault → the unattended run continues from here ──E Diff + write create missing stubs / upgrade existing (+ last_modified)F XML refresh partner-client-master-list.xml (+ <last_modified>)F2 Workbook refresh Complete-Partner-Clients-List_updated-<date>.xlsx (Drive-swap copy)F3 Deal reconcile pull all deals in cats 24/26/36 → reconcile-deals → census blocksF4 Route route → weekly candidate lists (winback import CSV + cat24/dormant)F5 Outreach audit outreach-pull-list → pull stage history for what changed →outreach-audit → Bitrix_Pipeline_Outreach_Audit_<date>.xlsxG Enrich client-enrich skill on new/changed stubsH Push partner-pipeline-push skill → Bitrix deals (human-approved)│ ── every stage that changes the vault → append one LEDGER entry ──```---## Why incremental (read this before "improving" Stage A)**Plinth `orders.fetch` silently truncates large result sets**: roughly 70 orders perresponse, no "has more" flag, and `offset` does not skip by order rows, so pages overlap.Verified 2026-07-22 on one long-standing account (Plinth customer 40115 in the sample'snumbering): one all-time call returned 70 of its 74 orders and dropped the newest ones,which understated LTV **and** can make an active client look like a Winback candidate. Clients with thelongest histories (exactly the Rhodium/Iridium segment) are the ones the truncation hits.Full story in [[DECISIONS]] (2026-07-22). The shipped sample reproduces the failure on2026-09-17: the first page for that day stops at 50 rows, and only the offset loop findsthe rest. Consequences:- **Never** ask the API for an all-time total. Lifetime value lives in the local history(`07-Automation/data/order_history.json`) = Ines's roster baseline + ingested deltas.- **Never** trust one response to be complete. Page with offset + dedupe until a pageadds zero new ids (`ingest` prints `added=N`; stop at 0).- **Never** demote a tier or route to Winback when a lifetime total *decreased*: historyonly grows; a decrease means data loss. `verify` (exit 2) catches this.## Stage A0: Baseline seed *(one time, before first run)*`seed-baseline --sheet <Ines's partner roster workbook> --tab <tab> --as-of <sheet cutoff date>`(dry-run first; see the [[Activation-Runbook]] and the activation checklist in[[partner-thresholds]]). Two passes: the **All Accounts** tab, then the **Complete PartnerClients List** tab with `--force`. Per-customer lifetime aggregates keyed by Plinthcustomer ID. The weekly delta counts only orders **after** `as_of`, so it must match thesheet's actual cutoff.## Stage A: Weekly delta pull from Plinth- Window: trailing `pull_window_days` (default 8: one day of overlap; dedupe makesoverlap free).- Slice by **single day** (`since` = `until`, `yyyymmdd`), `limit` = `page_size`(**hard cap 100**: the API rejects more).- Per slice: pull → save raw JSON page → `ingest --orders <page>` → if `added > 0` ORthe page had ≥ `suspect_row_count` rows, pull again with a higher `offset` and repeat**until `added=0`**. Offsets overlap; the dedupe on `order_ref` absorbs it.- Plinth caps at **10 requests / 10 seconds** and throttles (slows, does not error). A 6 AMunattended run does not care.- Read-only. No identity exposure beyond what Matt already owns.## Stage B: Compute lifetime metrics (local, no API)Per customer, from the history store:- `ltv`: baseline LTV + sum of `amount_field` (`grand_total`) over **counted**post-baseline orders.- `total_orders_count`: baseline count + counted post-baseline orders.- `repeat_revenue`: baseline repeat + post-baseline counted spend.- `first_order_date` / `last_order_date`: baseline vs ingested, whichever bounds.- Orders dated **on or before** `as_of` never re-count (already inside the baseline).- Undated orders (not yet finished) count as recent.- Customers seen in a delta with **no baseline row** compute from ingested orders aloneand surface with `has_baseline=False`; review these, and backfill via windowed pulls ifone looks like a long-tenured client.- **Approved merges fold here (2026-08-06).** `data/merge_registry.json` maps absorbedPlinth accounts to their surviving account. After per-account metrics are computed, eachabsorbed account is folded into its survivor (sums; date bounds), so classificationsees one combined client. Absorbed accounts never classify, never get stubs, neverenter the XML census, and `refresh-xml` removes an absorbed `pl-<id>` chunk if oneexists (self-healing). Entries are added ONLY on Matt's explicit approval, one LEDGERentry each; see [[DECISIONS]] 2026-08-06. A missing or malformed registry warns andruns unmerged; it never halts the 06:00 run.## Stage C: ClassifyApply the active `mode` from [[partner-thresholds]]:- `ltv_bands` → tier by lifetime spend band.- `multi_factor` → weighted score, then band.- `simple_gate` → single cutoff, partner client vs. not.Customers below `min_orders_to_qualify` or the lowest band are marked **Not a partner client**.A mailbox listed under `tier_overrides.emails` takes its override tier outright, before thebands and before `min_orders_to_qualify` (the per-email relationship override, 2026-08-12;scope is deliberately per-email, not per-domain).If thresholds are `TBD`, tier = `UNCLASSIFIED: thresholds not set` and the run is flaggedread-only.## Stage D: Export spreadsheet + verify- Written to `05-Outputs/partner-runs/partner-classification_<YYYY-MM-DD>.xlsx`.- One row per customer: identity, computed metrics, assigned tier, store group (forreconciliation), `has_baseline` / `post_baseline_orders`, and `stub_status`(new / exists / exists-changed).- A `Summary` sheet: counts per tier, new vs. existing, baseline `as_of`, last ingest,and whether the run was read-only or armed.- **Then run `verify`: mandatory.** It compares this run against the previous one andexits 2 if any customer's lifetime total or order count shrank. On exit 2 the runSTOPS regardless of autonomy: report the flagged customers to Matt, change nothing.- `autonomy: through_vault` → the unattended run continues past this point into E/F/F2 andthe read-only Bitrix stages F3 to F5. Only G (enrich) and H (push) wait for a human.## Stage E: Diff + write stubs *(human-approved)*- Match each classified **partner client** against `03-Clients/client-stubs/*.md` by email,then by `first_name + last_name`. All stubs live flat in `client-stubs/`.- **Missing** → create a stub via `_create_stub()` in the driver, which is the canonicalshape. `_templates/client-stub.md` was rebuilt 2026-08-12 to mirror that function exactly,so the template and the driver now agree; build any hand-made stub from the template.`## Research` starts `[TBD]`; `## Human Notes` empty; `last_modified` set to run date.- **Exists** → do **not** touch `## Human Notes` or `## Research`. Update computedfrontmatter (ltv, repeat_revenue, order counts, dates, user_group_correct) only where itchanged, and set/refresh `last_modified`. Record the client as `exists-changed`.- Follow [[CLAUDE-clients]]: the scraping-dump contract still governs. Note the retiredfields listed there (`enrichment_status`, `email_one_status`, `email_two_status`,`outreach_type`, `seniority`, `total_orders_usd`): do not reintroduce them.## Stage F: Refresh the XML map *(human-approved)*- `03-Clients/partner-client-master-list.xml` (**schema 3.0**, root`<partner_client_master_list>`; renamed from `crm_pipeline_mapping.xml`). Back up firstto `05-Outputs/partner-runs/`.- Every `<client>` carries both a `<bitrix_id>` (CRM contact id) and a `<plinth_id>`(Plinth backend user id). A blank `<bitrix_id />` means the client isn't in the CRM yet.- **Full partner coverage:** the map is a complete census of the partner base. Everyclassified partner client (Palladium and up) must have a `<client>` chunk, whether ornot they sit in a pipeline. `refresh-xml` does four things each run:1. **Stamp** `<last_modified>=TODAY` on every partner client already in the map(pipeline-mapped Bitrix chunks and previously-added map-only chunks alike).2. **Add** a chunk for every partner client missing from the map: identity from thestub, an auto-generated `<summary>`, a stub pointer, and an **empty pipeline section`<pipelines count="0" />`** to be populated when they enter a Bitrix pipeline.3. **Key** each added chunk `id="pl-<Plinth user_id>"` (stable, collision-proof vs Bitrixids). Same-person duplicate Plinth accounts (same name + email) share one chunk;colleagues on one company mailbox each get their own.4. **Remove** any chunk whose `pl-<id>` is an absorbed account in`data/merge_registry.json`: a merged client is one chunk, keyed by the survivor.- Idempotent: re-running only re-stamps; it never duplicates a chunk. `<summary>` isauto-generated; never hand-edit it. Non-partner customers are not auto-added.## Stage F2: Refresh the Drive-swap workbook`refresh-workbook --plan <plan.json>` regenerates the manual Google-Drive-swap workbook`05-Outputs/partner-runs/Complete-Partner-Clients-List_updated-<date>.xlsx` so it matchesthe run. The vault copy is **not** connected to Google Drive (Matt uploads it by hand), soeach run carries the human-maintained columns forward from the **previous** generated copy(the base; `--base` overrides, else the latest `…_updated-*.xlsx` in `05-Outputs/partner-runs/`).On the **Complete Partner Clients List** tab it refreshes only the Plinth-derived columns(User Group, the dated `Plinth LTV MM/DD/YY` column with its header renamed to the run date,Total Spent (Orders), # Orders, Repeat Revenue (Plinth), First/Last Order) matched by `Id`(Plinth user_id), and **appends** newly-qualified partner clients not already present (newrows get their computed tier in `Correct Group`). It rebuilds the **All Accounts** base toevery current customer. It **never** writes a human-maintained column on an existing row(Correct Group, Comments, Contacted, contact dates, Times/Total Pipeline, Current Stage(s),Pipeline Owner/Rep) and leaves the curated **2026 Partner Working List** tabbyte-identical. **Formatting matches the master exactly:** appended rows and the rebuiltAll Accounts tab clone the master's per-column style (currency `"$"#,##0.00`, `mm/dd/yyyy`dates, Arial), and Plinth cells on existing rows are normalized to it too, so a diff againstthe master shows only real data changes, never formatting drift. Human-column styles areleft alone. Read-only on the base; writes a new dated file (never overwrites the base).## Stage F3: Deal reconcile *(2026-08-06)*Claude pulls every deal in cats 24/26/36 (per-stage `list_records`, date-window paginationpast 50) into a deals JSON, then `reconcile-deals --deals <json>` absorbs it into thecensus: one block per (client, funnel) recording the newest deal, statusesactive/completed/**lost**, summaries + stub `outreach_pipelines` mirrored, backup first.Name-matched; unmatched/ambiguous deals are reported, never guessed. Idempotent.## Stage F4: Route *(2026-08-06)*`route` evaluates [[pipeline-routing-rules]] read-only and writes the weekly candidateartifacts to `05-Outputs/partner-runs/`: routing-candidates_<date>.json, the Winbackimport CSV (person-named; role-named to a review CSV), with cat24/dormant lists in theJSON. **Suppression is absolute:** `data/suppression_list.json` (names, emails and wholenever-contact domains) + any Not-Interested closure → excluded from everything, with theevidence unioned across identity twins. **Deal creation stays human-gated**: the run emitslists; loading them into Bitrix is done by a human (or an explicitly instructed session).## Stage F5: Outreach audit *(2026-08-10)*Produces the second weekly spreadsheet: `05-Outputs/Bitrix_Pipeline_Outreach_Audit_<date>.xlsx`,one sheet per outreach funnel: **24** Partner Tier Upgrade · **26** Winback ·**36** Quarterly Check-In · **16** Abandoned Carts. **Those four and only those four**(scope set by Matt 2026-08-10; 7 Days Delivery Up-Keep and Cold Outreach are deliberatelyout of scope. `AUDIT_FUNNELS` in the driver is the single place that defines this, andadding to it is a decision, not a tweak). Each row is a client with the details Ines needsto find them in Plinth (Plinth ID, email, phone, company, tier, LTV) plus **the date theywere last actually reached out to**. Read-only on Bitrix and on vault data.**Two passes, because the full sweep is too expensive to repeat weekly.**1. `outreach-pull-list --deals <enumeration.json>` reads the weekly deal enumeration and`data/outreach_touch_cache.json` and writes `outreach-pull-list_<date>.json` with`need_history` (deals to re-pull) and `need_contact` (deals needing a `get_record`contact lookup). A deal is served from cache only when its stage is **unchanged** sincethe cache was written AND either (a) that stage is terminal, or (b) the cached touch countis **zero**: it has never entered a send stage, so it cannot be sitting in one, and a sendwould have moved it. (b) is what keeps the big parked cohorts cheap. Deals *with* touchessitting in a send stage are always re-pulled, because a repeat send re-enters the samestage (cat 26 Follow-Up Email does exactly this). **Measured 2026-08-10** (four funnels):with the cache warm, the next run needed **a fraction of the first full run's historypulls and contact lookups**. The sample shows the same shape: 54 histories fresh and 9contact lookups on 2026-09-14, then 18 fresh, 36 from cache and 2 lookups on 2026-09-21.2. Claude pulls `get_stage_history` for the `need_history` ids and `get_record` for the`need_contact` ids, attaches them to the enumeration, then `outreach-audit --deals<that file>` computes the touches, resolves Plinth identity, writes the workbook, andupdates the cache.**What counts as contact.** Only stage entries with a real outbound email, call or LinkedInDM behind them: the per-funnel `SEND_TOKENS` sets in `partner_pipeline/stages.py`. Systemreactions (No - Reply, No Reply Nth Email (Auto), Email Defunct, Changed Companies, NotInterested, Completed) are **not** contact. One trap is encoded there and must not be"simplified": **cat 26 `FINAL_INVOICE` is called "Pre-Call Email" and IS a real send**,while an out-of-scope funnel uses that identical label for its creation stage, which sendsnothing. Never key the rule on the stage *name*. Cat 26 `UC_L2JMHY` ("Activated") is leftout on purpose: it is the intake bucket, and nothing is sent on that move.**Retired stages.** Six stage ids were deleted from their funnel config, so Bitrix returns abare token with no name and a naive audit silently drops them. They were identified on2026-08-10 by matching each stage-entry timestamp against the deal's outbound email timeline(see [[DECISIONS]]): cat 24 `PREPARATION` / `UC_CEAGAV` / `UC_V6JARB` are sends,`UC_N4JCMW` is a bulk move that is not; cat 26 `UC_PMLDJW` / `UC_GDHDJR` are the Feb 2026feedback-survey send + reminder. `RETIRED_STAGE_NAMES` keeps them named. **Do not deletethem**: without them clients reached only through those stages read as never contacted.**Touch count is best-effort, the date is not.** Same-minute workflow re-entries into onestage collapse to a single send (`SAME_SEND_GAP_MIN`, 2 minutes); a bulk re-stage days apartstill counts. Read "Last reached out" as reliable and "Touches" as indicative. A last contact olderthan `AUDIT_STALE_DAYS` (90) is shaded red on the sheet and counted on the Summary.**Enumerating the big parked stages: solved 2026-08-11.** A whole-day `created_on` filtertruncates at 200 rows, which used to leave `C26:UC_JUUNES` ("Client") and `C16:NEW`("New Abandoned Cart"), both larger than that, partially listed (measured 2026-08-10).Sub-day time bounds **do** work; they are not ignored. The bounds resolve against Bitrix'sown server offset, so pass `tz=Europe/Moscow` alongside the CDT wall-clock window and walknarrow windows; a 15-minute window returns a complete, untruncated set (the connectorreports "showing N of N"). The identical call **without** `tz` returns "No matching deals". **Always verifycoverage** (the union of your windows must equal the stage's reported total) rather thantrusting a single call. The recipe is verified; the mechanism inside the connector is not.**Gaps are recorded, not hidden.** The enumeration JSON still carries an `enumeration_gaps`block. If a whole funnel goes missing the run warns loudly and stamps "INCOMPLETE RUN" on theSummary sheet rather than shipping a clean-looking partial workbook.**Anyone the audit cannot place in Plinth** lands in `outreach-unresolved_<date>.json`, andthe next `outreach-pull-list` adds them to `need_contact` so one `get_record` closes the gappermanently: the contact email persists in the cache from then on.## Stage G: Enrich *(human-approved)*- Run the **`client-enrich`** skill on the new + changed stubs. It appends only under`## Research`, sources every claim, and logs its own LEDGER entry. Needs Apollo + Apifyconnected.## Stage H: Push to Bitrix *(human-approved, always)*- Run the **`partner-pipeline-push`** skill on the approved cohort. One deal per client,deal name = `first_name + " " + last_name`, tagged by User Group, enrichment profileposted as a deal comment. Needs Bitrix connected. **Never unattended.**---## Guardrails (non-negotiable)1. **Shared vault → LEDGER.** Any stage that creates/edits/moves a vault file appends onebatched entry to [[LEDGER]], attributed by session email (`hello@meridiandisplay.example`= Matt, `accounts@meridiandisplay.example` = Hazel), timestamp from the system clock.See [[CLAUDE]].2. **Human Notes are sacred.** Automation never writes `## Human Notes`. `## Research` isonly ever touched by `client-enrich`, never by the classifier.3. **Facts are sourced.** Enrichment claims carry a source URL or a `[TBD]` marker.4. **PII stays put.** Plinth returns live customer PII. It goes into the spreadsheet andstubs, not into chat summaries or anywhere external.5. **Bitrix is human-gated** regardless of `autonomy`.6. **The outreach audit is read-only on Bitrix.** Stage F5 reads stage history and dealrecords and writes nothing back. Reading Bitrix is not the same as routing into it:the hold in [[pipeline-routing-rules]] is on *creating deals and moving stages*, whichstays human-gated. Never let the audit "fix" a stage it thinks is wrong.7. **No-shrink.** Lifetime history only grows. If `verify` exits 2, the run stops and ahuman looks at it. No tier demotion, no Winback routing, no stub downgrades off a runwhose totals went backwards: that is the Plinth truncation bug's signature, not reality.## Run logThis table was never kept up. [[LEDGER]] is the reliable per-run record. The rows up to2026-08-10 record which runs happened on the full book; their figures are not carried intothis repo. The last two rows are theshipped sample's replay (`07-Automation/sample/run_sample.py`), with the real numbers from`05-Outputs/partner-runs/run-log_2026-09-14.txt` and `run-log_2026-09-21.txt`.| Run date | Mode | Armed? | Partner Clients | New stubs | Changed | Pushed | Notes ||----------|------|--------|----------|-----------|---------|--------|-------|| 2026-07-23 | ltv_bands | yes | n/a | n/a | n/a | no | Partner backfill: the run that grew the vault from the founding cohorts to the whole partner base || 2026-07-27 | ltv_bands | yes | [TBD] | [TBD] | [TBD] | no | Artifacts existed; figures not recorded || 2026-08-03 | ltv_bands | yes | n/a | n/a | n/a | no | verify OK || 2026-08-10 | ltv_bands | yes | [TBD] | [TBD] | [TBD] | no | First run producing both spreadsheets (F2 + F5 audit) || 2026-09-14 | ltv_bands | yes | 128 | 128 | 0 | no | Sample, first run after the 2026-09-07 seed (381 accounts). 29 orders added; verify OK; tiers Palladium 85 · Platinum 28 · Iridium 8 · Rhodium 7; 1 FLAGGED messy name; XML 128 chunks added; reconcile 45 new blocks; route winback 50 (1 role-named held) / cat24 29 / dormant 4 / excluded 45; audit 54 deals, 26 touched, 3 unresolved, 54 histories fresh || 2026-09-21 | ltv_bands | yes | 130 | 2 | 128 upgraded | no | Sample. 93 orders added; 2026-09-17 truncated at 50 and paged to convergence (+50, +19, +0); 1 new customer (has_baseline=False); verify OK; tiers Palladium 86 · Platinum 29 · Iridium 8 · Rhodium 7; XML 128 stamped + 2 added = 130; reconcile 2 updated, 43 current; route winback 50 / cat24 31 / dormant 4 / excluded 45; audit 54 deals, 28 touched, 18 histories fresh, 36 from cache |