Skip to main content

Data flows

How data moves through Complied: from the outside world into the system, between its modules, and back out. The page starts with the whole system and narrows to each flow, then covers the trust boundaries and the five end-to-end workflows. It is written for developers; for the same workflows from an operator's chair see running a project.

Notation: rounded boxes are external parties, rectangles are processes, cylinders are stores. Arrows carry data, not control.

Context​

Complied does not e-file with HPD. assemble-filing-package makes sure every decided HPD document exists and records a filing_packages row with join rows; a person files the paperwork. Lab results are typed in by staff against the sample rows.

Level 1: the system​

Process 3 is the Decide, Work, File and Money workspace. It is not a gated phase machine: projects.phase is a label the header's "…" menu writes, and a trigger records the history.

Level 2: ingest​

The ingest/ CLI, run by a systemd timer on the ingestion box, drives one domain loop, runFeedCitywide (domain/src/ingestion/). Eight feeds run nightly, each idempotent on (agency, source_id). The full pipeline is in ingestion.

The feeds are HPD violations, HPD complaints, HPD litigation, DOB violations, ECB violations, OATH hearings, DOF liens and NYC 311. After the feeds, the CLI refreshes the building rollups and stamps map_data_version, which tells the map exporter to publish a new snapshot. Citywide identity loads (Layer 1), historical backfill and the weekly records sync run from the same CLI against a direct Postgres connection. Obligations are not an ingest job: their rules run on read.

Level 2: project workspace​

origin is immutable. A project is one apartment or the building. Creating from violations calls create_projects_from_violations, which makes one project per project_scope and writes their project_event_links in one transaction. Creating from an obligation writes (or links) the obligations ledger row with the new project_id. An obligation or occupant-request project starts with no orders; Link HPD orders adds orders of its own scope later. The phase label sits beside this pipeline, not in it.

Level 2: field capture and documents​

Every generator renders a PDF, writes it to the documents bucket and inserts one documents row per document code through insertDocument() in _shared/documents.ts. The XRF run writes an INSP-RPT and a separate XRF-AFF; the paint-chip run writes PC-RPT, PC-AFF and PC-LAB. generate-abatement-report writes ABATE-RPT, from the abatement job's Generate tab. HTML-to-PDF rendering goes to the render service (pdf-reports). File layout and access are in documents.

Level 2: money​

A proposal is priced from the derived services and the rate cards, edited on the Proposal card, saved to proposals, then rendered to a PDF as a separate step. An invoice is priced from what was captured (non-calibration XRF readings, samples, abatement components) and the same price lists, not from the proposal. A service with no matching rate card contributes no line, so an invoice can total $0 and still return success. invoices_stamp_billed_client fixes the billed client when the invoice leaves draft; the portal sees only non-draft invoices billed to its client.

Trust boundaries​

  • The browser holds only the anon key and the user's JWT. Everything it reads or writes is filtered by RLS.
  • An edge function that runs as the service role authorizes its caller first: a user JWT that passes has_permission(key) (or is a platform operator), or the service-role bearer. Money functions also require the caller to be in the owner tenant.
  • Functions that take no ordinary user JWT set verify_jwt = false in supabase/config.toml and check their own credential: process-email-outbox and process-notification-digest (EMAIL_OUTBOX_CRON_SECRET or platform-operator JWT), handle-email-suppression (Resend's Svix signature; it fails closed when its secret is unset) and handle-email-unsubscribe (a one-time token). request-password-reset is unauthenticated by necessity, since its caller cannot sign in.
  • setProjectPhase is not an edge function. It is a client update under RLS, and the projects_check_update_permissions trigger requires advance_projects.
  • The map service never holds tenant data. It answers from a snapshot of public rows and fetches the caller's own tenant overlay from Postgres with the caller's JWT (map).

Workflow 1: staff project, violation to commercially done​

The phase dropdown can be set at any point and is not a step in this flow; setting closed does not wait on HPD or on payment and does not make the project read-only. A collaborator tenant with field_execution can sit in the inspector lane for that project only and never enters the money lane. project-walkthrough traces this flow table by table.

Workflow 2: agency ingest​

There is no UI actor: the job runs nightly from a systemd timer on the ingestion box (08:10 UTC), with the record and enforcement feeds pulled in full weekly. Quarantined rows can be resolved with resolve_unlinked_event; there is no dedicated page for it. /map and /violations read this data and never sync it. Citywide /violations reads public_events a page at a time and never asks for an exact count. My Buildings and the portal read portfolio_events, which scopes to the caller's tracked buildings in the database. A page load is the list plus one count_portfolio_events call for every figure beside a filter (agency rail, tabs, summary row, deadline chips). Each figure is the list's own compiled filters with one axis swapped, so it counts what picking it would list. Figures per group (the grouped tree's headers, the breakdown charts, the Buildings and Clients lists' per-row counts) come from one group_portfolio_events call each, and a group's own rows are fetched a page at a time when it is opened. The CSV export reads the filtered portfolio page by page rather than in one request.

Workflow 3: client portal​

The portal is read-only and provisioned by staff. A portal user sees a building through tenant_buildings.client_id, a project through its building, a document only when it was shared with that client, and an invoice only when it is not a draft and is billed to that client. It cannot see inspections, rate cards, licenses or vendor payments (proven in the db-tests leakage suite). Portal sign-in uses Supabase Auth. The emailed password reset (request-password-reset) looks up profiles only, so it serves staff logins.

Workflow 4: notifications and email​

Every email enters through enqueue_email(), which only the service role may call. The worker reads queued rows that are due, re-checks suppression, renders the template, takes the sender identity from tenant_branding when set, sends through Resend and logs the attempt; failures retry with backoff up to five attempts. The email-outbox-drain pg_cron job calls the worker every 5 minutes at the project URL from Vault (project_url, falling back to the local stack), with the scoped drain secret from Vault email_outbox_cron_bearer (must match the edge secret EMAIL_OUTBOX_CRON_SECRET — not the service-role key); invite-staff, invite-client-user and request-password-reset also kick the drain themselves with that same secret. A bounce or complaint webhook, or an unsubscribe link (handle-email-unsubscribe), writes email_suppressions, which is global by address. notify also writes a second notification_log row with channel = 'sms' and status = 'queued' when SMS is enabled; no SMS provider is called. The digest function rolls up the last 24 hours of documents, phase changes and invoices for each recipient whose digest time matches.

Workflow 5: cross-tenant collaboration​

An owner with manage_collaborators invites another tenant onto one project with a role (field_execution, abatement, clearance_sampling or lab_coordination). The project row is visible to any active grant. Field-table writes are role-scoped through has_active_collaboration_grant_for_role(): field_execution reaches inspections, XRF, floor plans and inspection_events; lab_coordination also reaches samples and chain of custody; abatement also reaches abatement_components. A collaborator additionally needs its own permission key (for example perform_field_work) on its own side. Field-report documents are readable by a collaborator whose role is listed on the document type. The 28 RCNY abatement versus clearance independence rule produces a warning only, at invite time and at visit time. Details: permissions.