Skip to content

Modum production search implementation plan

This plan does not authorize deployment or production changes. Implementation steps use checkboxes for tracking.

Goal: Deliver reliable semantic and keyword search within Modum's existing website, with complete coverage of the approved public corpus, controlled synchronization, an operational owner and a tested rollback.

Architecture: SiteVision remains the CMS. A signed RESTApp exports approved, anonymously readable published content and sends change events to a BCW-operated service. That service extracts documents, maintains an independent search index and serves results through a signed SiteVision WebApp on /sok.

Tech stack: Existing Next.js/TypeScript service, Node.js 22.13.0 or later, SQLite, Gemini embeddings, optional Mistral OCR, SiteVision RESTApp 2 and WebApp 2. The current application uses Next.js 16.3.7. Model providers remain subject to Modum's production decision.

Spec: Semantic search for Modum kommune. Implementation baseline: docs/implementation-progress.md and docs/sitevision-setup.md. This plan incorporates the authenticated inspection on 29 September 2026 and supersedes those documents' outdated access assumptions.

Outcome and boundaries

The first release returns source links and excerpts, not generated answers. It covers approved SiteVision pages, articles, PDF, DOCX and DOC files. It preserves /sok?query=..., the homepage and header search entry points, suggestions, filters, pagination and a working original-search fallback.

External systems such as public records, maps, job listings, fault reporting and waste collection do not become searchable merely because SiteVision links to them. Initially, use approved service links where appropriate. Indexing their underlying records requires a separate source agreement, connector and acceptance exercise.

The work ends when production is operating under an agreed support arrangement, Modum has accepted coverage and relevance, and rollback and recovery have been demonstrated. A running local prototype or an uploaded ZIP is not completion.

Global constraints

Verified baseline and evidence limits

AreaEvidenceConsequence
AccountSebastian Mandal, Bluecolorworks, authenticated through the Bluecolorworks Chrome profileInteractive access exists; unattended credentials have not been established
Content and templatesExisting page and template editors expose editing and publishing controlsWe can prepare the integration without requesting general editorial access again
Files/imagesBoth archives expose upload and creation controlsThe relevant archive management UI is available
Add-onsImport, custom RESTApp management, properties and restrictions panels openInstallation and configuration controls are available; no upload was attempted
Model RESTEnabled; access restricted to Automatisering and BluecolorworksGroup access is configured; an actual authenticated Model REST read and the tenant API contract remain to be tested
Service usersManagement panel opens and offers creationRelevant administration control exists; no new service account is currently required by the anonymous exporter design
General permissionsUsers, roles and page-permissions management panels are absentDo not claim unrestricted administration or attempt to work around these limits
SigningExisting unsigned custom app reports that unsigned apps/modules are prohibitedBoth BCW packages require valid signing and applicable trust
Certificate trustWebsite's additional trusted developer certificate list is emptyBCW trust has not been established; this does not rule out server-wide trust for other publishers
EnvironmentCloud panel identifies se48 ProductionNo staging environment has been verified
HostingLocal service and worker onlyProduction hosting, DNS, backups and supervision are still needed

This was a read-only inspection. Available controls establish UI access, not successful writes to every page or successful execution of our packages. A production signing certificate belongs to a developer team, not to an ordinary CMS login. SiteVision documents team membership, signing and website trust in its certificate workflow.

Already built locally, according to the implementation record:

Outside intervention and exactly when it is needed

Start the requests in rows A–D together once implementation is authorized. Local engineering can proceed while responses arrive. Do not send requests to anyone merely because this plan names them.

GateOwner outside the implementation agentRequired input or actionBlocksWork that continues while waiting
A. SigningBCW's SiteVision developer-team administrator, or Modum's existing SiteVision partner; SiteVision if no team existsConfirm developer-team membership and a permitted signing workflow; provide the public signing certificate and a way to sign both packages without sharing private credentials in chatInstalling/running our apps on production; staging too unless it has a suitable developer licenceConnector tests, UI completion, hosting preparation, operational work
B. Test environmentModum's website owner or SiteVision hosting partnerSupply an isolated staging site, account access and approved public fixtures; confirm its API/build/extension supportTests that publish, unpublish, restrict, move, trash and replace contentMocked tests and external-service staging
C. InfrastructureSebastian/BCW infrastructure ownerSelect hosting account, region, budget, domain and operator; grant deployment, secret-store and DNS accessManaged HTTPS deployment, backups, monitoring and live event deliveryContainer hardening and deployment/runbook preparation
D. Content and processingModum content owner and the person responsible for its supplier/data-processing decisionsApprove source roots, document categories, exclusions, model/OCR providers and regions, query-log retention, freshness and availability expectationsProduction indexing scope and production provider usePublic fixture work and provisional relevance tuning within existing authorization
E. Missing CMS rightsModum/SiteVision administrator, only if an actual operation is deniedPerform the specific blocked action or grant the smallest necessary permissionOnly the denied operationEvery independent task; do not request global admin pre-emptively
F. Trust and API exposureSebastian plus Modum's authorized site ownerApprove the specific certificate to trust and the connector's anonymous GET surface protected by its application key; approve any new unattended account if subsequently requiredApplying certificate trust or materially expanding accessSigned-package preparation and review of exact settings
G. Relevance and acceptanceModum service/content representativesJudge 50–100 resident tasks, resolve excluded/failed content, approve accessibility and critical service resultsPublic launchPerformance, recovery and integration tests
H. ReleaseModum release owner and Sebastian/BCW operatorApprove the exact packages, /sok switch, observation window and rollback ownerProduction activation and publicationAll preparation and rehearsal
I. Additional sourcesEach external-system owner/vendorAPI documentation, approved data scope, credentials and usage termsSearching records held outside SiteVisionCore SiteVision release; external connectors are separate scope

No additional API key is required just to finish local engineering. Working development Gemini and Mistral credentials already exist in ignored local configuration. Production needs approved billing ownership and dedicated quotas/credentials, or explicit approval to use the current ones. Previously tested Azure embedding credentials failed authentication; obtain replacements only if Azure is selected. A normal SiteVision login does not supply the signing credential or the connector key.

If staging cannot be supplied, stop before mutation-based integration tests. An agreed production pilot restricted to designated test content is a separate decision, not an automatic substitute. If signing is delayed, the existing public-crawl prototype can remain a demonstration; it must not be relabeled as the completed CMS integration. Direct Model REST polling is a fallback investigation, not a reason to abandon the built connector without evidence.

Delivery sequence and gates

  1. Establish a reproducible baseline and request the external inputs.
  2. Finalize the source contract and production targets with Modum.
  3. Harden the connector, synchronization, storage, API and SiteVision UI locally.
  4. Provision external staging and obtain signed packages and CMS staging access.
  5. Validate extraction and change propagation on representative content.
  6. Expand to the full approved corpus; tune relevance and complete load/accessibility/recovery checks.
  7. Obtain acceptance, rehearse rollback, install production versions and switch /sok.
  8. Observe the release and transfer operational responsibility.

Signing and CMS staging are the main external critical path. Do not serialize local UI/API work behind them. Hosting can also be prepared independently. No infrastructure migration is automatic: retain single-host SQLite if it meets the approved targets; change storage only on measured need.

Review focus

  1. A parent becomes private while descendant extraction is in flight. Search, suggestions, caches and old crawl copies must all stop exposing the descendant. Owned by Tasks 3, 4 and 7.
  2. Full-corpus refresh exceeds the five-minute lease. Content must not disappear repeatedly during normal operation, and slow files must not block eligibility checks. Owned by Task 4.
  3. Routine lease renewals invalidate pagination. Next-page navigation must remain usable while current access checks still remove revoked results. Owned by Tasks 4 and 5.
  4. External waste-collection questions and unrelated queries look semantically similar to old documents. The service must show useful approved destinations or a clear no-result state. Owned by Task 7.
  5. A model provider fails, SiteVision times out, or an index is restored from backup. Public UI must provide the documented fallback and must not expose content whose eligibility has expired. Owned by Tasks 5, 6, 8 and 9.

File and interface map

All paths below are relative to the semantic-modum application repository.

AreaExisting files to use/modifyPlanned additions and responsibility
Source policylib/search/ingestion.ts, lib/search/types.ts, integrations/sitevision/src/connector.jsdocs/source-contract.md, lib/search/source-policy.ts, tests/source-policy.test.ts: exact origins, exclusions and environment boundaries
Connector/export/eventsintegrations/sitevision/src/index.js, src/connector.js, src/config/index.html, manifest.json, package.jsonsrc/policy.js, src/outbox.js, test/policy.test.js, test/outbox.test.js: independently testable eligibility and delivery decisions
Synchronizationscripts/sync-worker.ts, lib/search/sync.ts, lib/search/store.ts, tests/sync.test.ts, tests/store.test.tslib/search/refresh.ts, tests/refresh.test.ts: eligibility scheduling independent of extraction and embedding
API/retrievallib/search/api.ts, lib/search/ranking.ts, lib/search/embeddings.ts, tests/api.test.ts, tests/search.test.tslib/search/query-cache.ts, tests/query-cache.test.ts: bounded query-vector reuse, separate from access checks
CMS search UIintegrations/sitevision-search/src/index.js, src/view.js, src/config/index.html, test/view.test.jssrc/main.js and scoped assets if progressive suggestions need client code; test/requests.test.js for request/timeout/error contracts
OperationsDockerfile, compose.yaml, .env.example, app/v1/health/route.ts, app/v1/admin/sync/status/route.tsscripts/backup-search.ts, scripts/restore-search.ts, scripts/load-search.ts, tests/operations.test.ts, docs/operations.md
Evaluation/releasescripts/evaluate-search.ts, README.md, docs/sitevision-setup.md, docs/implementation-progress.mdevaluation/resident-tasks.json, docs/acceptance.md, docs/release-runbook.md, docs/access-audit.md

Preserve these existing interfaces:

GET  /v1/search?query=...&limit=10&type=...&language=...&cursor=...
GET  /v1/suggest?query=...&limit=5
POST /v1/integrations/sitevision/events      Authorization: Bearer SEARCH_SYNC_TOKEN
GET  /v1/admin/sync/status                  Authorization: Bearer SEARCH_ADMIN_TOKEN
POST /v1/admin/sync/reconcile               Authorization: Bearer SEARCH_ADMIN_TOKEN
GET  /v1/health

GET /rest-api/modum-search-connector/manifest?cursor=...
GET /rest-api/modum-search-connector/documents/:sourceId
    X-BCW-Connector-Key: SITEVISION_CONNECTOR_KEY

lib/search/types.ts remains the canonical definition of SourceDocument, IndexedDocument and SearchResponse. Preserve items, nextCursor, total.value, total.relation, mode, requestId and degraded. Add contracts explicitly in their owning task; do not let the CMS module silently diverge from the service.

Task 1: Establish the baseline and access record

Owner: BCW implementation. Dependencies: none. Outside intervention: none for this task.

Files: docs/access-audit.md, docs/implementation-progress.md, docs/sitevision-setup.md, existing test/build configuration.

pnpm test
pnpm exec tsc --noEmit
pnpm lint
pnpm build
npm --prefix integrations/sitevision ci
npm --prefix integrations/sitevision run build
npm --prefix integrations/sitevision-search ci
npm --prefix integrations/sitevision-search test
npm --prefix integrations/sitevision-search run build

Exit: A reproducible baseline and an accurate access record. Full lint may reveal pre-existing issues; record them separately and resolve any issue that affects release.

Task 2: Finalize the public source and environment contract

Owner: BCW, with Modum's content owner. Dependencies: Task 1. Outside intervention: Gates B and D before tenant tests/full ingestion.

Files: docs/source-contract.md, lib/search/source-policy.ts, lib/search/ingestion.ts, lib/search/types.ts, tests/source-policy.test.ts, connector and WebApp configuration files.

Proposed targets for agreement, not existing guarantees: five-minute maximum eligibility lease; revocation visible within 60 seconds after receiver acknowledgement and no later than lease expiry when events fail; normal publication searchable within five minutes; API p95 at most one second and rendered results p95 at most two seconds at agreed peak traffic. Initially assume ten concurrent search users for a smoke load, then replace that assumption with measured/approved traffic before sizing production. No raw query logging by default.

Exit: Modum-approved source inventory and operational targets, plus a documented extraction method for each fixture. Provider changes trigger a separately versioned reindex.

Task 3: Harden and sign the SiteVision connector

Owner: BCW. Dependencies: Task 2 contract; signing can proceed alongside local work. Outside intervention: Gates A, B and F for signing, trust and installation.

Files: integrations/sitevision/src/connector.js, src/index.js, src/policy.js, src/outbox.js, src/config/index.html, manifest.json, package.json, test/policy.test.js, test/outbox.test.js.

Exit: Connector unit tests pass, packages are signed, and the staging manifest/document endpoints return the expected bounded public representation. No production installation yet.

Task 4: Make synchronization sustainable under short leases

Owner: BCW. Dependencies: Tasks 2–3 for tenant validation; mock implementation can start earlier. Outside intervention: Modum agreement on freshness and permission-change fixtures.

Files: lib/search/refresh.ts, lib/search/sync.ts, lib/search/store.ts, lib/search/types.ts, scripts/sync-worker.ts, tests/refresh.test.ts, tests/sync.test.ts, tests/store.test.ts, connector route implementation.

Current gap: Every two minutes the worker inventories and queues all sources. File refresh downloads/extracts before hash reuse; eligibility renewal competes with embedding work. Under load this can exceed five-minute leases. Every upsert also bumps the revision used by cursors and cached documents.

Proposed additional connector contract:

type EligibilityItem = {
  sourceId: string;
  eligible: boolean;
  eligibleUntil: string;
  ancestorIds: string[];
  url?: string;
  sourceRevision?: string;
};
// GET /eligibility?ids=<URL-encoded JSON array of at most 50 source IDs>
// Same HTTPS/key/Anonymous/online checks as other export routes.
// Response: { items: EligibilityItem[] } with exactly one item per requested ID.
// Invalid or incomplete reads fail the batch; absence must not imply eligibility.

Exit: Full approved-corpus renewal is sustainable with measured headroom, mutations converge correctly and incomplete scans cannot delete the corpus. If the lease target cannot be met within SiteVision's supported load, stop for a design decision on bounded current-access checks or a revised agreed policy. Do not silently lengthen leases.

Task 5: Complete API and retrieval production behavior

Owner: BCW. Dependencies: Tasks 1–2; completes after full-corpus testing. Outside intervention: none for code; traffic expectations and production provider ownership come from Gates C/D.

Files: lib/search/api.ts, lib/search/ranking.ts, lib/search/embeddings.ts, lib/search/query-cache.ts, tests/api.test.ts, tests/search.test.ts, tests/query-cache.test.ts.

Exit: API validation, degradation, pagination, tenant boundaries and load behavior pass the agreed test suite. Model costs and query budgets are measurable.

Task 6: Finish the actual SiteVision search experience

Owner: BCW. Dependencies: API contract; CMS staging for integration. Outside intervention: Modum approves the visual result and any shared-header/template edits.

Files: integrations/sitevision-search/src/index.js, src/view.js, src/config/index.html, optional src/main.js and scoped assets, test/view.test.js, test/requests.test.js, plus the selected SiteVision page/template configuration.

Exit: The installed staging module, not merely the standalone prototype, passes functional and accessibility acceptance with a working original-search fallback.

Task 7: Reconcile coverage and obtain relevance acceptance

Owner: BCW engineering and Modum editorial representatives. Dependencies: Tasks 2–6. Outside intervention: Gate G is mandatory before launch.

Files: evaluation/resident-tasks.json, scripts/evaluate-search.ts, lib/search/ingestion.ts, lib/search/files.ts, lib/search/ranking.ts, tests/ingestion.test.ts, tests/search.test.ts, docs/acceptance.md.

Exit: An approved corpus with accountable exceptions and a judged relevance report. More hits alone is not evidence of better search.

Task 8: Deploy and operate the external service

Owner: BCW engineering/operator. Dependencies: Gate C and provider decision. Outside intervention: Hosting/DNS/account access, spending approval where needed, alert recipient and support owner.

Files: Dockerfile, compose.yaml, .dockerignore, .env.example, app/v1/health/route.ts, app/v1/admin/sync/status/route.ts, scripts/backup-search.ts, scripts/restore-search.ts, scripts/load-search.ts, tests/operations.test.ts, docs/operations.md.

Exit: Managed staging service with HTTPS, persistent storage, supervised workers, working alerts and a successful recovery drill. Record actual monthly hosting estimate and measured model usage before production commitment.

Task 9: Validate CMS changes end to end

Owner: BCW with a Modum/SiteVision administrator for permission changes the account cannot perform. Dependencies: Tasks 3–8 staging deployment. Outside intervention: Gates B/E, and designated test content.

Files: docs/acceptance.md, regression tests in tests/sync.test.ts, tests/refresh.test.ts and connector tests for any defect found.

Exit: Signed packages and hosted service pass the documented mutation/failure matrix on real SiteVision, including tests needing administrator intervention.

Task 10: Release, rollback and operational handover

Owner: BCW release operator and Modum website owner. Dependencies: All prior exit conditions. Outside intervention: Gates F, G and H.

Files: docs/release-runbook.md, docs/operations.md, docs/acceptance.md, README.md, docs/implementation-progress.md; versioned packages and deployment configuration.

Exit: Accepted production service, named operating owner, documented support expectations and proven rollback. No other Modum content or external systems are modified as a side effect.

Evidence and review required for each engineering task

For a behavior change: add the named failing case, run it to demonstrate the failure, implement the smallest change, run the focused suite, review the diff and make a scoped commit. Do not add ceremonial tests for documentation or unchanged behavior. The named tests above describe required assertions; the executor must implement them against the actual interfaces rather than copying mock SiteVision internals into production code.

After cross-cutting changes, run:

pnpm test
pnpm exec tsc --noEmit
pnpm lint
pnpm build
npm --prefix integrations/sitevision test
npm --prefix integrations/sitevision run build
npm --prefix integrations/sitevision-search test
npm --prefix integrations/sitevision-search run build

The connector test command is added in Task 3. Evaluation and load tests require approved datasets, provider quotas and the correct environment; do not run mutation or load tests against production just to complete a checklist. Keep evidence of tenant version, package hashes, test outcomes, corpus accounting and approval decisions together in docs/acceptance.md.

Effort and scheduling

These are planning estimates after reviewing the prototype, not a fixed quote or a promise of calendar delivery. They assume one engineer, a staging tenant supplied promptly and no custom module that requires substantial new extraction work.

WorkEstimated engineering effortMain uncertainty
Baseline, source contract and access preparation1–2 working daysContent-owner availability and canonical URL/repository details
Connector and sustainable synchronization3–5 working daysTenant runtime behavior, event delivery and lease renewal load
API hardening and CMS UI completion2–4 working daysPagination under updates, template integration and accessibility
Hosting, monitoring, backups and load validation2–3 working daysInfrastructure access and measured capacity
Corpus/relevance acceptance, release and handover2–4 working daysEditorial judgments and tenant defects

Total initial budget: 10–18 engineer-days, plus external waiting time and a proposed seven-day observation period. Signing/team access, staging provisioning and Modum acceptance have no verified turnaround time. Re-estimate after the 30–50-item source-contract test. A new search database, unsupported custom modules, another model provider, strict availability requirements or external-system connectors are explicit scope changes.

Immediate next implementation actions

Once execution is requested, complete Task 1 and prepare one consolidated request for signing/team access, CMS staging, hosting ownership and Modum's content/acceptance owner. Continue Tasks 2–6 locally while those inputs arrive. Do not ask for blanket administrator access or unrelated API keys.

The first unavoidable external dependency is signed-app deployment to a suitable test environment. The first unavoidable editorial dependency is approving source scope and the expected answers for resident tasks. The final external decision is approval to switch the public search after the evidence is ready.

References

Plan self-review

The original specification's public scope, source extraction, files, anonymous eligibility, events, reconciliation, hybrid retrieval, /sok integration, native comparison, signing, operations, evaluation and rollback are covered by Tasks 1–10. External source records are explicitly separate scope. The baseline numbers remain historical observations. Proposed service and acceptance targets are marked for agreement. No production write, new key, certificate trust or deployment has been performed to prepare this plan.