Investigate: Onboarding simple enough that a developer who knows nothing about logging systems can get logging working in minutes
The current onboarding recipe was written for someone wearing two hats at once — the person setting up shared Grafana Cloud infrastructure, and the person writing application code. A real customer only ever wears the second hat, knows nothing about OTLP/Grafana/access policies, and will conclude "this library is bad" and reach for console.log instead if getting it working takes more than a few trivial steps. This investigation asks what's actually required vs. what's just how the recipe happens to be written today, and whether the library itself can remove some of that requirement, not just hide it in better docs.
IMPLEMENTATION RULES: Before implementing this plan, read and follow:
- WORKFLOW.md - The implementation process
- PLANS.md - Plan structure and best practices
Status: Completed 2026-07-14 — folded into INVESTIGATE-docs-site-structure.md
Every option here shipped except Option E1 (the public dashboard verification link), which moved to the newer investigation above since it's the same underlying "developer-facing docs/verification experience" question. See that doc's own [Q5] and Next Steps for E1's current status. Kept here in full as the historical record of the research and decisions — nothing below is stale, it's just no longer the doc to check for current status.
Goal: A developer who has never heard of OTLP, Grafana, or access policies can go from "I want to log from my code" to "it's working, confirmed by an actual read-back, not just a hopeful console message" using only: one secret they were handed, three function calls, and one self-test command.
Last Updated: 2026-07-14 — maintainer answered all 6 remaining open questions: ollacrm developers confirmed to have no Grafana Cloud login today (strengthens E1's case — it needs none); E1 confirmed over a bespoke live-status page; Option D deferred until a second server-shaped consumer exists; Option B ships TypeScript-first, Python later; the connection-string format is JSON→base64, generated by a small tools/ script, and stays strictly one-per-system (no shared-string + separate service_name argument). Nothing left blocking a PLAN.
Earlier (2026-07-13): Azure/GCP reference research added; Grafana Cloud LBAC finding resolves [Q6]'s credential-sharing risk; a "one combined read+write Access Policy" simplification was tried, then reverted after Grafana Cloud's own portal UI warned against combining read+write scopes — settled back on two Access Policies per system, per the vendor's own guidance; Option A accepted as a 3-way developer/operator/contributor split; E3's design work spun off to INVESTIGATE-selftest-cli.md
The problem, concretely
using/onboarding/index.md and using/onboarding/ollacrm/index.md currently ask a new consumer to, in order: pick a service name, navigate Grafana Cloud's portal to create an Access Policy (name/realm/scopes across a multi-select and a permissions table), navigate again to find an OTLP endpoint and Instance ID, hand-compute a base64 Basic Auth header, know which of six resulting env vars is a secret vs. a plain identifier, write a disposable validation script and read Grafana's API to confirm the token works, then finally get to the three lines of actual application code (sovdev_initialize, sovdev_log, sovdev_shutdown) — and even then, verification depends on a token (ollacrm-ingest) that is deliberately write-only and can't read anything back, so "did it work" still routes through someone else's dashboard access.
None of that is wrong for this project's own onboarding — it's real infrastructure work, done once, by someone who already understands the system. It's wrong as the first thing a customer developer sees. Confirmed directly: sovdev_initialize()'s connection info is 100% environment-variable-driven (logger.ts reads process.env.OTEL_EXPORTER_OTLP_* in a dozen places) — the function signature itself is already simple (service_name, optional version, optional peer services). The entire burden is in producing those six environment variables correctly, and that's an infrastructure task masquerading as a developer-onboarding task.
Current State (checked directly)
sovdev_initialize(service_name, service_version?, peer_services?)— three arguments, none of them connection info. Confirmed viatypescript/src/logger.ts:1308.- Required env vars, confirmed via
sovdev_validate_config()(logger.ts:949):OTEL_SERVICE_NAME,OTEL_EXPORTER_OTLP_LOGS_ENDPOINT,OTEL_EXPORTER_OTLP_METRICS_ENDPOINT,OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,OTEL_EXPORTER_OTLP_HEADERS(a hand-computed base64 Basic Auth string),OTEL_EXPORTER_OTLP_PROTOCOL. - The actual application code, once those env vars exist, is already minimal: initialize once, log freely,
sovdev_shutdown()onSIGTERM. Three concepts, not six. - Verification is currently asymmetric by design: the ingest token (
ollacrm-ingest,sovdev-logger-ingest) is write-only, matching least-privilege — but this means the developer holding it has no way to self-check via API, only via someone else's Grafana dashboard access. Not yet confirmed whether a new consumer even has a login to the shared Grafana Cloud org at all. - No existing mechanism bundles the six env vars into anything smaller. No setup CLI. No "connection string" concept anywhere in the codebase.
How established observability vendors actually solve this
Checked directly rather than assumed, since these products change:
- Sentry: a DSN (Data Source Name) is exactly Option B's idea already shipped by a major vendor — one opaque string bundling protocol, host, key, and project ID, that their SDK parses internally. Their onboarding verification is a live-polling web page ("waiting for events..." → confirmed within seconds of the first test event) — not a CLI. This only works because Sentry is true multi-tenant SaaS: the developer signs up directly, gets their own project, and the vendor's own hosted UI already has read access to it. Sovdev-logger's shared-stack model (one Grafana Cloud org, many onboarded systems) doesn't have that same one-to-one account relationship, so this exact pattern doesn't transplant cleanly — see [Q9].
- Datadog: architecturally different — a local Agent daemon sits between the app and Datadog's backend.
datadog-agent statusreports the Agent's own local health/connectivity, not a true backend read-back. Doesn't map onto sovdev-logger's direct-to-cloud OTLP model (no local agent in the middle). - Grafana Labs itself already ships official, maintained query CLIs — this matters directly for Option E, since it means the read-back half doesn't need to be built from scratch:
logcli(official Loki CLI, configurable viaLOKI_ADDR/LOKI_USERNAME/LOKI_PASSWORDagainst Grafana Cloud directly),tempo-cli(tempo-cli query api searchruns TraceQL against a remote Tempo instance over HTTP), andpromtool query(the standard Prometheus CLI, works against any Prometheus-compatible API, which Mimir — Grafana Cloud's metrics backend — is). All three could be documented/wrapped rather than reimplemented. - Grafana Cloud has a built-in "Share externally" feature — a genuinely public, no-login, read-only dashboard link (confirmed available on the free tier, not an Enterprise-only feature). A link like
<public-dashboard-url>?var-service_name=ollacrm-apiwould give a new developer almost exactly Sentry's "watch it appear live" experience, with zero credential of any kind changing hands — no read token, no CLI, nothing to share or revoke.
This changes the shape of Option E: the choice isn't "build a self-test tool or don't," it's which of three real, differently-costed approaches to use for the read-back half.
Azure and GCP — forward-looking reference, not a decision needed now
Sovdev-logger doesn't connect to either backend today (INVESTIGATE-external-backend-verification.md, Tier 4, covers whether/when to add them). Checked directly anyway, so this is on hand for that future work rather than researched twice:
- Azure Application Insights already ships Option B, for real, at Microsoft scale. A single
APPLICATIONINSIGHTS_CONNECTION_STRINGenv var bundles the ingestion endpoint, instrumentation key, Live Metrics endpoint, and Application ID into one opaque string that their SDK parses internally — this is a second major vendor (after Sentry's DSN) independently landing on "one bundled secret, not several env vars," which is exactly Option B's shape. (Connection strings in Application Insights) - Azure's read-back path is a real, official CLI, not a bespoke tool —
az monitor app-insights query --app <name> --analytics-query "traces | where timestamp > ago(30m) | ..."runs a Kusto (KQL) query against the ingested data directly from the command line. That's the same shape as Option E2 (wrap the vendor's own CLI rather than build one) — for an eventual Azure backend, E2 could point ataz monitor app-insights queryinstead oflogcli/promtool. (az monitor app-insights) - Azure also has a live-polling verification view — "Live Metrics" in the Application Insights portal, ~1 second latency, no extra ingestion cost — the closest thing to Sentry's "waiting for events" page in this comparison, but it's portal-based (requires an Azure login with access to that resource), so the same [Q1]-style access question ("does the developer actually have a login to look at this") applies to Azure exactly as it does to Grafana Cloud.
- Ingested data isn't instantly queryable — Microsoft's own docs note telemetry can take 2–5 minutes to become queryable after the first request. Directly analogous to this investigation's already-documented Grafana Cloud finding (
tools/dashboards/README.md's "verify with fresh data, not old data" — there it's metric expiration, here it's ingestion delay, but the practical lesson is the same: don't conclude "broken" from a query run too soon or against stale data). - The write/read credential split this investigation designed for Grafana ([Q6]) isn't a sovdev-logger-specific idea — Azure already has it. The connection string itself is effectively write-only (an app can create telemetry with it); querying it back via
az monitor app-insights queryrequires separate Azure AD/RBAC login, a different credential entirely. Same shape as this investigation's E-series resolution: the write credential and the read credential are different things, on purpose. - GCP's connection model is a different shape entirely — no bundled string. Cloud Logging/Monitoring/Trace use Application Default Credentials (ADC): a service-account JSON key file (
GOOGLE_APPLICATION_CREDENTIALS), workload identity, orgcloud auth login'd user credentials — a file plus a project ID, not a copy-pasteable string. GCP does support direct OTLP ingestion (thetelemetry.googleapis.comendpoint, all three OTLP transports), but auth there is still ADC-based (via the OpenTelemetry Collector's "Google Client Auth" extension), not a bearer token in a header the way Grafana Cloud's Access Policies work. (OTLP log ingestion overview, Google Client Auth extension) If GCP is ever added as a backend, Option B's "one bundled secret" framing won't map directly — GCP's own shape is closer to "one credentials file," which has different distribution/secret-management implications (a file to hand over and rotate, not a string to paste into one secret slot). - GCP's read-back path is also a real, official CLI —
gcloud logging read "FILTER" --limit N --format json(Cloud Logging query language, not LogQL/KQL) queries logs straight from the command line; Cloud Monitoring has equivalent read APIs for metrics. Same E2 shape again: for a GCP backend, wrapgcloud logging readrather than build a bespoke reader. (gcloud logging reference) - IAM permissions are already split write vs. read, same pattern as Azure and this investigation's Grafana design:
roles/cloudtrace.agent/roles/monitoring.metricWriter(write) are distinct roles fromroles/logging.viewer/roles/monitoring.viewer(read) — GCP's own IAM model already assumes the two are separate grants, not one all-purpose key. - No Sentry-style live-polling onboarding page found for GCP — Cloud Logging's "Logs Explorer" is the closest equivalent, and it's the same portal/login-gated shape as Grafana's dashboard and Azure's Live Metrics, not a public/no-login view.
Takeaway for later: three real backends, three different credential shapes (Sentry/Azure = one bundled secret; Grafana Cloud = several explicit env vars; GCP = a credentials file + project ID) — Option B's design, whenever it's built, should keep that in mind rather than assume every future backend fits the same "base64 blob" format Grafana Cloud's env vars suggest. All three, however, agree on the write ≠ read credential split ([Q6]) and all three ship (or could easily use) an official CLI for the read-back half (E2), reinforcing that E2 is a genuinely reusable pattern across backends, not a Grafana-specific convenience.
Options
Option A: Split the docs by audience — three roles, not two — no library changes
Decided this session. Three distinct audiences, not the two originally proposed here — each gets its own doc, because each is a genuinely different job, even though today one person (the sovdev-logger maintainer) happens to fill two of the three roles:
- Developer — writes application code that consumes
sovdev-logger(e.g. ollacrm's engineers). Starts from "you were handed this.envsnippet"; covers onlynpm install+ the three function calls (sovdev_initialize/sovdev_log/sovdev_shutdown) + how to confirm it's working. Never touches the Grafana Cloud portal. Lives underusing/onboarding/(developer-facing quickstart). - Operator — sets up the Grafana Cloud connection for a new onboarding project: picks the service name, creates the Access Policy + token, finds the OTLP endpoint + Instance ID, computes the env vars, validates the token actually works, then hands the finished
.envsnippet to the developer. This is everything currently inusing/onboarding/index.mdsteps 1–5. Needs Grafana Cloud portal access and understanding of Access Policies/OTLP — but not sovdev-logger source-code knowledge. Today this is always the sovdev-logger maintainer, since they administer the one shared stack — but the doc should be written for the role, not the person, since a future consumer project's own admin could plausibly do this instead. New doc, structurally separate from the developer quickstart. - Contributor — works on sovdev-logger's own codebase: adds features, fixes bugs, runs its test suite, publishes releases. Already served by the existing
contributor/*docs (testing, publishing, implementation patterns, API contract) — not new, and out of scope for this option's actual work, but worth stating explicitly so "operator" doesn't get folded into "contributor" by default: operating a new project's Grafana Cloud connection requires zero sovdev-logger source-code changes, and a contributor fixing a bug inlogger.tsnever needs to touch a real Access Policy at all. Distinct roles, distinct docs, distinct audiences.
Pros: zero code risk, ships immediately; matches this project's own established pattern of separating contributor/ (maintainer-facing) from using/ (consumer-facing) — this just adds one more real seam inside using/ itself, between the operator's infra work and the developer's app code.
Cons: doesn't reduce the actual number of moving parts — a developer who ever has to regenerate the secret (token rotation, a second environment) still hits the full complexity; doesn't fix the write-only-token self-verification gap (that's Option E's job, not this one's).
Option B: A bundled "connection string" — one secret instead of six env vars
Add an optional single environment variable (e.g. SOVDEV_LOGGER_CONNECTION_STRING) — a base64-encoded blob containing the endpoint, Instance ID, and token together — that sovdev_initialize() decodes internally (setting the underlying process.env.OTEL_EXPORTER_OTLP_* vars itself, before any of the existing env-var-reading code runs) if present. The developer gets one opaque string, pastes it into one secret, done — no base64 math, no knowing which of six vars is sensitive, closer to how Sentry/Datadog's single-DSN onboarding works.
Pros: directly removes the hardest, most error-prone step (hand-computing a Basic Auth header) and collapses "six env vars, one of which is secret" into "one secret"; implementable without touching the dozen existing process.env.OTEL_EXPORTER_OTLP_* read sites, since the connection string can be expanded into those same variables at the very start of sovdev_initialize(); the six-env-var path keeps working unchanged for this project's own tooling and anyone who prefers it.
Cons: new library surface to design, document, and keep in sync across every future language port; whoever generates the connection string (ops/maintainer) still needs a small helper to produce it — that helper doesn't exist yet either.
Option C: A setup CLI
Ship npx sovdev-logger init (or similar) — an interactive prompt that takes the pieces someone already has (endpoint, Instance ID, token) and either writes a ready .env file or prints a connection string (Option B's format). Doesn't remove needing the underlying credential in hand, but removes the manual encoding/formatting step and its associated mistakes (unquoted headers, wrong Instance ID, etc.).
Pros: low-risk, purely additive; makes whoever does the ops-side setup (Option A) faster and less error-prone too, not just the developer. Cons: doesn't address the developer-facing story at all by itself — only useful combined with A or B.
Option E: A single self-test command — write, then read back, before anything else
Maintainer's own framing, verbatim, and it's the sharpest requirement in this investigation: "a test I can do that verifies I've got the needed config and it logs data and it can read back the data. How to read and use the logging system is something I do after I know it's working, and it will differ on gcp/grafana/azure anyway." This reframes verification entirely: not "open a dashboard" (backend-specific, assumes Grafana access the developer may not have — [Q1]), but one command whose job is exactly "does my config work, end to end," with the actual exploring-your-data experience deferred to later and out of scope for this one test.
The real obstacle: the developer's own ingest token is write-only by design ([Q1] in the prior round of this onboarding work established this deliberately, for blast-radius isolation between systems). Their credential alone can never read anything back — so this test needs some read access from somewhere.
The resolution — three rounds now, settled on the vendor's own recommendation. Originally this section proposed handing every onboarded system the same shared, stack-wide read credential (accepting that anyone holding it could read every system's data). That tradeoff turned out not to be necessary. Grafana Cloud Access Policies support Label-Based Access Control (LBAC): a label selector (Prometheus syntax, e.g. {service_name="ollacrm-api"}) can be attached to a logs:read/metrics:read scope, so the resulting token can only read data matching that selector — not the whole stack. (Use label-based access control (LBAC) with access policies) A second round tried combining Write and LBAC-scoped Read onto one Access Policy per system (the Scopes table is a per-resource checkbox grid, so this is technically possible) — simpler for the developer, one secret instead of two. Reverted after checking directly in the real portal: editing this project's own ollacrm-ingest/sovdev-logger-ingest policies to add Read alongside their existing Write scopes surfaced Grafana Cloud's own UI warning verbatim: "An access policy with combined read and write permissions might pose a security risk. We recommend that you create a more restrictive policy, with either read or write permissions." That's the vendor's own guidance directly contradicting the "one combined policy" simplification — worth taking at face value rather than dismissing as generic caution, since a single leaked token under that design could both inject fake data and read every real reading back. Settled: two Access Policies per system, as originally designed before this session's brief detour — a write-only one (unchanged from today) and a separate, LBAC-scoped read-only one. See [Q6] for the one thing still unconfirmed.
With a read path decided, the actual "prove it works" step has three real sub-options, cheapest first:
- E1 — a public dashboard link, no credential at all. Use Grafana Cloud's built-in "Share externally" feature on the already-built dashboard, handed to the new developer pre-filtered to their own
service_namevia URL query param. They open it, run their code, and watch it populate — Sentry's exact UX, with zero new tooling and zero credential-sharing decision needed at all. Cheapest possible option; doesn't need a read credential of any kind, LBAC-scoped or otherwise. - E2 — document/wrap Grafana's own official CLIs. Point developers at
logcli/promtool(install instructions + the exact env vars, pre-filled from their own separate, LBAC-scoped read token — see [Q6]) instead of building custom query logic — leans on maintained, professional tooling instead of reinventing LogQL/PromQL query construction. No bespoke tool to build or keep in sync with Loki/Prometheus API changes. - E3 — a bespoke self-test command, shipped inside the npm package. Shipped. What was originally proposed: push a uniquely-marked log + metric, query it back automatically, report a clean pass/fail with diagnostics — formalizing the exact pattern used by hand, repeatedly, throughout this project's own Grafana Cloud validation work this session. The only one of the three that's genuinely scriptable/CI-friendly (E1 needs a human looking at a browser; E2 still needs someone to read
logcli's output and judge it) and the only one that can give a precise, sovdev-logger-specific diagnostic (which credential is wrong vs. which signal is missing) rather than a generic query result. Its own design document (spun off since local UIS's reachability turned out to need real design work) shipped assovdev-selftest, abinentry insovdev-logger— seeINVESTIGATE-selftest-cli.mdfor the full build record. One item from that work is still deferred: ollacrm doesn't yet have their own LBAC-scoped read credential, so the shipped tool isn't yet safe to hand to them as-is (it currently reuses the maintainer's own unscoped verify token).
Recommendation within E: E1 first (near-zero cost, ships immediately, needs no credential decision at all, and is literally how a much bigger, more polished vendor solves the identical problem). E3 is now a stronger near-term candidate than previously written — the LBAC finding removes the main risk-based objection to it (shared-credential exposure), leaving only real-but-ordinary packaging effort (a bin entry, cross-language maintenance) as the remaining cost, not an access-control tradeoff. Still: don't build it purely speculatively — build it once E1 is shipped and there's an actual scriptable/CI-friendly need (e.g. ollacrm wants this gated in their own deploy pipeline). E2 sits in between and may not be worth building at all if E1 covers the actual need.
Option D: Simplify the SIGTERM boilerplate too
Add a convenience function (e.g. sovdev_register_shutdown_handlers()) that wires up the process.on('SIGTERM', ...) / process.on('SIGINT', ...) pattern internally, so a server developer doesn't need to know this pattern exists at all — just call one more function at startup alongside sovdev_initialize().
Pros: removes the one piece of application code (not infra) that still requires understanding something sovdev-logger-specific (the flush-vs-shutdown distinction, why it matters, the exact signal-handler shape). Cons: slightly reduces control for consumers with more specific shutdown-sequencing needs (multiple things to clean up in a particular order); most frameworks already have their own graceful-shutdown hook this could integrate with instead of adding a competing one — needs a per-framework design, not a one-size answer.
Recommendation
A + B + E1 together as the actual onboarding story, shippable almost immediately; E3/C and D as later, evidence-gated additions. The docs split (A) is close to free — now decided as a three-way split (developer / operator / contributor, not the original two), since "operator" (sets up a new project's Grafana Cloud connection) and "contributor" (works on sovdev-logger's own codebase) turned out to be genuinely different jobs that shouldn't share a doc just because one person fills both today. B removes the hardest manual step (hand-computing a Basic Auth header). E1 (a public, pre-filtered dashboard link) closes the "prove it works" loop the maintainer described, at near-zero engineering cost, by copying a pattern a much bigger vendor (Sentry) already validated — and it needs no credential decision at all, since no credential changes hands. E3 (a bespoke self-test CLI, folding in Option C) is real, additive value once a scriptable/CI-friendly check is actually needed — the credential-sharing risk that previously made it feel like the riskiest option is resolved by Grafana Cloud's LBAC feature (a separate, per-system read policy scoped to that system's own service_name — see [Q6]), so the only remaining cost is ordinary build/maintenance effort and one extra policy to mint per system, not an access-control tradeoff. Still shouldn't be built speculatively ahead of an actual need, matching this project's own repeated preference for building from evidence. D is a genuine simplification but affects the application-code shape (not just onboarding) and deserves its own design pass.
If forced to rank by what matters most to the actual customer, cheapest-to-most-expensive: E1 (proof it works, ships today) → B (fewer things to configure before E1 even matters) → A (the docs framing around both) → E3/C (now lower-risk than originally assessed, thanks to LBAC, but still worth waiting for a real scriptable-verification need before building).
Open Questions — all resolved
[Q1]Does a new consumer (e.g. an ollacrm developer) have Grafana Cloud login access today? Resolved 2026-07-14: no. Confirmed directly by the maintainer. This settles the point in E1's favor decisively — "open the dashboard to verify" was never actually available to a real developer under today's access model; a public, no-login "Share externally" link is not just nicer UX, it's the only verification path that works for them at all.[Q2]Commit to Option B's connection-string format and who generates it. Resolved 2026-07-14: JSON ({endpoint, instanceId, token, protocol}or equivalent) → base64-encoded into one opaque string. Generated by a small newtools/script (matching this repo's existingtools/dashboards/tools/validationpattern) — not a documented shell one-liner, and not folded into a future setup CLI (Option C stays a separate, unbuilt option).[Q3]Should the connection string be shareable across systems (via a separateservice_nameargument), or strictly one-per-system? Resolved 2026-07-14: strictly one-per-system, always. No shared-string path. Matches the deliberate write-only, per-system credential isolation already established elsewhere in this project (the<service-name>-ingesttokens, [Q6]'s LBAC read tokens) — a connection string is a repackaging of that same credential, not a new sharing model layered on top.[Q4]Is Option D (shutdown-handler convenience function) worth designing now? Resolved 2026-07-14: no, defer. Wait until a second real server-shaped consumer exists (ollacrm is the first) so the design isn't guessed against a single use case.[Q5]Does Option B need Python parity from day one? Resolved 2026-07-14: no, TypeScript-first. Ships for ollacrm's actual stack now; Python gets the connection-string format once there's a real Python consumer to design it against, matching this project's existing pattern for other TS-only features (client_name,service_principal,acting_user).- [Q6] Resolved (2026-07-13, fully confirmed 2026-07-14) — confirmed Grafana Cloud Access Policies support Label-Based Access Control (LBAC): a label selector scopes a
logs:read/metrics:readpermission to just one system's data. (LBAC docs) Settled on two Access Policies per onboarded system, not one — a write-only policy (unchanged from today) plus a separate, LBAC-scoped read-only policy. A "combine both onto one policy" idea was tried and reverted: editing this project's own real Access Policies in the portal to add Read alongside existing Write scopes surfaced Grafana Cloud's own UI warning — "An access policy with combined read and write permissions might pose a security risk. We recommend that you create a more restrictive policy, with either read or write permissions." — the vendor's own guidance directly against it, checked live, not assumed. Both remaining sub-points now confirmed directly, not just from docs: (a) the "Add label selector" control is real and usable on this account's plan tier — it's an expandable "Label selectors (0)" section below the Scopes table (not a per-checkbox button, as first guessed), confirmed by actually creatingollacrm-verifywith a working selector (service_name=~"^ollacrm-api.*"— note the regex operator; an exact-match attempt was tried first and would have silently brokensovdev-selftest's read-back, since it writes under a-selftest-suffixed disposable name); (b) LBAC genuinely doesn't cover traces — Grafana's own UI states this directly ("Available only with read permissions for metrics and logs") — confirmed live, not just from the docs page.traces:readstays stack-wide on every such policy; accepted as a known gap, not a blocker (no tool reads traces back yet). - [Q7] Moved — E3's distribution and pluggable-backend design questions (this Q and [Q8]) now live in
INVESTIGATE-selftest-cli.md's own [Q5]/[Q2], since a first design pass turned "if/when E3 is built" into real enough design work to warrant its own document. - [Q8] Moved — see
INVESTIGATE-selftest-cli.md's [Q1] (backend selection) and [Q2] (credential-model abstraction). [Q9]Confirm E1 over a bespoke live-status page? Resolved 2026-07-14: yes, E1. Confirmed directly by the maintainer — ~90% of Sentry's UX for ~0% of the infra cost, no service to stand up or maintain, and reinforced by [Q1]'s answer (a public link is the only option that doesn't assume Grafana Cloud access the developer doesn't have).
Next Steps
- Maintainer decides [Q1]–[Q5], [Q9] — done 2026-07-14 ([Q6]–[Q8] were already resolved/moved earlier)
- Set up Grafana Cloud "Share externally" on the dashboard, confirm the pre-filtered-by-
service_nameURL pattern actually works (Option E1) — decided, likely the single highest-value, lowest-cost action in this whole investigation, still the only piece not done as of 2026-07-14 - Split
using/onboarding/into an operator doc and a developer-facing checklist (Option A) — done 2026-07-14, differently than specified. Not a 3-way generic template split:using/onboarding/index.md(operator recipe) gained an automated verification step;using/onboarding/ollacrm/index.mdwas rewritten from a historical worked example into a clean, ollacrm-specific 4-step checklist. Closer to a real developer quickstart than originally planned, but not yet a generic reusable template for the next customer after ollacrm — revisit that generalization when a second customer onboards. - Design and implement the connection-string format (Option B) — done 2026-07-14, not as specified. Not a single JSON→base64 opaque string. Instead:
onboard-system.shtakes raw tokens, runs a realsovdev-selftestcheck, and only writes a plain multi-var handover file if it passes — verified both directions (real pass → correct file; broken token → no file at all). The maintainer judged "mechanically-verified before handover" a stronger property than "one bundled string," and chose this instead. - Option E3 (bespoke self-test CLI, Grafana Cloud + UIS) shipped — see
INVESTIGATE-selftest-cli.md. Now also the fail-fast stage of the CI consistency gate (seeINVESTIGATE-tenant-write-isolation.md's sibling work). - Mint ollacrm their own LBAC-scoped, read-only Access Policy so the shipped
sovdev-selftesttool is safe to actually hand them — done 2026-07-14.ollacrm-verifycreated live in the real portal, confirming [Q6]'s last unconfirmed sub-point directly: the "Label selectors" control (an expandable "Label selectors (0)" section below the Scopes table, not a per-checkbox button as first guessed) genuinely exists and works on this account's plan tier. One real mismatch caught and fixed during setup: the first attempt used an exact-match selector (service_name="ollacrm-api"), which would have silently brokensovdev-selftest's read-back (it writes underollacrm-api-selftest, a suffixed disposable name) — corrected to the regex form (service_name=~"^ollacrm-api.*") [Q3] already specified. Also newly confirmed live: label selectors don't covertraces(Grafana's own UI states this directly), sotraces:readstays stack-wide on this policy — a known, accepted gap. The standardized per-customer recipe (both the ingest and verify Access Policy steps, and the full env var template covering both the app's OTLP vars andsovdev-selftest'sGRAFANA_CLOUD_*vars) is now documented inusing/onboarding/index.mdsteps 2-3 and 5-6, so future customers get one consistent process instead of this ad-hoc discovery. Still open: wiringollacrm-verify's vars into ollacrm's actualdeploy.yml(tracked inusing/onboarding/ollacrm/index.md). - Option D (shutdown-handler convenience function) — deferred, revisit once a second server-shaped consumer exists
- Options A and B — executed directly, no
PLAN-*.mddrafted (matches this project's own precedent for small, single-session work, e.g.INVESTIGATE-repo-and-package-ownership.md) - Option E1 remains — small enough to also execute directly once picked up, no plan needed
- Once E1 ships, this investigation can move to
completed/