Optional Providers, and Galaxy CoDex over live per-tool dashboards

Status

accepted

Context and decision

Issue #79 asked for Galaxy usage — tool executions, unique-user accounts, and public-instance availability. The v1 spec already anticipated this shape (docs/spec/0001-boast-v1.md: “further Providers are optional and enabled via Manifest/flags”), but no non-default registry existed yet: providers/mod.rs’s render_providers DEFAULT column read “yes” throughout “for want of a contrast” (ADR-0009), and ADR-0009 itself considered making Docker Hub opt-in and rejected it only “for now”, explicitly flagging “revisit if an opt-in tier is built for other reasons.” Galaxy is that reason: most Projects have no Galaxy wrapper at all, so fetching it by default would add a mostly-empty request to every run rather than a useful one, in a way Docker Hub and Quay (most bioinformatics tools do have a container) do not share.

Two decisions follow from that, both settled in the issue’s own long triage thread before any code was written.

1. Providers can now be optional, disabled by default. providers::optional_providers() is a second registry, disjoint from default_providers(); a user opts a Provider in with --enable-provider <name> (repeatable, unknown name is a CLI usage error) or a Manifest’s enable_providers list. boast providers renders both registries in one table, default rows marked “yes”, optional rows “no” — the exact contrast ADR-0009 was waiting for. An explicit CLI --enable-provider overrides a Project’s Manifest selection entirely (never merges), the same override rule --topic already applies to Cohort selection, so the two stay consistent for a reader who already knows one of them.

2. Galaxy CoDex’s communities/all/resources/tools.json is the source of truth — not the Galaxy Europe Grafana dashboard, and not ToolShed download counts. Both alternatives were live options in the issue thread and both were ruled out on evidence, not preference:

• The Galaxy Europe Grafana dashboard (stats.galaxyproject.eu) reports numbers roughly an order of magnitude higher than CoDex for the same tool (267 vs. 104 runs for rasusa) — “something is different about how the two collect stats,” never fully diagnosed. It also only covers one server (usegalaxy.eu), not the cross-server picture the issue asked for, and it’s a live query interface rather than a bulk-fetchable dataset.
• ToolShed downloads count wrapper-bundle fetches — an installation/update event, not a tool execution. It measures how many Galaxy admins installed a wrapper, not how many researchers ran it, which is the opposite of what “Galaxy usage” was asked to mean.

CoDex republishes one JSON file cross-server, keyless, and reflects the recommended metric shape the issue settled on (execution counts and account counts, not installs). Its own known weakness — the published figures can lag the live servers by an unpublished amount — is disclosed on every Metric via a note long enough to reach the Notices footer (ADR-0005), rather than silently presented as current.

A second wrinkle CoDex forces: one upstream tool can have several Galaxy wrappers, one per subcommand (“suites” in CoDex’s vocabulary — vcflib alone maps to 23). The issue’s own resolution: sum runs across every matched suite, but take the maximum user_accounts across them rather than summing, since the same account can run more than one subcommand and summing would inflate a “how many people” figure into “how many (person, subcommand) pairs.” public_instances unions availability across the matched suites, naming which of the four major public servers (UseGalaxy.org, .org.au, .eu, .fr) matched. Every matched suite ID is recorded in a Provider Note so the aggregation is auditable, and every repository CoDex’s Homepage field doesn’t exactly match is NotApplicable, never a real zero (ADR-0002) — Homepage is deliberately the only field compared; CoDex’s Suite source names the wrapper repository (typically under galaxyproject/tools-iuc), not the upstream Project being measured, and using it would misattribute one repo’s Galaxy usage to a completely different upstream.

Considered options

• Keep every Provider in one always-fetched registry, and make Galaxy request-cheap enough not to matter. No new architecture. Rejected: the cost isn’t request count, it’s relevance — a NotApplicable on every run for the ~99% of Projects with no Galaxy wrapper is noise in every default Report, not a performance question a cheaper request would fix.
• Query the Galaxy Europe Grafana dashboard instead of CoDex, since it’s the most current data. Rejected on the evidence above: an unexplained order-of-magnitude discrepancy against CoDex, single-server-only coverage, and no bulk endpoint — a live dashboard built for humans clicking through a UI, not a source boast about can fetch once and record provenance for.
• Sum user_accounts across every matched suite, matching how runs aggregates. Simpler, one code path for both figures. Rejected: the issue’s own domain expert flagged this would double-count one account using several subcommands of the same underlying tool — the max is the honest “at least this many distinct accounts” figure; the note says so explicitly rather than letting the number imply more precision than it has.
• Treat Suite source (the wrapper repo) as an acceptable secondary match when Homepage is absent. Would raise the match rate. Rejected: it would attribute usage to whichever team happens to maintain the Galaxy wrapper (often galaxyproject/tools-iuc, a shared multi-tool repo) rather than the upstream Project actually being measured — a wrong answer is worse than NotApplicable here (ADR-0002).

Consequences

• providers::optional_providers() and resolve_optional_providers() are the durable extension point for every future non-default Provider, not a Galaxy-specific mechanism — the next opt-in Provider (Altmetric already asks for a key, but stays default/keyless-degraded rather than opt-in; a genuinely optional source like a paid API tier would fit this registry instead) adds one entry to optional_providers() and is done.
• enable_providers joins topics/priority_topics as a Manifest field with the same override shape: CLI-given always replaces a Project’s Manifest value, never merges with it. A reader who understands one already understands the other.
• A Category gained a fifth member, Usage, sitting between Downloads and Citations in every Report format and boast providers’ grouping — CATEGORY_ORDER is the one place this ordering is declared, shared by report.rs and diff.rs. The Snapshot schema version advanced (2 → 3) for the new Category, though nothing about deserialising an older Snapshot changes: it simply never contained a "usage"-tagged row.
• Galaxy’s runs and user_accounts — the two cumulative counts CoDex’s own snapshot can go stale on — each carry the lag caveat on the Metric itself, not just a one-time warning, because Snapshots are re-rendered offline later (ADR-0001) with no re-fetch — the same reasoning ADR-0005 already established for Provider licence notices applies to a data-freshness caveat just as much as a legal one. public_instances doesn’t carry it: installation is a current-state fact CoDex derives the same way regardless of refresh timing, not a count that accumulates staleness.
• A future Provider facing the same “several wrappers, one upstream” shape has a worked precedent: sum what genuinely aggregates (executions), take the maximum of what would double-count under naive summation (accounts), and say which rule applied to which number in the Metric’s own note rather than leaving a reader to guess.