Mattermost Engineering Knowledge Base

PR Status Tracker

Running status of all active PRs across Mattermost repos. Last updated: 2026-10-23 17:10 UTC

Sentry Fix PRs

Total user impact addressed: ~39,000+ users across 8 PRs.

Paid Search Attribution: Findings and Best Practices

Source data: UTM_Raw_Data_Sept_1_to_Oct_15.csv (Marketo export, 5,125 unique people created 2026-09-01 to 2026-10-15) and how_lead_source_is_set_in_marketo.docx/.html (Person Source rules, audit of Oct 15, 2026).

1. Scope

List imports (3,632 rows, 71% of the export) can be ignored for web campaign attribution. None of them has any utm_*, ft_*, gclid or FT Referrer value, and Most Recent Lead Source is blank on all of them. Their Person Source was set by hand (3,468 “Unknown”, 150 “Live Event”, 12 “Paid Social”, 2 other). They are mostly ABM and event lists.

From 62 Minutes to 14: How We Sharded Our Go Test Suite

At Mattermost, our server test suite had a problem: it took about 62 minutes to run on every pull request. For master builds with the race detector enabled, that number climbed to over 90 minutes. We recently shipped test-level sharding (#35739) that brings PR runs down to ~14 minutes — a 77% improvement.

Thanks to Jesse (lieut-data), Doug (wiggin77), and Eva (esarafianou) for a thorough code review that caught real issues and made the implementation significantly better.

Server Test Coverage Strategy

Date: 2026-03-23
Baseline: 33.84% (201,671 lines across 853 files)
Source: Codecov commit report

Current State

The 33.84% figure is real production coverage — generated code (retrylayer, timerlayer), mocks, and test utilities are correctly excluded from the denominator. The gap between 1,287 total files and 853 instrumented files is mostly generated layers and mock code that Go’s coverage tool correctly ignores.

Website Forms and Tracking Audit

Repo audited: /Users/pavelzeman/projects/mattermost/mattermost-website (Astro, deployed through Cloudflare Workers; commit 4aeda543).

Severity: High = loses or corrupts attribution today. Medium = inconsistent or a risk. Low = hygiene.

1. Summary

All lead capture goes through one adapter, src/components/utils/marketo-forms.ts (1,087 lines). That is good news for the rebuild, because there is one place to change. But the adapter is entirely Marketo-specific, so it has to be replaced for Pardot, not just reconfigured.

Workstream Tracker

Last updated: 2026-03-20

1. Sentry Issue Triage & Fix

Goal: Identify, fix, and PR low-hanging-fruit Sentry issues across Mattermost projects.

Full log: Sentry Fixes Log

Summary (as of 2026-03-20)

• 11 issues triaged across server, mobile (iOS + Android), and supporting libraries
• 8 draft PRs created with tests
• ~39,000+ users impacted across all fixes
• 1 upgrade advisory published (QC cluster panic, fixed in v11.0.1+)
• 1 skipped (single modified fork)
• 1 in progress (VF — nil pointer in patchUser/UpdateUser)

Biggest wins by user impact

Next Steps

• Complete VF investigation (nil pointer in patchUser/UpdateUser, 14 events)
• Get reviewer assignments on green PRs
• Pavel: build/test #9610 (iOS SQLite) on M4 MacBook Air

2. CI Improvements

Goal: Faster CI pipelines for mattermost/mattermost server.

Marketo and Salesforce Configuration Issues

Issues found in how UTM and attribution data is captured and stored across Marketo and Salesforce, based on the campaign-by-year analysis in sfdc-utm-campaigns-over-the-years.md.

Provenance: The findings below came from a separate analysis that was pasted in. I did not run the Salesforce queries and have not verified the claims. Items that are diagnoses or guesses, not observed facts, are marked Hypothesis. Where the Marketo CSV I analyzed supports or contradicts a claim, that is noted.

CI Coverage Persistence — Current State & Options

Date: 2026-03-23 (updated)
Context: PR #35743 re-enabled server test coverage with 4-shard parallelism. Coverage uploads to Codecov succeed, but the developer experience is incomplete.

Where We Are

What works:

• 4-shard coverage run completes without OOM (6-8 min per shard)
• Merged cover.out uploads to Codecov successfully (3.5MB, 853 files)
• Codecov commit report exists: link

What doesn’t work:

• No PR comments from Codecov showing coverage diff
• No codecov.yml — everything runs on Codecov defaults
• Generated code inflates the denominator (see below)
• No cover.out artifact saved for historical analysis
• No baseline on master yet — Codecov can’t compute diffs without a reference point

The Numbers Problem

Go’s coverage tool instruments everything compiled into the test binary, including generated wrappers, mocks, and test infrastructure. This inflates the denominator:

CI Pipeline Analysis v2: mattermost/mattermost

Last updated: 2026-04-09 — based on master run 24166497256. Follow-up to v1 analysis from 2026-03-20.

Progress Summary

Three weeks of CI improvements have reduced the PR critical path from ~60min to ~11min — a 5.5x speedup.

What Shipped

New Issues Discovered

1. Binary parameters mode hangs — running with fullyparallel causes runner kills (PR #35992 investigating)
2. Race detector effectively disabled — condition if [[ github.ref_name == 'master' && fullyparallel != true ]] never fires since fullyparallel defaults to true
3. Build is now the bottleneck — at ~11min, it’s the longest job on the critical path (previously dwarfed by 60min tests)

1. Workflow Inventory

No changes to the workflow inventory since v1. The repository still contains 31 workflow YAML files. See v1 analysis for the full list.

PR Daily Status

Running daily log of PR status updates. Newest entries at top.

2026-10-30 (Friday)

27 open PRs. Since 2026-10-29: no merges, no new PRs, no new unresolved threads, no new reviews. Only change: mattermost#35722 (draft) is now MERGEABLE/BLOCKED (was reported unknown yesterday).

🟢 Near-Ready (approved, CI green, MERGEABLE):

• mattermost#36181 — Strip debug symbols — APPROVED ✅, CI green (61 checks, 0 failing), MERGEABLE/CLEAN. ⚠️ 2 unresolved threads (JulienTant, gate stripping on MM_DEBUG).

🟡 Awaiting Review (CI green):

What Not to Do: Writing Go Tests That Survive Parallel Execution

When we set out to re-enable test coverage with sharding on our server CI, we hit a wall within minutes: 45 tests crashed — not failed, crashed — taking down the entire test binary for the package. The culprit was a single line of code that’s perfectly fine in serial tests but lethal under parallel execution.

This post catalogues what we learned about writing Go tests that work in parallel mode, drawn from debugging real failures across our 16,000-test codebase.

Salesforce UTM Campaigns Over the Years

Lead counts by UTM campaign value, grouped by year, from Salesforce. 23,809 leads across 11 years (2016 to 2026).

Provenance: These figures were pasted in from a separate analysis. I did not run the query and have not verified the numbers. The Salesforce field used for the campaign value, the field used to assign the year (likely created date), and the query date are not stated. Only the per-year lead totals were checked: they sum to 23,809.

CI Pipeline Analysis: mattermost/mattermost

Last updated: 2026-03-20 — based on actual YAML files from the mattermost/mattermost repository and real timing data from GitHub Actions API.

1. Workflow Inventory

The repository contains 31 workflow YAML files in .github/workflows/. Here is a complete inventory:

2. Job Catalog

2.1 Server CI (server-ci.yml) — The Critical Path

Triggers: push to master/release-*, pull_request on server/** paths.

Code Review Checklist

Before Writing Code

• grep -rn the codebase for the pattern you’re fixing — fix ALL occurrences, not just the one you found
• Check real API contracts for every input: does it throw, return null, coerce, or silently succeed?
• Understand the type system: identify all enum variants and type union members before writing logic

Production Code

• Use exhaustive when/switch on enums and type unions — no else catch-alls that assume a type
• Every error return/throw gets a unique detail string — never reuse the same message for different failure modes
• Defensive nil/null checks at trust boundaries
• No silent swallowing of unexpected cases

Tests

• Test the actual function, not a copy of its logic
• If a test helper mirrors production code (e.g., coroutine callback), document it and note it must stay in sync
• Mocks match real behavior: verify what the real API does for each input before mocking
• Cover all type/enum variants, not just the happy path

CI / Shell Scripts

• Run the full script locally under set -euo pipefail before pushing — no experimenting in CI
• Initialize all arrays with =() to avoid “unbound variable” under set -u
• Test with empty inputs and empty arrays
• Escape shell metacharacters in variables
• YAML-parse workflow files — don’t hand-edit complex YAML blindly

PR Submission

• Sentry impact summary at the top of the PR description
• Include a release-note: block where required
• No fork migration mentions — employees push to org branches
• No internal/private URLs in public-facing text

Proposed Plan: Rebooting Advertising, Lead Capture and Attribution

Status: Proposal for review. Draft dated 2026-10-15. Scope: Google Ads and Microsoft (Bing) Ads, the mattermost.com website and landing pages, Marketo (migrating to Pardot by December 2026), Salesforce, and the reporting layer. Premise: We are willing to discard the current setup and rebuild it. Historical data is archived, not migrated as-is.

Pardot is now sold as Marketing Cloud Account Engagement. This document says “Pardot” throughout, as the team does.

1. Why we are doing this

Evidence from the three analyses in this folder (best-practices.md, sfdc-utm-campaigns-over-the-years.md, marketo-sfdc-configuration-issues.md). The Marketo export figures cover Sept 1 to Oct 15, 2026. The Salesforce findings come from a pasted analysis I could not verify.