Mattermost Engineering Knowledge Base

PR Status Tracker

Fri, 23 Oct 2026 00:00:00 +0000

Running status of all active PRs across Mattermost repos. Last updated: 2026-10-23 17:10 UTC

Sentry Fix PRs

#	Sentry ID	Repo	Title	PR	Status	Events	Users
1	SERVER-VF	mattermost	Nil pointer in UpdateUser	#35717	✅ Merged	14	TBD
2	SERVER-W9	gosaml2	SAML decrypt panic + expired test cert	#9	✅ Merged	11	1
3	MOBILE-IOS-YND+YNE	mobile	Fatal SQLite crashes from concurrent DB access	#9610	✅ Merged	46,000+	36,000+
4	MOBILE-ANDROID-AX68	mobile	RNN race condition crash during activity destroy	#9617	✅ Merged	9,150	1,580
5	MOBILE-ANDROID-AZ02+AYPW	rn-network-client	OkHttp header crashes from non-ASCII chars	#160	✅ Merged	1,395	654
6	SERVER-W1	mattermost	`createProfileImage` panic on empty username	#35722	⚠️ CR (Draft)	1,793	167
7	SERVER-TY	mattermost	Nil License().IsCloud() sweep — 33 call sites	#35718	🟡 Draft ❌	1,497	1
8	MOBILE-ANDROID-AWY6	mobile	Push notification crash on non-Map error data	#9618	✅ Merged	4,806	574

Total user impact addressed: ~39,000+ users across 8 PRs.

Paid Search Attribution: Findings and Best Practices

Thu, 15 Oct 2026 00:00:00 +0000

Source data: UTM_Raw_Data_Sept_1_to_Oct_15.csv (Marketo export, 5,125 unique people created 2026-09-01 to 2026-10-15) and how_lead_source_is_set_in_marketo.docx/.html (Person Source rules, audit of Oct 15, 2026).

1. Scope

List imports (3,632 rows, 71% of the export) can be ignored for web campaign attribution. None of them has any utm_*, ft_*, gclid or FT Referrer value, and Most Recent Lead Source is blank on all of them. Their Person Source was set by hand (3,468 “Unknown”, 150 “Live Event”, 12 “Paid Social”, 2 other). They are mostly ABM and event lists.

From 62 Minutes to 14: How We Sharded Our Go Test Suite

Mon, 01 Jan 0001 00:00:00 +0000

At Mattermost, our server test suite had a problem: it took about 62 minutes to run on every pull request. For master builds with the race detector enabled, that number climbed to over 90 minutes. We recently shipped test-level sharding (#35739) that brings PR runs down to ~14 minutes — a 77% improvement.

Thanks to Jesse (lieut-data), Doug (wiggin77), and Eva (esarafianou) for a thorough code review that caught real issues and made the implementation significantly better.

Server Test Coverage Strategy

Mon, 01 Jan 0001 00:00:00 +0000

Date: 2026-03-23
Baseline: 33.84% (201,671 lines across 853 files)
Source: Codecov commit report

Current State

Metric	Value
Production Go files	1,287
Files with coverage data	853
Total statements instrumented	146,513
Statements covered	53,132 (36.3%)
Statements uncovered	93,381

The 33.84% figure is real production coverage — generated code (retrylayer, timerlayer), mocks, and test utilities are correctly excluded from the denominator. The gap between 1,287 total files and 853 instrumented files is mostly generated layers and mock code that Go’s coverage tool correctly ignores.

Website Forms and Tracking Audit

Mon, 01 Jan 0001 00:00:00 +0000

Repo audited: /Users/pavelzeman/projects/mattermost/mattermost-website (Astro, deployed through Cloudflare Workers; commit 4aeda543).

Severity: High = loses or corrupts attribution today. Medium = inconsistent or a risk. Low = hygiene.

1. Summary

All lead capture goes through one adapter, src/components/utils/marketo-forms.ts (1,087 lines). That is good news for the rebuild, because there is one place to change. But the adapter is entirely Marketo-specific, so it has to be replaced for Pardot, not just reconfigured.

Workstream Tracker

Mon, 01 Jan 0001 00:00:00 +0000

Last updated: 2026-03-20

1. Sentry Issue Triage & Fix

Goal: Identify, fix, and PR low-hanging-fruit Sentry issues across Mattermost projects.

Full log: Sentry Fixes Log

Summary (as of 2026-03-20)

11 issues triaged across server, mobile (iOS + Android), and supporting libraries
8 draft PRs created with tests
~39,000+ users impacted across all fixes
1 upgrade advisory published (QC cluster panic, fixed in v11.0.1+)
1 skipped (single modified fork)
1 in progress (VF — nil pointer in patchUser/UpdateUser)

Biggest wins by user impact

Issue	Platform	Users	PR
YND+YNE — SQLite concurrent access	iOS	36,000+	#9610
AX68 — RNN activity destroy race	Android	1,580	#9611
AZ02+AYPW — OkHttp header crashes	Android	654	#155
AWY6 — Push notification type cast	Android	574	#9609
W1 — Empty username panic	Server	167 instances	#35700

Next Steps

Complete VF investigation (nil pointer in patchUser/UpdateUser, 14 events)
Get reviewer assignments on green PRs
Pavel: build/test #9610 (iOS SQLite) on M4 MacBook Air

2. CI Improvements

Goal: Faster CI pipelines for mattermost/mattermost server.

Marketo and Salesforce Configuration Issues

Thu, 15 Oct 2026 00:00:00 +0000

Issues found in how UTM and attribution data is captured and stored across Marketo and Salesforce, based on the campaign-by-year analysis in sfdc-utm-campaigns-over-the-years.md.

Provenance: The findings below came from a separate analysis that was pasted in. I did not run the Salesforce queries and have not verified the claims. Items that are diagnoses or guesses, not observed facts, are marked Hypothesis. Where the Marketo CSV I analyzed supports or contradicts a claim, that is noted.

CI Coverage Persistence — Current State & Options

Mon, 01 Jan 0001 00:00:00 +0000

Date: 2026-03-23 (updated)
Context: PR #35743 re-enabled server test coverage with 4-shard parallelism. Coverage uploads to Codecov succeed, but the developer experience is incomplete.

Where We Are

What works:

4-shard coverage run completes without OOM (6-8 min per shard)
Merged cover.out uploads to Codecov successfully (3.5MB, 853 files)
Codecov commit report exists: link

What doesn’t work:

No PR comments from Codecov showing coverage diff
No codecov.yml — everything runs on Codecov defaults
Generated code inflates the denominator (see below)
No cover.out artifact saved for historical analysis
No baseline on master yet — Codecov can’t compute diffs without a reference point

The Numbers Problem

Go’s coverage tool instruments everything compiled into the test binary, including generated wrappers, mocks, and test infrastructure. This inflates the denominator:

CI Pipeline Analysis v2: mattermost/mattermost

Mon, 01 Jan 0001 00:00:00 +0000

Last updated: 2026-04-09 — based on master run 24166497256. Follow-up to v1 analysis from 2026-03-20.

Progress Summary

Three weeks of CI improvements have reduced the PR critical path from ~60min to ~11min — a 5.5x speedup.

Metric	v1 (2026-03-20)	v2 (2026-04-09)	Change
PR critical path (wall-clock)	~60min	~11min	-82%
Master critical path	~93min (with -race)	~11min (no race)	-88%
Postgres (PR, longest shard)	~60min (1 runner)	~8m 42s (4 shards)	-85%
Postgres FIPS (PR, longest shard)	~60min (1 runner)	~9m 01s (4 shards)	-85%
mmctl	~8min	~10min	unchanged
Build	~10min	~10m 40s	unchanged
Binary parameters (master)	~87min	15m 47s ❌ (killed)	broken
Race detection (master)	every push (~93min)	effectively disabled	⚠️
Test coverage	disabled (OOM)	in progress (PR #35743)	🔄

What Shipped

Phase	PR	Status	Impact
Phase 4: Server test sharding	#35739	✅ Merged	4 parallel shards per test matrix
Fullyparallel mode	#35816	✅ Merged	`t.Parallel()` within each shard
Phase 5: Test coverage	#35743	🔄 In progress	4 coverage shards, codecov.yml consolidated
Phase 6: Race detector	—	⚠️ Changed	Effectively disabled; nightly workflow needed

New Issues Discovered

Binary parameters mode hangs — running with fullyparallel causes runner kills (PR #35992 investigating)
Race detector effectively disabled — condition if [[ github.ref_name == 'master' && fullyparallel != true ]] never fires since fullyparallel defaults to true
Build is now the bottleneck — at ~11min, it’s the longest job on the critical path (previously dwarfed by 60min tests)

1. Workflow Inventory

No changes to the workflow inventory since v1. The repository still contains 31 workflow YAML files. See v1 analysis for the full list.

PR Daily Status

Mon, 01 Jan 0001 00:00:00 +0000

Running daily log of PR status updates. Newest entries at top.

2026-10-30 (Friday)

27 open PRs. Since 2026-10-29: no merges, no new PRs, no new unresolved threads, no new reviews. Only change: mattermost#35722 (draft) is now MERGEABLE/BLOCKED (was reported unknown yesterday).

🟢 Near-Ready (approved, CI green, MERGEABLE):

mattermost#36181 — Strip debug symbols — APPROVED ✅, CI green (61 checks, 0 failing), MERGEABLE/CLEAN. ⚠️ 2 unresolved threads (JulienTant, gate stripping on MM_DEBUG).

🟡 Awaiting Review (CI green):

What Not to Do: Writing Go Tests That Survive Parallel Execution

Mon, 01 Jan 0001 00:00:00 +0000

When we set out to re-enable test coverage with sharding on our server CI, we hit a wall within minutes: 45 tests crashed — not failed, crashed — taking down the entire test binary for the package. The culprit was a single line of code that’s perfectly fine in serial tests but lethal under parallel execution.

This post catalogues what we learned about writing Go tests that work in parallel mode, drawn from debugging real failures across our 16,000-test codebase.

Salesforce UTM Campaigns Over the Years

Thu, 15 Oct 2026 00:00:00 +0000

Lead counts by UTM campaign value, grouped by year, from Salesforce. 23,809 leads across 11 years (2016 to 2026).

Provenance: These figures were pasted in from a separate analysis. I did not run the query and have not verified the numbers. The Salesforce field used for the campaign value, the field used to assign the year (likely created date), and the query date are not stated. Only the per-year lead totals were checked: they sum to 23,809.

CI Pipeline Analysis: mattermost/mattermost

Mon, 01 Jan 0001 00:00:00 +0000

Last updated: 2026-03-20 — based on actual YAML files from the mattermost/mattermost repository and real timing data from GitHub Actions API.

1. Workflow Inventory

The repository contains 31 workflow YAML files in .github/workflows/. Here is a complete inventory:

#	File	Name	Triggers	Purpose
1	`api.yml`	API	push(master), pull_request	Builds OpenAPI spec in `api/` dir
2	`build-opensearch-image.yml`	Opensearch Docker Image	push(master) when Dockerfile changes	Builds/pushes opensearch 2.7.0 image
3	`build-server-image.yml`	BuildEnv Docker Image	push(master), PR, workflow_dispatch when Dockerfiles change	Builds/tests/pushes build-server and build-server-fips Docker images
4	`claude.yml`	Claude Code	issue_comment, PR review comment, issues, PR review	AI assistant for @claude mentions
5	`codeql-analysis.yml`	CodeQL	PR(master), schedule(30 5,17 * * *)	Security scanning for Go and JavaScript
6	`docker-push-mirrored.yml`	Push mirrored docker images	push(master) when mirror scripts change	Mirrors Docker images
7	`docs-impact-review.yml`	Docs Impact Review	pull_request	AI-powered docs impact analysis via Claude
8	`e2e-fulltests-ci.yml`	E2E Tests	workflow_dispatch	Full E2E test orchestrator (Cypress + Playwright), triggered by Argo Events
9	`e2e-tests-check.yml`	E2E Tests Check	PR when e2e-tests/, webapp/platform/client/, etc. change	Lint/typecheck E2E tests, trigger E2E runs for E2E-only PRs
10	`e2e-tests-ci-template.yml`	E2E Tests Template	workflow_call	Legacy reusable E2E template (used by e2e-fulltests-ci.yml)
11	`e2e-tests-ci.yml`	E2E Tests (pull request)	workflow_dispatch	PR E2E test orchestrator — resolves PR, checks file changes, calls Cypress + Playwright
12	`e2e-tests-cypress-template.yml`	E2E Tests - Cypress Template	workflow_call	Reusable Cypress E2E template with retries, reporting, sharding
13	`e2e-tests-cypress.yml`	E2E Tests - Cypress	workflow_call	Cypress E2E entry point — runs “full” tests with 40 workers
14	`e2e-tests-on-merge.yml`	E2E Tests (master/release - merge)	workflow_dispatch	Post-merge E2E: enterprise + FIPS editions, Cypress + Playwright
15	`e2e-tests-on-release.yml`	E2E Tests (release cut)	workflow_dispatch	Release E2E: enterprise + FIPS editions from `mattermost/` registry
16	`e2e-tests-override-status.yml`	E2E Tests - Override Status	workflow_dispatch	Manual override of failed E2E status checks
17	`e2e-tests-playwright-template.yml`	E2E Tests - Playwright Template	workflow_call	Reusable Playwright template with sharding, S3 upload, retries
18	`e2e-tests-playwright.yml`	E2E Tests - Playwright	workflow_call	Playwright E2E entry point — runs “full” tests with 4 shards
19	`e2e-tests-verified-label.yml`	E2E Tests/verified	PR labeled	Override failed E2E checks via `E2E Tests/verified` label
20	`i18n-ci-pr.yml`	i18n CI PR	PR when i18n files change	Checks only English translation files changed (non-weblate PRs)
21	`i18n-ci-template.yml`	i18n CI Template	workflow_call	Template for i18n checks
22	`mmctl-test-template.yml`	mmctl CI	workflow_call	Reusable mmctl test template (docker-based)
23	`scorecards-analysis.yml`	Scorecards	branch_protection_rule, schedule(44 6 * * *)	OSSF Scorecard supply-chain security
24	`sentry.yaml`	Sentry Upload	workflow_run(Server CI completed)	Creates Sentry release on Server CI completion
25	`server-ci-artifacts.yml`	Server CI Artifacts	workflow_run(Server CI completed)	Uploads artifacts to S3, builds Docker image, Wiz security scan
26	`server-ci-report.yml`	Server CI Report	workflow_run(Server CI completed)	Publishes JUnit test reports, comments flaky tests on PRs
27	`server-ci.yml`	Server CI	*push(master, release-), PR(server/)	Main server CI: lint, tests, build — THE critical workflow
28	`server-test-template.yml`	Server Test Template	workflow_call	Reusable server Go test template
29	`tag-public-module.yaml`	Tag Public Module	workflow_dispatch	Tags `server/public` module with semver
30	`webapp-ci.yml`	Web App CI	push(master, release-), PR(webapp/*)	Webapp lint, typecheck, tests (sharded), build, coverage

2. Job Catalog

2.1 Server CI (`server-ci.yml`) — The Critical Path

Triggers: push to master/release-*, pull_request on server/** paths.

Code Review Checklist

Mon, 01 Jan 0001 00:00:00 +0000

Before Writing Code

grep -rn the codebase for the pattern you’re fixing — fix ALL occurrences, not just the one you found
Check real API contracts for every input: does it throw, return null, coerce, or silently succeed?
Understand the type system: identify all enum variants and type union members before writing logic

Production Code

Use exhaustive when/switch on enums and type unions — no else catch-alls that assume a type
Every error return/throw gets a unique detail string — never reuse the same message for different failure modes
Defensive nil/null checks at trust boundaries
No silent swallowing of unexpected cases

Tests

Test the actual function, not a copy of its logic
If a test helper mirrors production code (e.g., coroutine callback), document it and note it must stay in sync
Mocks match real behavior: verify what the real API does for each input before mocking
Cover all type/enum variants, not just the happy path

CI / Shell Scripts

Run the full script locally under set -euo pipefail before pushing — no experimenting in CI
Initialize all arrays with =() to avoid “unbound variable” under set -u
Test with empty inputs and empty arrays
Escape shell metacharacters in variables
YAML-parse workflow files — don’t hand-edit complex YAML blindly

PR Submission

Sentry impact summary at the top of the PR description
Include a release-note: block where required
No fork migration mentions — employees push to org branches
No internal/private URLs in public-facing text

Proposed Plan: Rebooting Advertising, Lead Capture and Attribution

Thu, 15 Oct 2026 00:00:00 +0000

Status: Proposal for review. Draft dated 2026-10-15. Scope: Google Ads and Microsoft (Bing) Ads, the mattermost.com website and landing pages, Marketo (migrating to Pardot by December 2026), Salesforce, and the reporting layer. Premise: We are willing to discard the current setup and rebuild it. Historical data is archived, not migrated as-is.

Pardot is now sold as Marketing Cloud Account Engagement. This document says “Pardot” throughout, as the team does.

1. Why we are doing this

Evidence from the three analyses in this folder (best-practices.md, sfdc-utm-campaigns-over-the-years.md, marketo-sfdc-configuration-issues.md). The Marketo export figures cover Sept 1 to Oct 15, 2026. The Salesforce findings come from a pasted analysis I could not verify.

Server Test Sharding Analysis

Mon, 01 Jan 0001 00:00:00 +0000

Last updated: 2026-03-23 — includes benchmark results from live CI runs with per-test sharding.

Context: Why Are We Back at 60 Minutes?

In late 2025, Claudio’s team shipped selective parallelism — enabling t.Parallel() within the heaviest packages (api4, app) and adding a DB connection pool to support concurrent test execution. This cut test runtime from ~40 min to ~15 min.

However, that improvement was only applied to the coverage job (test-coverage with fullyparallel: true and continue-on-error: true). The main PR-blocking test job (test-postgres-normal) still runs with fullyparallel: false — meaning all tests within a package execute serially, and we’re back at ~60 min.

mmworkbrain Workflow — Full Setup Guide

Tue, 06 Oct 2026 00:00:00 +0000

This documents every piece of the mmworkbrain knowledge base — from repo structure to deployment — so anyone can replicate it from scratch.

Overview

mmworkbrain is a Hugo static site using the Google Docsy theme, deployed automatically to Cloudflare Pages on every push to main. The site is optionally protected by Cloudflare Zero Trust (Access) for controlled visibility.

Stack: GitHub (source) → Cloudflare Pages (build + host) → Cloudflare Access (auth)

Code Coverage Investment Strategy

Mon, 01 Jan 0001 00:00:00 +0000

Analysis of server test coverage data from PR #35743, identifying the highest-impact opportunities to improve coverage with the least effort.

Current State

Metric	Value
Total statements (after exclusions)	228,191
Covered statements	69,628
Overall coverage	30.5%
Excluded (generated/test infra)	49,908 statements

Exclusions: store/retrylayer, store/timerlayer, store/storetest, **/mocks, *_serial_gen.go, plugin/plugintest, mmctl/mocks, model/client4.go.

The Big Picture

Two packages alone — channels/app and channels/api4 — account for 134,808 uncovered statements, or 85% of the gap between 30.5% and 100%. Getting just these two packages to 50% coverage would push the overall number from 30.5% to 52.2%.

Go 1.25 Race Condition: Deep Analysis

Mon, 01 Jan 0001 00:00:00 +0000

Technical analysis of the master CI breakage caused by Go 1.25, why the first fix failed, and why the second fix should work.

Timeline

Time (UTC)	Event
Mar 27 19:34	Last green master CI (run 23664056808)
Mar 27 20:11	PR #35817 merges — Go 1.24 → 1.25.8
Mar 27 20:11	First failed master CI (run 23665493896)
Mar 27–31	9 consecutive failures, 0% pass rate
Mar 31 15:32	PR #35881 merges — first fix attempt
Mar 31 15:32	Master CI still fails (run 23805732245)
Mar 31 16:49	PR #35891 opened — second fix attempt

Before Go 1.25: 22/23 green runs (96% pass rate) over 3 days.

Security Issue: GitHub Actions persist-credentials

Mon, 01 Jan 0001 00:00:00 +0000

Summary

The mattermost/mattermost repository’s GitHub Actions workflows leave the GITHUB_TOKEN persisted in the Git configuration (.git/config) throughout workflow runs. This is the default behavior of actions/checkout and represents a supply-chain attack surface — any step running after checkout (including third-party actions) can read and exfiltrate the token.

Severity: Medium (credential exposure via supply-chain compromise) Effort to fix: Low (one-line change per checkout step) Risk of fix: Very low (no functional change unless a step explicitly needs Git push access)

mm-certification-platform: Security Findings Response

Mon, 01 Jan 0001 00:00:00 +0000

Review date: 2026-03-25 Reviewer: External security team Platform: mm-certification-platform (Mattermost Certification Lab) Commits: 81dbb50 · 5e1008d

Summary

The security review identified 31 findings across four severity levels.

Severity	Count	Fixed	Pending decision / manual
Critical	5	4	1
High	5	4	1
Medium	15	13	2
Low	6	3	3

Fix complexity (per finding): 🟢 Simple (< 10 lines, single file) · 🟡 Medium (multi-file or logic change) · 🔴 Complex (cross-module refactor or design change)

Mobile Observability Audit & Strategy v2

Mon, 01 Jan 0001 00:00:00 +0000

September 20, 2026 · Updated after reviewing Sentry Mobile Vitals + Grafana/Prometheus data

Previous version: v1 (September 19, 2026)

What Changed Since v1

v1 was based on code analysis and API queries. v2 incorporates hands-on review of actual dashboards:

Grafana/Prometheus — Server-side client performance metrics (/api/v4/client_perf pipeline)
Sentry Mobile Vitals — Native app start times, frame rates, crash-free rates (90-day window)

The core thesis of v1 is confirmed: we have a massive instrumentation gap between what Sentry measures and what users experience. But now we can quantify it.

Mobile Observability Audit & Strategy v3

Mon, 01 Jan 0001 00:00:00 +0000

September 20, 2026 · Updated with contributor attribution, Grafana coverage analysis, and fast-iteration strategy

Previous versions: v1 · v2

What Changed Since v2

v3 addresses three questions:

Who built the homegrown telemetry? — Full contributor attribution for the Prometheus/Grafana pipeline
Is the Grafana dashboard showing everything? — Analysis of what’s actually collected vs. what’s displayed
How to iterate faster than the monthly release cycle? — Beta/TestFlight strategy for rapid validation

1. Who Built the Homegrown Telemetry Infrastructure

The custom performance telemetry pipeline (mobile → /api/v4/client_perf → Prometheus → Grafana) was built by a small group across server, webapp, and mobile. Here’s the full attribution:

Large Scope PRs

Thu, 24 Sep 2026 00:00:00 +0000

Author: harshilsharma63 Date: September 24, 2026 Prepared by: Tommy ⚙️

Executive Summary

Harshil has 11 open PRs across 5 repos implementing ABAC (Attribute-Based Access Control) channel permissions. The work spans two features — Channel Read Access (earlier) and Channel Write Access (newer) — delivered in two iterations each, with the earlier iterations appearing superseded but not closed. The largest PR touches 83 files with ~4,400 changed lines.

The core concern: the implementation spreads enforcement checks across dozens of individual API handlers rather than using middleware, creating a large review surface and ongoing maintenance burden.

iOS SQLite Background Crash Analysis (0xdead10cc)

Thu, 10 Sep 2026 00:00:00 +0000

Summary

The Mattermost iOS app suffers from ~188,000 crash events across Sentry related to SQLite database lock contention during iOS background/suspension transitions. PR mattermost-mobile#10110 fixes the main app process, but the Notification Service Extension (Gekidou) accounts for ~94% of these crashes and remains unprotected.

The Crash Mechanism: `0xdead10cc`

When iOS suspends an app (or extension), it checks whether the process holds any file locks. If a SQLite WAL/shared lock is held on the App Group database at the moment of suspension, iOS terminates the process with exception code 0xdead10cc (“dead lock”) via the RUNNINGBOARD subsystem.

GitHub Actions Cache & Queue Time Analysis

Fri, 10 Apr 2026 00:00:00 +0000

Analysis of GitHub Actions cache utilization and job queue delays across Mattermost repositories, conducted April 10, 2026.

Executive Summary

Cache pressure: Both mattermost/mattermost (10.3 GB) and mattermost/mattermost-mobile (10.0 GB) are near or at their per-repo 10 GB cache limits. Org-level 50 GB limit likely saturated across all repos.
Queue delays: Standard 2-core runners show zero queuing. 8-core runners (ubuntu-latest-8-cores) regularly queue for 5-22 minutes when Server CI runs overlap.
Root causes: Per-branch cache duplication, stale build branches (168 in mobile), tag caches that are never reused, and limited 8-core runner concurrency.

Cache Analysis

mattermost/mattermost

10.3 GB across 48 caches

Playbooks Improvements POC — PR #2222 Analysis

Fri, 10 Apr 2026 00:00:00 +0000

Analysis of PR #2222 — “Playbooks Improvements - POC” by @catalintomai. Analyzed: 2026-04-10

Size

41,259 additions / 2,557 deletions across 264 files
89 new files, 175 modified
Created Mar 30, 2026 — 11 days old, actively updated (last: Apr 9)

Component Breakdown

Layer	Files	Notes
webapp	119	UI components: checklists, backstage, modals, RHS, profile
server	77	App logic (47), API handlers (12), SQL store (8), tests
e2e-tests	62	Cypress integration tests
client	5	Go client library

Key Feature Areas

Condition-based actions — condition_service.go, condition_actions_spec.js
Task lockdown — task_lockdown.go, task_lockdown_test.go, task_lockdown_spec.js
Creation rules — creation_rules.go, creation_rules_test.go
Role-based assignment — role_based_assignment_spec.js
Playbook events — playbook_events.tsx (1,058 lines new)
Template engine — template_engine_test.go (729 lines new)
Property service improvements — property_service.go, field rename cascade
Permissions overhaul — permissions_service.go (+273/-16), permissions_service_test.go (1,829 lines new)
Run creation enforcement — new validation and rules

Hotspots (largest files)

File	+Lines	Type
`permissions_service_test.go`	1,829	New test file
`playbook_run_service.go`	+1,756/-220	Core business logic, heavily modified
`playbook_events.tsx`	1,058	New UI component
`run_playbook_modal.test.tsx`	933	New webapp test
`condition_actions_spec.js`	868	New E2E test

Test Coverage

Significantly better than the Wiki POC. This PR has tests at every layer:

Wiki/Pages POC — PR #34607 Analysis

Fri, 10 Apr 2026 00:00:00 +0000

Analysis of PR #34607 — “[Fast Futures] Wiki/Pages POC” by @catalintomai. Analyzed: 2026-04-10

Size

170,410 additions / 7,296 deletions across 704 files — one of the largest PRs in the repo
Created Nov 27, 2025 — 4.5 months old at time of analysis

Component Breakdown

Area	Files	Notes
`webapp/channels`	424	Bulk of the PR — full React UI for wiki/pages
`e2e-tests/playwright`	78	Comprehensive E2E test suite
`server/channels/app`	55	Business logic layer
`server/channels/store`	34	Database layer (store interfaces + SQL)
`server/public/model`	19	Data models
`server/channels/db/migrations`	19	Database migrations
`server/channels/api4`	17	REST API endpoints
`webapp/platform`	13	Platform-level webapp changes
`server/cmd/mmctl`	6	CLI support
`e2e-tests/cypress`	5	Cypress E2E tests
`api/`	4	OpenAPI spec (new `wikis.yaml`)
Other	30	Plugin API, metrics, Grafana dashboards, i18n, build scripts

This touches every layer of the stack: API spec → REST handlers → app logic → store/SQL → migrations → models → plugin API → webapp UI → E2E tests → metrics/Grafana dashboards.

ARM64 Docker Image Feasibility Analysis

Wed, 01 Apr 2026 00:00:00 +0000

This document analyzes the feasibility and required work to add ARM64 support to Mattermost’s official Docker images, based on examination of the actual build infrastructure as of April 2026.

Executive Summary

Feasibility: High — most of the groundwork already exists. The Mattermost server already builds and ships linux/arm64 binaries. The main Dockerfile uses multi-arch base images. Plugins already build for arm64. The primary work is in the Docker image build/push pipeline (adding --platform linux/arm64,linux/amd64 to buildx) and ensuring the MM_PACKAGE URL resolves correctly for each architecture.

Team Email API Behavior Analysis

Wed, 01 Apr 2026 00:00:00 +0000

Summary

The team email field is silently server-controlled on both create and update. On create, it’s overridden with the creating user’s email. On update, it’s excluded from the sanitized allowlist. On read, it’s stripped entirely via Sanitize(). The API accepts email in the request body but ignores it — the worst possible behavior from an API design perspective.

Create Path

API Handler → App → Store

server/channels/api4/team.go:78 — createTeam handler decodes the request body (including email) and lowercases it (line 84: team.Email = strings.ToLower(team.Email)). Then calls c.App.CreateTeamWithUser().

AWS Domain Inventory

Tue, 24 Mar 2026 00:00:00 +0000

Scan of all Route53 hosted zones across the Mattermost AWS org. Covers DNS resolution, HTTP liveness, subdomain counts, and cleanup recommendations. Raw data and script at pavelzeman/aws-domains.

Access Methodology

This scan used two access paths due to IAM permission constraints:

Layer	Method Used	Ideal Method
List org accounts	`organizations:ListAccounts` from management account	Same — only root account can enumerate
Cross-account Route53	AWS Config aggregator (fallback)	Assume `OrganizationAccountAccessRole` per member account from management account
Registered domains	`route53domains:ListDomains` from management account	Same — SCP blocks this on member accounts

Why the ideal path failed: mattermost-mgmt-ro (ReadOnlyAccess) lacks sts:AssumeRole. 122059300807_AWSAdministratorAccess is a member account — it cannot enumerate the org or assume roles in sibling accounts.

CI Supply Chain Risk Analysis

Mon, 23 Mar 2026 00:00:00 +0000

Analysis of supply chain attack vectors in CI workflows across mattermost/mattermost and mattermost/mattermost-mobile repositories. Triggered by a review comment identifying that npx junit-report-merger@7 executes unverified code from npm at CI runtime.

Executive Summary

The mattermost server repo has excellent GitHub Actions hygiene — all external actions are SHA-pinned — but introduces supply chain risk through 3 unpinned npx calls that fetch and execute npm packages at CI runtime with no integrity verification.

CI Analysis: mattermost-mobile

Fri, 20 Mar 2026 00:00:00 +0000

Run analyzed: #23360024337 (PR #9616)
Total wall time: ~11 minutes (19:51:19 → 20:02:28 UTC)
Single job, fully sequential — no parallelism at all.

Current State: Step-by-Step Timing Breakdown

The workflow has one job (test) with two composite action steps running sequentially:

Step	Duration	What it does
Set up job	~0.5s	Runner provisioning (Ubuntu 22.04)
ci/checkout-repo	~3s	Shallow clone (`depth=1`) — fast ✅
ci/test	2m 38s	Lint + typecheck (see breakdown below)
ci/test-coverage	8m 9s	Jest with coverage (see breakdown below)
Post steps	~14s	Cache save, cleanup

ci/test breakdown (2m 38s)

Sub-step	Duration	Notes
setup-node	~4s	Downloads Node 22.14.0
npm cache restore	~18s	818 MB cache download + decompress
`npm ci --ignore-scripts`	~58s	Full install from lockfile
patch-npm-deps (Sentry CLI)	~6s
`npm run check` (lint + tsc)	~75s	ESLint ~75s, tsc runs after but seems fast or included
check-i18n	~6s

ci/test-coverage breakdown (8m 9s)

Sub-step	Duration	Notes
setup-node	~0.5s	Already cached from previous step
npm cache restore	~9s	Cache hit (same job)
`npm ci --ignore-scripts` (again!)	~48s	Redundant — deps already installed
patch-npm-deps (again!)	~0.5s
Download main coverage artifact	~1s
`npm run test:ci:coverage`	~6m 57s	Jest `--runInBand --coverage`
Compare + post coverage	~2s

Bottlenecks Identified

🔴 Critical: Jest runs with `--runInBand` (6m 57s)

The test command uses --runInBand which forces serial test execution on a single core. GitHub-hosted ubuntu-22.04 runners have 2 cores. This is the single biggest bottleneck.

Hugo Theme Deployment Troubleshooting

Thu, 19 Mar 2026 00:47:00 +0000

Meta Note: This was our first real engineering challenge captured in this knowledge base - troubleshooting the Hugo Book theme setup that powers this very site!

Problem Statement

Symptom: Hugo builds complete successfully on Cloudflare Pages, but the deployed site returns 404 for all pages.

Investigation: Build logs show no errors, deployment completes with “success” status, but only RSS/XML files are generated - no HTML pages.

Environment

Platform: Cloudflare Pages
Hugo Version: 0.146.0 (pinned)
Theme: Hugo Book (alex-shpak/hugo-book)
Build Command: hugo --minify
Repository: GitHub with auto-deployment

Root Causes Identified

1. Configuration File Precedence Issue

Problem: Hugo configuration split between config.yaml and hugo.toml

Android Bearer Token Corruption (AYPW)

Mon, 01 Jan 0001 00:00:00 +0000

Investigation into why the Mattermost mobile JS layer sends a corrupted bearer token containing control character 0x02 to the Android native network client, causing OkHttp crashes.

Sentry Issue Summary

Issue: MATTERMOST-MOBILE-ANDROID-AYPW
Error: IllegalArgumentException: Unexpected char 0x02 at 9 in Authorization value
Location: okhttp3.Headers$Companion.checkValue → called from Extensions.kt:applyHeaders
Crash path: upload → buildUploadCall → buildRequest → applyHeaders
Count: 20 events, first seen 2025-12-01, last seen 2026-03-19
Severity: Unhandled (UncaughtExceptionHandler) — app crashes
Device: Samsung SM-F766N (Galaxy Z Flip 3)

The “0x02 at position 9” Clue

The Authorization header value is BEARER <token>. Position 9 (0-indexed) is the 2nd character of the actual token (after "BEARER " = 8 chars). The token itself contains a 0x02 (STX control character) near the beginning.

Enterprise CI Cross-Repo Failure Analysis - March 26, 2026

Mon, 01 Jan 0001 00:00:00 +0000

Executive Summary

On March 26, 2026, a cross-repository dependency break caused all server PRs branched before 07:54 UTC to fail Enterprise CI. The root cause was a timing/ordering issue — the server-side methods were merged first (#35157 at 07:54 UTC), enterprise code was updated to use them (#2088 at 08:31 UTC), but any server PR that didn’t include #35157’s changes would fail when tested against the updated enterprise code. The fix was simply rebasing affected server PRs onto current master.

i18n Cross-Repo Coordination Problem

Mon, 01 Jan 0001 00:00:00 +0000

How It Works Today

The Pipeline

mmgotool i18n extract (in mattermost/tools/mmgotool/commands/i18n.go) walks Go source files across multiple directories:
- server/ (mattermost server code)
- ../../enterprise/ (enterprise code, relative to server/)
- ../model/, ../plugin/ (shared packages)
It parses Go AST looking for calls to NewAppError(), T(), NewUserFacingError(), translateFunc(), etc., extracting string literal IDs.
It merges extracted IDs with existing server/i18n/en.json — adding new keys (with empty translation), removing stale keys.
Enterprise CI (ci-template.yml) checks out both repos side-by-side, then:

Mattermost Integrations Marketplace Roadmap — 2026 and Beyond

Mon, 01 Jan 0001 00:00:00 +0000

Context

Mattermost’s current marketplace is a read-only catalog — a stateless Go service (mattermost/mattermost-marketplace) serving a manually curated plugins.json file via AWS Lambda + S3 + CloudFront at api.integrations.mattermost.com. The public website at mattermost.com/marketplace is a separate CMS frontend (likely WordPress). There is no developer portal, no self-service submission pipeline, no monetization infrastructure, and no automated code review. Every plugin listing is manually managed by the Mattermost team.

The goal is to evolve this into a living platform ecosystem: open to 3rd-party developers, capable of hosting paid/free integrations, with Mattermost retaining quality and security control — ideally augmented by AI review — delivered in incremental, shippable stages.

MATTERMOST-SERVER-QC / QK / QH / WF — Nil pointer in channel permission check

Mon, 01 Jan 0001 00:00:00 +0000

Issue

Nil pointer dereference panic in isChannelArchivedAndHidden during channel permission checks. Occurs when accessing a deleted or non-existent channel through various API endpoints (get channel stats, create post, upload file, search files).

Impact: Server crashes (panic/fatal) when users interact with deleted channels or channels that fail to load from the database.

Status: Fixed

Fixed in: PR #30628 ([MM-63624] Fix potential nil dereference in app.isChannelArchivedAndHidden)
Merged: 2025-04-03 by @streamer45
Available in: v11.0.1 and later
Jira: MM-63624

Affected Instances (still hitting this as of 2026-03-19)

Instance	Sentry Issue	Events	Last Seen
chat-matter.com	QC	~122	2026-03-19
chat.unilox.com	QC	~66	2026-03-19
chat.stil.dk	QC	~35	recent
mattermost.ilabt.imec.be	QC	~3	recent
mm.common.fk.dks.lanit.ru	QC	~2	recent
mm-kocho.madhermit.net	QK	71	2026-01-03
mattermost.demetio.com	QH	40	2026-01-13
chat.gebit.local	WF	8	2026-03-06

Note: These instances are running custom/forked builds (Sentry release tags are commit hashes, not version tags), predating the April 2025 fix.

Mobile Observability Audit & Strategy

Mon, 01 Jan 0001 00:00:00 +0000

September 2026 · Authored for the Mobile Platform Team

Executive Summary

On September 19, 2026, Jason Blais reported degraded mobile app performance on iOS — slow channel switches, dropped reactions, truncated messages, and mark-unread failures. All symptoms worsened on degraded cellular networks and disappeared on good WiFi. The engineering team’s response was to ask Jason to test in airplane mode and provide screen recordings — a classic “works on my end” anti-pattern that shifts the burden of instrumentation onto the user.

PRFAQ - Mattermost Terraform Provider

Mon, 01 Jan 0001 00:00:00 +0000

PRFAQ: Mattermost Terraform Provider

Document Status: Draft Author: Pavel, SVP Engineering Date: April 2026

Press Release

Mattermost Launches Official Terraform Provider, Enabling Infrastructure-as-Code for Mission-Critical Collaboration

Palo Alto, CA — Mattermost, the leading open-source platform for secure, mission-critical collaboration, today announced the general availability of its official Terraform Provider. The provider enables engineering and platform teams to declare, version-control, and automatically enforce their entire Mattermost configuration as code — bringing the same operational rigor used for cloud infrastructure to the collaboration layer.

Schalla Cursor Workflow

Mon, 01 Jan 0001 00:00:00 +0000

The Oxide AI-Assisted Development Workflow

What this is

A structured method for building non-trivial software with an AI coding agent (Cursor, Claude Code, or similar) where architectural integrity is non-negotiable and multiple engineers or sessions need to pick up where another left off without drift.

Phase 0: Problem Definition (human only, no AI)

Do this before opening any tool.

Write a single document that answers four questions with precision:

1. What is the one-sentence demo?

Not goals, not features — the exact scenario you would screen-record in 90 seconds to prove the thing works.

Sentry Issue Triage & Fix Workflow

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the end-to-end process for triaging Sentry issues, creating fixes, and getting them through code review. It is designed for both human engineers and AI agents working on Mattermost codebases.

Overview

Sentry Issue → Investigate → Check if already fixed → Find Source Code → Write Failing Test
→ Validate test fails → Create Fix → Validate test passes (+ all other tests)
→ Draft PR (with customer impact) → Human Review → Merge → Update Sentry

Step 1: Identify the Issue

Sources

Sentry Dashboard: mattermost-mr.sentry.io
API: GET /api/0/projects/mattermost-mr/{project}/issues/?query=is:unresolved&sort=freq

Active Projects (skip `-test` projects)

Project	Platform	Repository
`mattermost-server`	Go	mattermost/mattermost (server/v8)
`mattermost-desktop`	Electron	mattermost/desktop
`mattermost-mobile-ios`	React Native / Swift	mattermost/mattermost-mobile
`mattermost-mobile-android`	React Native / Java/Kotlin	mattermost/mattermost-mobile

Prioritization Criteria

Quick wins first: Simple, clearly identifiable root cause with a narrow fix
Customer impact: Number of distinct instances/domains affected (extract from url tag)
User impact: High userCount in Sentry = more users affected
Frequency: High count = happening often (but check if it’s one instance or many)
Recency: lastSeen today means it’s actively happening

What Makes a Good Quick Win

Missing nil/null checks (Go nil pointer, JS undefined)
Missing bounds checks before array/slice access
Type assertion without checking (Go interface{} cast)
Input validation gaps (malformed data from external sources)
Off-by-one errors

Step 2: Investigate the Stacktrace

Via Sentry API

# Get issue details
curl -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
 "https://us.sentry.io/api/0/issues/{issue_id}/events/latest/"

# Get customer domains from url tag
curl -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
 "https://us.sentry.io/api/0/issues/{issue_id}/tags/url/?per_page=100"

# Look at the exception entries for:
# - Exception type and value
# - Stack frames (focus on inApp=true frames)
# - Tags (release version, OS, device, url)

Key Information to Extract

Exact file and line number where the crash occurs
Call chain leading to the crash (what triggered it)
Release version (is this in latest? or old version only?)
Frequency pattern (increasing, stable, declining?)
Customer domains (deduplicated from url tag — who is affected?)

Step 3: Check if Already Fixed

Before writing any code, check if the bug exists on the latest main/master branch.

Sentry Low-Hanging Fruit

Mon, 01 Jan 0001 00:00:00 +0000

Identified 2026-03-19. Categorized by actionability after verifying each issue against current master and checking whether release commits exist in the upstream repository.

Key finding: The majority of server-side Sentry issues originate from customer forks with custom code not present in upstream Mattermost. Several others are already fixed in newer releases.

Category 1: Current — Reproducible on Master

Issues that exist in the current upstream codebase and can be fixed with a PR.

Sustained Engineering Bug Triage Process

Mon, 01 Jan 0001 00:00:00 +0000

SET (Sustained Engineering Tool) is a Go CLI that automates how we triage, prioritize, and manage incoming Jira bugs for the Sustained Engineering team. It runs as scheduled GitHub Actions workflows against our Jira instance, with all decisions posted to Mattermost channels for visibility. The tool uses Claude (Anthropic API) for AI-assisted classification and an Atlassian Compass integration to map issues to the correct owning team via component ownership.

Source: mattermost/set

Test time.Sleep Audit

Mon, 01 Jan 0001 00:00:00 +0000

Audit of time.Sleep calls in server/ test files in mattermost/mattermost. These are fragile timing dependencies that break under the race detector (which slows execution 2-10x) and contribute to flaky tests.

Total: 191 occurrences across test files.

Recommended replacement: require.Eventually(t, func() bool { ... }, timeout, interval) or channel/sync-based signaling.

High Risk (≥1s) — 66 occurrences

These are both slow AND fragile. Each adds seconds to test runtime while still being too short under -race.

Mattermost Integrations Marketplace — Execution Plan

Mon, 01 Jan 0001 00:00:00 +0000

Iteration of iterative-bouncing-nest.md with operational constraints, repo structure, infrastructure, telemetry, GitHub milestones, and GitHub issues.

Context

The existing marketplace is a stateless read-only Lambda (mattermost/mattermost-marketplace) serving a compiled plugins.json with no database, no auth, no telemetry, and no developer self-service.

The goal is to evolve it into a living platform ecosystem — open submissions, AI review, monetization, private enterprise registries — in incremental shippable stages that never destabilize production.

Android Development Setup

Mon, 01 Jan 0001 00:00:00 +0000

Minimal setup for running Kotlin unit tests in the OpenClaw container. Not a full Android build environment — just enough for ./gradlew test (JVM unit tests).

Install JDK 17

curl -sL https://download.oracle.com/java/17/archive/jdk-17.0.12_linux-x64_bin.tar.gz | tar -xz -C $HOME
export JAVA_HOME=$HOME/jdk-17.0.12
export PATH=$JAVA_HOME/bin:$PATH

Install Android SDK (minimal)

# Command-line tools
mkdir -p $HOME/android-sdk/cmdline-tools
cd $HOME/android-sdk/cmdline-tools
curl -sL https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip -o tools.zip
$JAVA_HOME/bin/jar xf tools.zip # unzip not available in container
mv cmdline-tools latest
rm tools.zip
chmod +x latest/bin/*

# Accept licenses + install platform
export ANDROID_HOME=$HOME/android-sdk
yes | $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --licenses
$ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager "platforms;android-34" "build-tools;34.0.0"

Disk usage

Component	Size
JDK 17	~305 MB
Android SDK (minimal)	~437 MB
Total	~740 MB

Running Kotlin unit tests

For libraries without a Gradle wrapper (like react-native-network-client), use the standalone test-runner pattern:

Microsoft Intune SDK Crashes — iOS

Mon, 01 Jan 0001 00:00:00 +0000

Date: 2026-03-20
Status: Research complete — upstream issue filing recommended

Executive Summary

Two crashes in the Microsoft Intune SDK (IntuneMAMSwift.framework) are collectively affecting ~12,400 unique users of Mattermost iOS. Both crashes originate entirely within SDK code — Mattermost cannot fix them directly. The SDK is closed-source (binary-only xcframeworks), so code contributions are not possible. The recommended path is filing detailed issues on the SDK’s GitHub repo and upgrading to the latest SDK version (21.5.1), which includes a launch-time crash fix.

Debugging Stripped Mattermost Binaries with Delve

Thu, 23 Apr 2026 00:00:00 +0000

When Mattermost ships production binaries built with go build -ldflags='-s -w', DWARF debug info and the symbol table are stripped to reduce binary size. This guide covers how to preserve that debug information separately and use Delve to debug stripped binaries effectively.

Prerequisites

Tool	Minimum Version	Purpose
`dlv`	1.21+	Go debugger with `--debug-info` flag and debuginfod support
`go`	1.21+	Match the version used to build the binary
`objcopy`	any (binutils)	Extract and reattach debug sections
`file`	any	Quick check: stripped vs. not stripped
`readelf`	any	Inspect ELF sections, build IDs

Install Delve:

Playwright E2E Sleep vs Wait Pattern Analysis

Thu, 23 Apr 2026 00:00:00 +0000

Analysis date: 2026-04-23 · Source: mattermost/mattermost HEAD · Directory: e2e-tests/playwright/

Executive Summary

The Playwright E2E test suite contains 229 fixed-delay sleeps totaling 245.6 seconds (4+ minutes) of unconditional waiting. Over 93% of this sleep time is concentrated in the ABAC (Attribute-Based Access Control) test files under system_console/abac/. The single worst file (abac/support.ts) is a shared helper used by multiple ABAC tests, meaning its sleeps multiply across test runs.

While the suite also uses 2,340 proper async waits (a 10:1 ratio of good waits to sleeps), the fixed delays are significant contributors to slow shard execution times — especially since ABAC tests also set extended timeouts (up to 5 minutes per test).

GitHub Actions Workflow Audit

Tue, 31 Mar 2026 00:00:00 +0000

Audit performed on 2026-03-31. The repository contains 65 total workflows (64 active, 1 disabled).

Summary

Category	Count
✅ Active & healthy	29
⚠️ Active but failing	7
🔇 Disabled	1
💀 Stale (no runs in 30+ days)	20
❓ Manual/rare	3
🔁 Potentially redundant	5

Key finding: There are 20+ E2E-related workflows, many stale or superseded. This is the biggest cleanup opportunity.

All Workflows

Core CI (Server & Web App)

Workflow	Path	Last Run	Result	Trigger	Status
Server CI	server-ci.yml	2026-03-31	✅ success	PR	✅ Active
Server CI PR	server-ci-pr.yml	2026-03-30	✅ success	PR	✅ Active
Server CI Master	server-ci-master.yml	2026-03-27	✅ success	push	✅ Active
Server Test Template	server-test-template.yml	2026-03-20	❌ failure	push	⚠️ Failing
Server CI Artifacts	server-ci-artifacts.yml	2026-03-31	🔄 running	workflow_run	✅ Active
Server CI Report	server-ci-report.yml	2026-03-31	✅ success	workflow_run	✅ Active
Web App CI	webapp-ci.yml	2026-03-31	✅ success	PR	✅ Active
Web App CI PR	webapp-ci-pr.yml	2026-03-26	✅ success	PR	✅ Active
Web App CI Master	webapp-ci-master.yml	2026-03-27	✅ success	push	✅ Active
mmctl CI	mmctl-test-template.yml	2025-04-28	❌ failure	push	💀 11 months stale
i18n CI PR	i18n-ci-pr.yml	2026-03-31	✅ success	PR	✅ Active
i18n CI Template	i18n-ci-template.yml	—	—	(template)	💀 Never run
API	api.yml	2026-03-31	✅ success	PR	✅ Active
Playbooks CI	playbooks-ci.yml	2026-03-20	❌ failure	PR	⚠️ Failing
Playbooks CodeQL	playbooks-codeql.yml	2026-03-20	✅ success	PR	✅ Active
Tools CI	tools-ci.yml	2026-03-31	✅ success	PR	✅ Active

E2E Testing — Current Generation

Workflow	Path	Last Run	Result	Trigger	Status
E2E Tests (pull request)	e2e-tests-ci.yml	2026-03-31	✅ success	dispatch	✅ Active
E2E Tests	e2e-fulltests-ci.yml	2026-03-31	🔄 running	dispatch	✅ Active
E2E Tests Check	e2e-tests-check.yml	2026-03-31	✅ success	PR	✅ Active
E2E Tests/verified	e2e-tests-verified-label.yml	2026-03-31	⏭️ skipped	PR	✅ Active
E2E Tests (master/release - merge)	e2e-tests-on-merge.yml	2026-03-31	✅ success	dispatch	✅ Active
E2E Tests (release cut)	e2e-tests-on-release.yml	2026-03-27	❌ failure	dispatch	⚠️ Failing
E2E Coverage Gate	e2e-coverage-gate.yml	2026-03-23	✅ success	PR	✅ Active

E2E Testing — Stale / Superseded

Workflow	Path	Last Run	Result	Status
E2E Tests Template	e2e-tests-ci-template.yml	2025-05-21	❌ failure	💀 10 months stale
E2E Tests (PR)	e2e-tests-pr.yml	2026-01-31	✅ success	💀 59 days stale
E2E Tests - Cypress Template	e2e-tests-cypress-template.yml	2026-01-31	❌ failure	💀 Stale + failing
E2E Tests - Playwright Template	e2e-tests-playwright-template.yml	2026-02-13	❌ failure	💀 Stale + failing
E2E Tests - Playwright	e2e-tests-playwright.yml	2026-01-08	❌ failure	💀 Stale + failing
E2E Tests - Cypress	e2e-tests-cypress.yml	—	—	💀 Never run
E2E Tests - Override Status	e2e-tests-override-status.yml	—	—	💀 Never run
[TM0] E2E Tests (on merge)	test-e2e-on-merge.yml	2026-02-08	❌ failure	💀 Stale + failing
[A3] E2E Tests (on release)	test-e2e-on-release.yml	2026-02-09	❌ cancelled	💀 Stale
[A2] E2E Tests (on merge - release)	test-e2e-on-merge-release.yml	2026-02-09	❌ cancelled	💀 Stale
[A1] E2E Tests (on merge - master)	test-e2e-on-merge-master.yml	2026-02-09	✅ success	💀 Stale
E2E Tests (Testcontainers)	e2e-tests-testcontainers.yml	2026-02-08	❌ failure	💀 Stale + failing
@mattermost/testcontainers	e2e-tests-testcontainers-check.yml	2026-02-20	✅ success	💀 39 days stale

E2E AI Agents (Experimental)

Workflow	Path	Last Run	Result	Status
e2e-ai-test-generation	e2e-ai-test-generation.yml	2026-02-18	❌ failure	💀 Stale + failing
e2e-test-execution	e2e-test-execution.yml	2026-02-18	❌ failure	💀 Stale + failing
E2E-Agents Impact Analysis	e2e-ai-impact-analysis.yml	2026-02-18	✅ success	💀 41 days stale
E2E-Agents Gap Analysis	e2e-ai-gap-check.yml	2026-02-18	✅ success	💀 41 days stale

Security & Supply Chain

Workflow	Path	Last Run	Result	Trigger	Status
CodeQL	codeql-analysis.yml	2026-03-31	✅ success	PR	✅ Active
Scorecards supply-chain security	scorecards-analysis.yml	2026-03-31	✅ success	schedule	✅ Active
Snyk SBOM	snyk-sbom.yml	2025-10-30	❌ failure	release	💀 5 months stale
Detect GHRC Typosquatting	detect-ghrc-typosquatting.yml	2025-08-26	✅ success	PR	💀 7 months stale

Infrastructure & Build

Workflow	Path	Last Run	Result	Trigger	Status
BuildEnv Docker Image	build-server-image.yml	2026-03-30	❌ failure	PR	⚠️ Failing
Opensearch Docker Image	build-opensearch-image.yml	2026-03-07	✅ success	push	✅ Active
Push mirrored docker images	docker-push-mirrored.yml	2026-01-20	✅ success	push	💀 70 days stale
Sentry Upload	sentry.yaml	2026-03-31	✅ success	workflow_run	✅ Active

AI Assistants & Code Review

Workflow	Path	Last Run	Result	Trigger	Status
Claude Code	claude.yml	2026-03-31	⏭️ skipped	issue_comment	✅ Active
Copilot code review	(dynamic)	2026-03-17	✅ success	dynamic	✅ Active
Copilot coding agent	(dynamic)	2025-11-12	❌ cancelled	dynamic	💀 4.5 months stale
Documentation Impact Review	docs-impact-review.yml	2026-03-31	❌ failure	PR	⚠️ Failing

Developer Docs

Workflow	Path	Last Run	Result	Trigger	Status
Developer Docs Build	dev-docs-build.yml	2026-03-06	✅ success	PR	✅ Active
Developer Docs Preview	dev-docs-preview-env.yml	2026-03-06	❌ failure	PR	⚠️ Failing
dev-docs-check-label-preview-env	dev-docs-check-label.yml	2026-03-05	⏭️ skipped	PR	✅ Active

PR Analysis & Testing

Workflow	Path	Last Run	Result	Trigger	Status
PR Test Analysis	pr-test-analysis.yml	2026-03-31	✅ success	PR	✅ Active
Tests/included	pr-test-analyzer.yml	2026-03-10	❌ cancelled	PR	✅ Active
Tests/verified	pr-test-analysis-verified.yml	2026-03-11	⏭️ skipped	PR	✅ Active
PR Test Analysis Override	pr-test-analysis-override.yml	2026-03-31	⏭️ skipped	issue_comment	✅ Active

Dependency Management & Misc

Workflow	Path	Last Run	Result	Trigger	Status
Dependabot Updates	(dynamic)	2026-03-30	✅ success	dynamic	✅ Active
Dependency Graph	(dynamic)	2026-03-27	✅ success	dynamic	✅ Active
Tag Public Module	tag-public-module.yaml	2026-03-10	✅ success	dispatch	❓ Manual
Lighthouse Metrics	lighthouse.yml	2025-12-16	✅ success	dispatch	💀 3.5 months stale
ESR Upgrade	esrupgrade.yml	2025-07-23	❌ failure	push	🔇 Disabled

Cleanup Recommendations

🔴 Priority 1 — Delete Immediately (stale, failing, never run)

These workflows provide zero value and clutter the Actions tab.

AWS Self-Hosted GitHub Runners

Mon, 01 Jan 0001 00:00:00 +0000

Evaluation of complementing GitHub-hosted runners with self-hosted runners on AWS to reduce CI queue times and costs.

Background

After enabling test sharding and fullyparallel mode, each merge to mattermost/mattermost spawns ~50+ concurrent jobs (4 test shards, 4 FIPS shards, 4 coverage shards, 40 Cypress E2E workers, 4 Playwright workers, plus lint/build/checks). Individual jobs complete faster (7-9 min vs 25-30 min), but peak concurrency is much higher, causing queue delays when multiple merges land close together.

Config Pending Restart / Reload API

Mon, 01 Jan 0001 00:00:00 +0000

Current Config Change Architecture

Config Store (`server/config/store.go`)

The Store struct wraps a backing store (file or database) and manages config state:

Set() (line 168) — Validates new config, persists via backing store, compares old/new, and if changed calls invokeConfigListeners(oldCfg, newCfg).
Load() (line 244) — Reads config from backing store, applies defaults/env overrides, invokes listeners if changed.

API Layer (`server/channels/api4/config.go`)

Routes registered in InitConfig() (line 33):

PUT /config → updateConfig (line 120) — Full config replacement
PUT /config/patch → patchConfig (line 279) — Partial update
POST /config/reload → configReload (line 100) — Already exists! Calls ReloadConfig() which calls configStore.Load()

Save Flow

API handler calls App.SaveConfig() → PlatformService.SaveConfig() (line 91 in server/channels/app/platform/config.go)
Plugin hook ConfigurationWillBeSaved fires
configStore.Set(newCfg) persists and invokes listeners
Cluster notification via clusterIFace.ConfigChanged()

Config Listener System

AddConfigListener(func(old, new *model.Config)) registers callbacks invoked on every config change. Key registrations:

Nick Misasi — Human–Agent Teaming with Cursor

Mon, 14 Sep 2026 00:00:00 +0000

Analysis of Nick Misasi’s writeup on 60 days of human–agent teaming using Cursor and Cursor Cloud Agents, posted to Private Core on September 14, 2026.

Core Philosophy

The human stays the architect, agents are the labor force. Nick plans, decides what’s done, merges, and owns what gets said publicly. Agents implement, unstick blocked work, and do first-pass review.

Toolchain

Tool	Role
Cursor IDE	Local “control plane” — planning, spawning agents, deciding when work is done
Cursor Cloud Agents	Remote sandboxed agents that clone repos and open PRs without touching the laptop
cursorctl	CLI to spawn/follow-up/inspect cloud agents programmatically (cursor-utils#1, ai-marketplace#26)
Cloud Agent Environments	Pre-configured remote workspaces with repo access, tools, credentials
GitHub Cursor Webhook	Triggers agents automatically from GitHub events (CI failure, review comment)

Key Workflow Patterns

1. Follow-up is the default, not a restart

When a cloud agent finishes a task, Nick sends it back to the same agent for the next step (fix CI, address review). 75% of his agents had 2+ runs. This preserves context — the agent already knows the code, the PR, the intent.

CodeRabbit: Full Repository Review

Mon, 01 Jan 0001 00:00:00 +0000

CodeRabbit only reviews PR diffs — there’s no “scan entire repo” button. But you can trick it by constructing a PR where every file appears as a new addition.

The Working Approach

# 1. Clone the repo
gh repo clone org/repo && cd repo

# 2. Create a branch that deletes everything
git checkout -b empty-base
git rm -rf .
git commit --allow-empty -m "empty base for full repo review"
git push origin empty-base

# 3. Branch off that, restore all files from main
git checkout -b restore-all-files
git checkout main -- .
git add -A
git commit -m "restore all files for review"
git push origin restore-all-files

# 4. Open PR: restore-all-files → empty-base
# (all files show as additions, shared history with main)
gh pr create --base empty-base --head restore-all-files \
 --title "chore: CodeRabbit full repository review" \
 --body "Temporary PR for full CodeRabbit analysis. Do not merge."

# 5. If CodeRabbit doesn't auto-trigger (non-default base branch),
# summon it manually:
gh api repos/org/repo/issues/NUMBER/comments \
 -f body="@coderabbitai full review"

Why This Works

empty-base branches off main (shared history → GitHub allows the PR)
Then deletes all files (empty tree)
restore-all-files branches off empty-base and restores everything from main
The PR diff: 0 deletions, all files as additions → CodeRabbit reviews everything

What Doesn’t Work

Approach	Why it fails
Delete-all → restore-all on same branch, PR into main	GitHub computes net diff = zero files changed
Orphan branch (no common history)	GitHub rejects: “no history in common”
Branch that deletes everything, PR into main	All files show as deletions — CodeRabbit ignores removed code
Trigger file + `@coderabbitai full review`	`full review` only reviews files in the PR diff, not the whole repo

After Review

Address findings via separate PRs on main
Close the review PR (don’t merge)
Delete both branches: empty-base and restore-all-files

Notes

CodeRabbit may not auto-trigger on PRs targeting non-default branches — use @coderabbitai full review to summon it
For large repos, CodeRabbit may cap at ~300 files per review
This is a one-time trick for onboarding repos; ongoing reviews happen naturally via normal PRs

First Used

April 2, 2026 on mattermost/terraform-provider-mattermost → PR #51
Found 11 issues across Go provider code, dependency security, and documentation
Took 5 attempts to get the git trick right (PRs #47–#51)

Salesforce Org History — How We Got Here

Mon, 01 Jan 0001 00:00:00 +0000

Chronological timeline of the Mattermost Salesforce org, reconstructed from user creation dates, trigger installation dates, validation rule creation dates, and Apex class metadata. Data sourced from partial sandbox (mattermost--partialsb).

2016 — Org Creation

May 2016: Steve Green (pSales profile) — first user, likely the founding sales rep
Aug 2016: Hanna Park joins as first admin/ops person
- Creates first validation rule: CLOSE_LOST_REQ_Loss_Reason on Opportunity

2017 — Early Growth

Mar–Jul 2017: Wave of Minimum Access users added (Niall McCarthy, Niamh Merritt, Nirmalya De, Omar Zaibak, Samiksha Khanna) — early engineering/product team getting read-only access
May 2017: Gavin Beeman — first sales hire still active today
Jul 2017: Hanna Park installs ClearBit (cbit) — first managed package, used for data enrichment. Adds triggers on Account, Contact, Lead
Aug 2017: Hanna Park creates NDA_Expiration_Date validation rule on Account

2018 — First Integrations

Apr 2018: Eric Mowery installs DiscoverOrg (DSCORGPKG) — B2B contact data enrichment. 6 triggers added to Account, Contact, Lead
Jul 2018: Marianne Lynch writes first custom Apex code — territory management triggers (FindTerritoryID, UpdateAccountTerritoryIDv1). These are still active today, 8 years later.

2019 — The Aorta Systems Era

A Salesforce consulting firm is brought in to build out the core sales operations workflow. This is the most consequential period in the org’s history.

Website Link Checking Scripts

Mon, 01 Jan 0001 00:00:00 +0000

Two Python scripts for validating links on the Mattermost website (production or staging). Both are async, use aiohttp, and are designed for CI or ad-hoc use.

1. Redirect Target Validator

Tests that all redirect target URLs in the explicit redirects CSV actually resolve (not 404). Pulls the CSV directly from GitHub via gh CLI, or accepts a local file.

Requirements: pip install aiohttp

#!/usr/bin/env python3
"""Test that all redirect target URLs in the explicit redirects CSV resolve (not 404).

Usage:
 python test-redirect-targets.py # staging (default)
 python test-redirect-targets.py --base https://mattermost.com # production
 python test-redirect-targets.py --csv path/to/local.csv # local CSV file

Requirements: pip install aiohttp
"""

import argparse
import asyncio
import csv
import io
import subprocess
import sys
from dataclasses import dataclass

import aiohttp

DEFAULT_BASE = "https://web.mattermost.workers.dev"
GH_CSV_PATH = "repos/mattermost/mattermost-website/contents/docs/cloudflare-redirects/1-explicit-redirects.csv"
CONCURRENCY = 5
TIMEOUT = 15
DELAY = 0.2


@dataclass
class Result:
 source: str
 target: str
 test_url: str
 status: int
 ok: bool


def fetch_csv_from_github() -> str:
 """Pull the CSV content via gh CLI."""
 result = subprocess.run(
 ["gh", "api", GH_CSV_PATH, "--jq", ".content"],
 capture_output=True, text=True, check=True,
 )
 import base64
 return base64.b64decode(result.stdout.strip()).decode()


def load_rows(csv_path: str | None) -> list[dict]:
 if csv_path:
 with open(csv_path) as f:
 return list(csv.DictReader(f))
 else:
 return list(csv.DictReader(io.StringIO(fetch_csv_from_github())))


def rewrite_url(target: str, base: str) -> str:
 if target.startswith("https://mattermost.com"):
 return target.replace("https://mattermost.com", base, 1)
 return target


async def check_url(
 sem: asyncio.Semaphore,
 session: aiohttp.ClientSession,
 source: str,
 target: str,
 base: str,
) -> Result:
 test_url = rewrite_url(target, base)
 async with sem:
 try:
 async with session.head(
 test_url,
 timeout=aiohttp.ClientTimeout(total=TIMEOUT),
 allow_redirects=False,
 ) as resp:
 status = resp.status
 except Exception:
 status = 0
 await asyncio.sleep(DELAY)
 ok = status in (200, 301, 302)
 return Result(source=source, target=target, test_url=test_url, status=status, ok=ok)


async def main(base: str, csv_path: str | None):
 rows = load_rows(csv_path)
 print(f"Testing {len(rows)} redirect targets against {base}")
 print(f"Concurrency: {CONCURRENCY} | Timeout: {TIMEOUT}s | Delay: {DELAY}s\n")

 sem = asyncio.Semaphore(CONCURRENCY)
 async with aiohttp.ClientSession() as session:
 tasks = [
 check_url(sem, session, row["source_url"], row["target_url"], base)
 for row in rows
 ]
 results = await asyncio.gather(*tasks)

 passed = [r for r in results if r.ok]
 failed = [r for r in results if not r.ok]

 print(f"=== RESULTS ===")
 print(f"✅ {len(passed)} ❌ {len(failed)} (out of {len(results)})\n")

 if failed:
 print("--- BROKEN TARGETS ---")
 for r in failed:
 print(f" {r.status} | {r.source} → {r.target}")
 print()
 sys.exit(1)
 else:
 print("All targets OK.")


if __name__ == "__main__":
 parser = argparse.ArgumentParser(description=__doc__)
 parser.add_argument("--base", default=DEFAULT_BASE, help="Base URL to test against")
 parser.add_argument("--csv", default=None, help="Local CSV path (skips GitHub fetch)")
 args = parser.parse_args()
 asyncio.run(main(args.base, args.csv))

Key details:

OpenClaw vs Cursor — Pavel vs Nick Workflow

Mon, 14 Sep 2026 00:00:00 +0000

Comparison of two human–agent teaming approaches used at Mattermost engineering, based on Nick Misasi’s 60-day Cursor writeup and Pavel Zeman’s OpenClaw setup.

The Two Setups

Pavel (OpenClaw + Claude Code): One persistent AI agent (Tommy) running on an EC2 instance, always on, with long-term memory, cron-based PR monitoring, and sub-agent dispatch for parallel work. Connected via SSH + webchat + Telegram.

Nick (Cursor + Cloud Agents): Cursor IDE as the control plane, spawning ephemeral cloud agent VMs per task. 100+ agents in 60 days, each isolated, disposable, and repo-scoped. Dispatch via cursorctl CLI and GitHub webhooks.

Celigo / NetSuite Connector Removal Plan

Mon, 01 Jan 0001 00:00:00 +0000

Plan to safely uninstall the two dormant Celigo managed packages from the Salesforce org. These packages have not been licensed since October 28, 2024 but their triggers still fire on every Account, Contact, and Opportunity update.

Background

Celigo (integrator.io) was the integration middleware managing two flows:

Salesforce ↔ NetSuite — the “SFV2-SFNSIO” connector, migrated to the integrator.io model in early 2023
BambooHR ↔ NetSuite — employee/department data sync

Both flows were made redundant by:

Parallel Test Global State Trade-offs

Fri, 27 Mar 2026 01:30:00 +0000

The Problem

When running Go tests with t.Parallel(), global mutable state becomes a race condition hazard. The Mattermost server codebase introduced TestOverrideLogRootPath, a global string variable used by tests to override the log root directory for security validation. This creates data races when multiple parallel tests modify the same global variable.

The immediate trigger for addressing this was Go 1.24’s new incompatibility between t.Setenv() and t.Parallel(), which forced us to find alternative approaches to environment variable manipulation in parallel tests.

Race-Safe Mock Assertions in Parallel Tests

Tue, 31 Mar 2026 14:50:00 +0000

This is v2 of the parallel test safety series. See also: Parallel Test Global State Trade-offs (v1).

The Problem

Go 1.25 ships a stricter race detector that catches a class of races invisible to Go 1.24. One critical pattern: testify mock assertions on arguments that reference live objects with background goroutines.

This pattern broke three tests on every Mattermost master build for 4+ days after the Go 1.24→1.25 upgrade (#35817):

TestReplicaLagQuery
TestInvalidReplicaLagDataSource
TestMetrics/ensure_advanced_metrics_have_database_metrics

The failures only appeared on master because PR CI does not run with -race — the race was invisible during the Go 1.25 PR review.

CI Queue Time Report — April 10, 2026

Fri, 10 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (push to master — primary, 14 jobs on 8-core)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	49	10.3 GB	10 GB	⚠️ 103%
`mattermost/mattermost-mobile`	14	5.9 GB	10 GB	✅ 59%

Mobile was at 10.0 GB earlier today; manual cleanup of tag caches and stale build branch caches freed ~4 GB.

CI Queue Time Report — April 13, 2026

Mon, 13 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (push to master — primary, 14 jobs on 8-core)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	45	10.5 GB	10 GB	⚠️ 105%
`mattermost/mattermost-mobile`	18	6.0 GB	10 GB	✅ 60%

The monorepo remains over capacity (105%, up from 103% on Apr 10). Mobile is stable at ~60%.

CI Queue Time Report — April 14, 2026

Tue, 14 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (push to master — primary, 14 jobs on 8-core)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	36	9.8 GB	10 GB	⚠️ 98%
`mattermost/mattermost-mobile`	22	6.1 GB	10 GB	✅ 61%

The monorepo dropped from 45 caches (105%) on Apr 13 to 36 caches (98%) — back under the 10 GB limit for the first time in weeks. Likely due to cache eviction catching up. Mobile stable at ~61%.

CI Queue Time Report — April 15, 2026

Wed, 15 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (push to master — primary, 14 jobs on 8-core)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	63	9.9 GB	10 GB	⚠️ 99%
`mattermost/mattermost-mobile`	24	6.1 GB	10 GB	✅ 61%

The monorepo surged from 36 caches (98%) yesterday to 63 caches (99%) today — nearly doubling the cache count while staying just under the 10 GB limit. This is the highest cache count we’ve observed. Mobile remains stable at ~61%.

CI Queue Time Report — April 16, 2026

Thu, 16 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, 14 jobs on 8-core when fully enabled)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	52	9.9 GB	10 GB	⚠️ 99%
`mattermost/mattermost-mobile`	42	10.1 GB	10 GB	🔴 101%

The monorepo dropped from 63 caches yesterday to 52 caches today while staying at 99% utilization — cache eviction is working. Mobile has crossed the 10 GB limit for the first time, surging from 24 caches (6.1 GB, 61%) yesterday to 42 caches (10.1 GB, 101%) today. This is a significant spike that likely triggers aggressive eviction.

CI Queue Time Report — April 17, 2026

Fri, 17 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, 14 jobs on 8-core when fully enabled)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	43	9.9 GB	10 GB	🔴 99%
`mattermost/mattermost-mobile`	32	9.6 GB	10 GB	⚠️ 96%

The monorepo dropped from 52 caches yesterday to 43 caches today, continuing the downward trend (63 → 52 → 43), while utilization holds steady at 99%. Mobile pulled back from 101% to 96% (42 → 32 caches, 10.1 → 9.6 GB) — eviction cleaned up the overshoot from yesterday.

CI Queue Time Report — April 20, 2026

Mon, 20 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, 14 jobs on 8-core when fully enabled)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	59	9.7 GB	10 GB	🔴 97%
`mattermost/mattermost-mobile`	26	9.7 GB	10 GB	🔴 97%

The monorepo cache count jumped from 43 (Apr 17) to 59 caches today — a significant increase likely driven by weekend branch activity. Utilization remains high at 97%. Mobile dropped further to 26 caches (from 32 on Apr 17) while staying at 97% utilization — fewer caches but each is larger on average.

CI Queue Time Report — April 21, 2026

Tue, 21 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, 14 jobs on 8-core when fully enabled)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	42	10.0 GB	10 GB	🔴 100%
`mattermost/mattermost-mobile`	28	9.7 GB	10 GB	🔴 97%

Monorepo cache count dropped from 59 (Apr 20) to 42 caches — likely eviction pressure from hitting the 10 GB limit. Utilization has reached 100%, up from 97% yesterday. Mobile slightly increased to 28 caches (from 26) while staying at 97%.

CI Queue Time Report — April 22, 2026

Wed, 22 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, 14 jobs on 8-core when fully enabled)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	54	10.2 GB	10 GB	🔴 102%
`mattermost/mattermost-mobile`	32	9.5 GB	10 GB	🟡 95%

Monorepo cache count jumped from 42 (Apr 21) to 54 caches, and size has exceeded the nominal 10 GB limit at 10.2 GB (10,947 MB). GitHub allows brief overages before enforcing eviction. Mobile increased to 32 caches (from 28) while dropping slightly to 95%.

CI Queue Time Report — April 23, 2026

Thu, 23 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, 14 jobs on 8-core when fully enabled)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	53	10.2 GB	10 GB	🔴 102%
`mattermost/mattermost-mobile`	36	9.5 GB	10 GB	🟡 95%

Monorepo cache count dropped slightly from 54 (Apr 22) to 53, while size remains at 10.2 GB (10,967 MB) — still over the nominal 10 GB limit. Mobile jumped from 32 to 36 caches while size held at 9.5 GB (10,264 MB).

CI Queue Time Report — April 24, 2026

Fri, 24 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, 14 jobs on 8-core when fully enabled)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	42	9.7 GB	10 GB	🟡 97%
`mattermost/mattermost-mobile`	28	10.0 GB	10 GB	🔴 100%

Monorepo dropped significantly from 53 caches (Apr 23) to 42, and size fell from 10.2 GB to 9.7 GB (10,425 MB) — finally back under the 10 GB limit. Mobile went the opposite direction: down from 36 to 28 caches but size stayed at 10.0 GB (10,726 MB), now hitting the limit.

CI Queue Time Report — April 27, 2026

Mon, 27 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, 14 jobs on 8-core when fully enabled)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	46	10.3 GB	10 GB	🔴 103%
`mattermost/mattermost-mobile`	29	9.9 GB	10 GB	🟡 99%

Monorepo bounced back over the limit — up from 42 caches / 9.7 GB (Apr 24) to 46 caches / 10.3 GB (11,103 MB). The weekend’s Server CI activity on multiple branches (MM-68196, MM-68457, remove-property-fields-protected-index) created new caches that pushed it over. Mobile held steady at 29 caches (was 28) and 9.9 GB (10,579 MB), slightly down from the 10.0 GB limit hit on Apr 24.

CI Queue Time Report — April 28, 2026

Tue, 28 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 9 jobs on 8-core with coverage shards)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	45	10.4 GB	10 GB	🔴 104%
`mattermost/mattermost-mobile`	30	9.5 GB	10 GB	🟡 95%

Monorepo remains over the 10 GB limit — 45 caches / 10.4 GB (10,351 MB), down 1 cache from yesterday’s 46 but size essentially unchanged. Mobile improved slightly: 30 caches (was 29) at 9.5 GB (9,476 MB), down from 9.9 GB on Apr 27.

CI Queue Time Report — April 29, 2026

Wed, 29 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 14 jobs on 8-core with all shards + FIPS)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	44	10.3 GB	10 GB	🔴 103%
`mattermost/mattermost-mobile`	34	9.0 GB	10 GB	🟡 90%

Monorepo: 44 caches / 10.3 GB (11,083 MB), down 1 cache from yesterday’s 45 but still over the 10 GB limit. Mobile improved significantly: 34 caches (was 30) at 9.0 GB (9,617 MB), down from 9.5 GB yesterday — heading in the right direction.

CI Queue Time Report — April 30, 2026

Thu, 30 Apr 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 14 jobs on 8-core with all shards + FIPS)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	40	9.7 GB	10 GB	🟡 97%
`mattermost/mattermost-mobile`	38	10.0 GB	10 GB	🔴 100%

Monorepo: 40 caches / 9.7 GB (10,422 MB), down from 44 caches / 10.3 GB yesterday — finally dropped below the 10 GB limit. Mobile reversed course: 38 caches at 10.0 GB (10,777 MB), up from 34 caches / 9.0 GB yesterday — now at the limit.

CI Queue Time Report — May 1, 2026

Fri, 01 May 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 14 jobs on 8-core with all shards + FIPS)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	54	9.8 GB	10 GB	🟡 98%
`mattermost/mattermost-mobile`	39	9.8 GB	10 GB	🟡 98%

Monorepo jumped from 40→54 caches and from 97%→98% utilization — back near the limit after yesterday’s dip below 10 GB. Mobile eased slightly: 39 caches at 9.8 GB (down from 38 caches / 10.0 GB yesterday). Both repos now converge at ~98%.

CI Queue Time Report — May 4, 2026

Mon, 04 May 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 14 jobs on 8-core with all shards + FIPS)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	46	9.9 GB	10 GB	🔴 99%
`mattermost/mattermost-mobile`	34	9.7 GB	10 GB	🟡 97%

Monorepo dropped from 54→46 caches but size crept up to 9.9 GB (99%), the highest observed level. Mobile continued its decline: 39→34 caches, 9.8→9.7 GB. Both repos remain under pressure but mobile is trending slightly better.

CI Queue Time Report — May 5, 2026

Tue, 05 May 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 14 jobs on 8-core with all shards + FIPS)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	36	9.8 GB	10 GB	🔴 98%
`mattermost/mattermost-mobile`	28	9.9 GB	10 GB	🔴 99%

Monorepo dropped from 46→36 caches and 9.9→9.8 GB — modest improvement, still critically high. Mobile reversed its prior trend: 34→28 caches but size increased to 9.9 GB (99%), surpassing the monorepo for the first time. Both repos remain under extreme pressure.

CI Queue Time Report — May 6, 2026

Wed, 06 May 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 14 jobs on 8-core with all shards + FIPS)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	41	9.8 GB	10 GB	🔴 98%
`mattermost/mattermost-mobile`	32	9.8 GB	10 GB	🔴 98%

Both repos remain at critical 98% utilization. Monorepo caches ticked up from 36→41, while mobile dropped from 32 caches at 9.8 GB — down slightly from yesterday’s 99%. Still deep in the danger zone.

CI Queue Time Report — May 7, 2026

Thu, 07 May 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 14 jobs on 8-core with all shards + FIPS)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	36	10.0 GB	10 GB	🔴 100%
`mattermost/mattermost-mobile`	32	9.8 GB	10 GB	🔴 98%

Monorepo hit the full 10 GB ceiling — up from 9.8 GB / 41 caches yesterday to 10.0 GB / 36 caches today. Fewer caches but larger average size (278 MB vs 239 MB). LRU eviction is actively culling smaller caches while bigger ones persist. Mobile is stable at 32 caches / 9.8 GB.

CI Queue Time Report — May 8, 2026

Fri, 08 May 2026 00:00:00 +0000

Daily report on GitHub Actions job queue delays and cache utilization for mattermost/mattermost.

How This Report Was Generated

Data collected via GitHub Actions REST API, querying jobs from three Server CI workflows:

# Queue times: for each of the last 5 completed runs per workflow,
# compute started_at - created_at per job
export GH_TOKEN=$(cat .tokens/github-mattermost)
for workflow_id in 56372552 56742408 56627862; do
 # 56372552 = Server CI (primary, up to 14 jobs on 8-core with all shards + FIPS)
 # 56742408 = Server CI Master (reporting/coverage)
 # 56627862 = Server CI PR (pull request CI)
 for run_id in $(curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/workflows/${workflow_id}/runs?per_page=5&status=completed" \
 | jq -r '.workflow_runs[].id'); do
 sleep 1
 curl -s -H "Authorization: Bearer $GH_TOKEN" \
 -H "Accept: application/vnd.github+json" \
 "https://api.github.com/repos/mattermost/mattermost/actions/runs/${run_id}/jobs?per_page=100" \
 | jq '.jobs[] | {name, started_at, created_at, labels,
 queue_seconds: (((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601)))}'
 done
done

# Cache usage
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost/actions/cache/usage"
curl -s -H "Authorization: Bearer $GH_TOKEN" \
 "https://api.github.com/repos/mattermost/mattermost-mobile/actions/cache/usage"

Cache Utilization

Repository	Caches	Size	Limit	Utilization
`mattermost/mattermost`	36	9.8 GB	10 GB	🔴 98%
`mattermost/mattermost-mobile`	31	9.9 GB	10 GB	🔴 99%

Monorepo dropped slightly from yesterday’s 100% ceiling (10.0 GB / 36 caches) to 9.8 GB / 36 caches — LRU eviction freed ~200 MB while maintaining the same cache count. Mobile ticked up from 9.8 GB / 32 caches to 9.9 GB / 31 caches, approaching the ceiling. Both repos remain critically close to the 10 GB limit.

Senior Full-Stack Engineer — new in top 50

Thu, 22 Oct 2026 00:00:00 +0000

2026-10-22 · first seen since 2026-10-16 09:21 · rank is within the role’s overall top 50

#1 ★NEW · Kevin Smyth · Austin, Texas, United States · Staff Software Engineer · Turo · score: 0.852

Greenhouse · LinkedIn | AI: 0.07 | GH fraud: weak (+4 weak: country mismatch, proxy, bot-like IP, new email) | ⚠ Potential duplicate

This candidate shows genuine technical depth in exactly the stack Mattermost needs: Go/TypeScript full-stack work on a production ChatGPT App integration, PostgreSQL query optimization with measured before/after latency, and REST/WebSocket API design at meaningful scale (4,200 concurrent sessions, 300K-vehicle inventory). The AI-agent answer is concrete and credible — daily Claude Code/Cursor use for test scaffolding and refactor candidates, reviewed in weekly code review, with a quantified zero-regression claim and evidence of mentoring others on the workflow, which directly matches the role’s ‘core required skill’ bar rather than just checking a box. Career trajectory (Barclays -> Tailor Brands -> WeWork -> Turo, Staff-level) shows steady seniority growth and some enterprise/compliance exposure (encryption at rest/in transit, SOC 2 mentioned on resume), which is a plus for the regulated-environment nice-to-have. Two things worth flagging before an outreach: the stated ‘8+ years’ self-reported experience doesn’t reconcile with a resume spanning 2011-present (13+ years) and a resume summary claiming ‘15+ years’ — worth clarifying in a screen, as it’s a moderate inconsistency rather than a disqualifier. More importantly, there’s no evidence in the application of US government clearance eligibility, ITAR/EAR status, or defense/intelligence domain exposure, all of which are hard requirements here — this should be confirmed early since it could be a gating issue regardless of technical fit. Net: strong technical and AI-workflow match, recommend a screen focused on clearance/citizenship eligibility and experience-timeline clarification.

Senior Full-Stack Engineer — top 50

Thu, 22 Oct 2026 00:00:00 +0000

2026-10-22 · ranked by composite score · ★NEW = first seen since 2026-10-16 09:21

#1 ★NEW · Kevin Smyth · Austin, Texas, United States · Staff Software Engineer · Turo · score: 0.852

Greenhouse · LinkedIn | AI: 0.07 | GH fraud: weak (+4 weak: country mismatch, proxy, bot-like IP, new email) | ⚠ Potential duplicate

Senior Mobile Engineer — new in top 50

Thu, 22 Oct 2026 00:00:00 +0000

2026-10-22 · first seen since 2026-10-16 09:21 · rank is within the role’s overall top 50

#8 ★NEW · James Jackson · Miami, Florida, United States · Senior Staff Software Engineer · FranShares · score: 0.777

Greenhouse | AI: 0.22 | GH fraud: strong — VOIP phone (+5 weak: country mismatch, proxy, bot-like IP, IP history, new email) | ⚠ Potential duplicate

This candidate has strong, well-evidenced mobile reliability experience: offline-capable workflows with SQLite persistence, idempotent background sync, exponential backoff/retry logic, and native integration work (Keychain/Keystore, push notifications) in a React Native + TypeScript codebase at FranShares, plus a long prior tenure at iRobot suggesting real embedded/mobile systems depth. The profiling detail (Instruments, Android Perfetto, cold-start optimization) and testing discipline (XCTest, Espresso) match the role’s emphasis on production debugging and DDIL-style reliability work closely.

Senior Mobile Engineer — top 50

Thu, 22 Oct 2026 00:00:00 +0000

2026-10-22 · ranked by composite score · ★NEW = first seen since 2026-10-16 09:21

#1 · Joshua Lytle · Issaquah, Washington, United States · Principal Software Engineer · NomadGo, Inc. · score: 0.871

Greenhouse · LinkedIn | AI: 0.14 | GH fraud: strong — datacenter IP (+3 weak: proxy, bot-like IP, IP history)

This is a strong technical match: the candidate has shipped iOS since 2008 and cross-platform (including React Native SDK work) since 2011, with recent hands-on experience building connectivity-loss handling (startup, sign-in, count submission) and MDM/managed-app-configuration support for store devices at NomadGo — essentially a dry run for the DDIL and enterprise-distribution requirements in this role. They also show real performance engineering chops (adaptive inference throttling and OCR interval gating to cut AR scanner battery drain) and testing/CI discipline (1,000 unit tests, 100 automated UI tests, built partly with agentic tooling), plus prior SDK work integrated by major retail engineering teams (Target, Lowe’s, Staples, Macy’s), evidencing experience shipping code others depend on.

Staff Full Stack Engineer — new in top 50

Thu, 22 Oct 2026 00:00:00 +0000

2026-10-22 · first seen since 2026-10-16 09:21 · rank is within the role’s overall top 50

#1 ★NEW · Pierre Cabell · Cuyahoga Falls, Ohio, United States · Staff Full-Stack Engineer · UVeye · score: 0.889

Greenhouse · LinkedIn | AI: 0.12 | GH fraud: strong — datacenter IP, VOIP phone (+5 weak: TZ mismatch, proxy, bot-like IP, IP history, new email)

This candidate is a strong match on paper: current Staff title at UVeye doing exactly the described work — Go/React/TypeScript/PostgreSQL/Redis, architecture ownership across four product teams, a real-time WebSocket layer with explicit reconnect/ordering/backpressure design, and a documented testing overhaul (48min to 14min pipeline). The AI-agent story is specific and credible rather than buzzword-y: shared Claude Code skills, MCP servers for internal APIs, a CI review agent adopted org-wide, and explicit written guardrails for where agents shouldn’t touch auth/payments/security — this is genuinely above-average AI-tooling maturity for a staff-level hire. Mentoring and informal-influence answers are concrete (named outcomes: one engineer now leads API redesign, another runs a testing guild) rather than vague self-praise, and the regulated-environment experience (ITAR, FedRAMP-aligned controls, air-gapped military deployment) is a near-exact match to the ‘regulated/high-compliance’ and EAR/ITAR requirements.

Staff Full Stack Engineer — top 50

Thu, 22 Oct 2026 00:00:00 +0000

2026-10-22 · ranked by composite score · ★NEW = first seen since 2026-10-16 09:21

#1 ★NEW · Pierre Cabell · Cuyahoga Falls, Ohio, United States · Staff Full-Stack Engineer · UVeye · score: 0.889

Greenhouse · LinkedIn | AI: 0.12 | GH fraud: strong — datacenter IP, VOIP phone (+5 weak: TZ mismatch, proxy, bot-like IP, IP history, new email)

Senior Full-Stack Engineer — new in top 50

Fri, 16 Oct 2026 00:00:00 +0000

2026-10-16 · first seen since 2026-10-13 09:19 · rank is within the role’s overall top 50

No candidates.

Senior Full-Stack Engineer — top 50

Fri, 16 Oct 2026 00:00:00 +0000

2026-10-16 · ranked by composite score · ★NEW = first seen since 2026-10-13 09:19

#1 · Kevin Wathen · Hilliard, Florida, United States · Senior Software Engineer · Airbnb · score: 0.843

Greenhouse · LinkedIn | AI: 0.13 | GH fraud: strong — datacenter IP (+4 weak: proxy, bot-like IP, IP history, new email) | ⚠ Potential duplicate

This candidate presents a strong technical match for the role: current hands-on experience with Go, React/TypeScript, PostgreSQL, WebSockets, and Redis, plus explicit daily use of Claude Code and Cursor for test generation, refactoring, and even building custom MCP servers for agentic workflows — directly matching the ‘AI-forward’ requirement rather than just checking a box. The resume also shows relevant enterprise/security-adjacent work (tenant isolation, role-based access control, compliance hardening) which aligns well with Mattermost’s regulated-environment context, even though there’s no explicit defense/intelligence domain experience or mention of security clearance status, which should be confirmed directly.

Senior Mobile Engineer — new in top 50

Fri, 16 Oct 2026 00:00:00 +0000

2026-10-16 · first seen since 2026-10-13 09:19 · rank is within the role’s overall top 50

#26 ★NEW · Hammoud Hammoud · Palos Verdes Peninsula, California, United States · Senior Mobile Engineer (Contract) · Honda · score: 0.684

Greenhouse · LinkedIn | AI: 0.12 | GH fraud: ok

This candidate has strong, well-evidenced native mobile depth: six years on Tandem’s t:slim app building and maintaining a BLE communication layer for a Bluetooth medical device, diagnosing race conditions in a concurrent upload pipeline and rearchitecting it around a synchronized operation queue, using Crashlytics/Datadog and Xcode Instruments for production debugging, and driving unit test coverage from under 40% to 75-80% across a 16-person team. The current Honda contract adds more BLE/offline-connectivity work (an internet-free camper control app) and a concrete, specific example of agentic AI coding workflow — using Jira tickets, Figma specs, and a BLE protocol doc as agent context, shipped as small stacked PRs — which is a credible, non-generic AI usage story.

Senior Mobile Engineer — top 50

Fri, 16 Oct 2026 00:00:00 +0000

2026-10-16 · ranked by composite score · ★NEW = first seen since 2026-10-13 09:19

#1 · Joshua Lytle · Issaquah, Washington, United States · Principal Software Engineer · NomadGo, Inc. · score: 0.871

Greenhouse · LinkedIn | AI: 0.14 | GH fraud: strong — datacenter IP (+3 weak: proxy, bot-like IP, IP history)

Staff Full Stack Engineer — new in top 50

Fri, 16 Oct 2026 00:00:00 +0000

2026-10-16 · first seen since 2026-10-13 09:19 · rank is within the role’s overall top 50

#16 ★NEW · Preston Matheson · Portland, Oregon, United States · Lead Software Engineer · Lightspeed Systems · score: 0.738

Greenhouse · LinkedIn | AI: 0.20 | GH fraud: weak (+1 weak: new email)

This candidate shows strong staff-level technical substance: production Go and React/TypeScript experience, distributed systems work at meaningful scale (tens of millions of requests/day, 6.67M devices), and a track record of leading architecture decisions like unifying a split data model used across five services and redesigning a hot-path policy resolution system. Compliance-adjacent experience is present and relevant — a DoD contract enabled by their feature work and resolving MOE audit/security findings in Singapore — which maps reasonably well to the regulated-environment requirement, though explicit clearance/ITAR eligibility isn’t confirmed in the text. Mentoring and informal-authority influence are well evidenced: leading a team of all-senior engineers, mentoring a staff engineer, and driving org-wide adoption of agentic AI workflows against initial resistance, with a structured mechanism (weekly show-and-tell) for spreading practices — this is a genuinely strong signal for the ‘AI-agentic standards’ and ‘influence without authority’ requirements.

Staff Full Stack Engineer — top 50

Fri, 16 Oct 2026 00:00:00 +0000

2026-10-16 · ranked by composite score · ★NEW = first seen since 2026-10-13 09:19

#1 · Jonathan Pyers · Troutman, North Carolina, United States · Solution Architect and Team Lead - Senior Software Engineer, Full-Stack · RTS Labs · score: 0.859

Greenhouse · LinkedIn | AI: 0.07 | GH fraud: ok

This is a strong match on paper: genuine Go + React/TypeScript production experience, PostgreSQL, a track record of end-to-end ownership (solution architect scoping and delivering client systems, MCP server for company-wide estimating), and real regulated-environment exposure — five years at Lawrence Livermore National Laboratory supporting mission-critical National Ignition Facility operations, plus HHS government consulting at Ad Hoc. The LLNL background strongly suggests clearance/ITAR eligibility even though not explicitly stated, which directly addresses a required qualification many candidates can’t meet. The AI-agent fluency signal is unusually credible — they didn’t just name a tool, they built their own agent harness (public repo) and actively maintain standards artifacts for agents across projects, which is close to the ‘org-wide AI-agentic engineering standards’ this role wants someone to establish.

Senior Full-Stack Engineer — new in top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · first seen since 2026-10-01 00:00 · rank is within the role’s overall top 50

#8 ★NEW · David Mullins · Durango, Colorado, United States · Data Engineer · Mill Creek Renewables (EPC) · score: 0.785

Greenhouse | AI: 0.04 | GH fraud: ok

This is a strong full-stack engineer with genuine depth: schema design, query/index tuning, dimensional modeling, CI-gated testing, and a track record of owning systems end-to-end (the Mill Creek ETL/orchestrator rebuild scaling from 5 to 30+ projects is a concrete, verifiable outcome). The AI-agent answer is a clear positive signal — it describes a specific failure mode (plausible-looking wrong mappings) and a specific mitigation (tests against real data plus human sign-off), which is more substantive than most applicants offer. The defense/export-control networking background (5G, DOCSIS, precision timing, Lockheed/Qualcomm lab work) is a genuine nice-to-have match for Mattermost’s defense/intel customer base and suggests clearance eligibility, though it isn’t stated explicitly.

Senior Full-Stack Engineer — new in top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · first seen since 2026-10-12 00:00 · rank is within the role’s overall top 50

#8 ★NEW · David Mullins · Durango, Colorado, United States · Data Engineer · Mill Creek Renewables (EPC) · score: 0.785

Greenhouse | AI: 0.04 | GH fraud: ok

Senior Full-Stack Engineer — top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · ranked by composite score · ★NEW = first seen since 2026-10-01 00:00

#1 · Kevin Wathen · Hilliard, Florida, United States · Senior Software Engineer · Airbnb · score: 0.843

Greenhouse · LinkedIn | AI: 0.13 | GH fraud: strong — datacenter IP (+4 weak: proxy, bot-like IP, IP history, new email) | ⚠ Potential duplicate

Senior Full-Stack Engineer — top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · ranked by composite score · ★NEW = first seen since 2026-10-12 00:00

#1 · Kevin Wathen · Hilliard, Florida, United States · Senior Software Engineer · Airbnb · score: 0.843

Greenhouse · LinkedIn | AI: 0.13 | GH fraud: strong — datacenter IP (+4 weak: proxy, bot-like IP, IP history, new email) | ⚠ Potential duplicate

Senior Mobile Engineer — new in top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · first seen since 2026-10-01 00:00 · rank is within the role’s overall top 50

#1 ★NEW · Joshua Lytle · Issaquah, Washington, United States · Principal Software Engineer · NomadGo, Inc. · score: 0.871

Greenhouse · LinkedIn | AI: 0.14 | GH fraud: strong — datacenter IP (+3 weak: proxy, bot-like IP, IP history)

Senior Mobile Engineer — new in top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · first seen since 2026-10-12 00:00 · rank is within the role’s overall top 50

#3 ★NEW · William Mccaffrey · Tampa, Florida, United States · Senior Software Engineer, Technical Lead · JetBlue · score: 0.853

Greenhouse · LinkedIn | AI: 0.12 | GH fraud: strong — VOIP phone (+2 weak: TZ mismatch, new email) | ⚠ Potential duplicate

This candidate is a strong technical match: current React Native/TypeScript work at JetBlue includes an offline-first data layer with queued writes, idempotent sync, and conflict handling built specifically to survive intermittent connectivity and app restarts — essentially a dry run for DDIL requirements. They also cite native-layer profiling (Instruments/Android profilers) for startup and jank fixes, crash-analytics-driven production debugging, and networking resilience work (retry/backoff, lifecycle-aware background tasks), all of which map closely to the role’s required skills. Skills list also name-checks MDM/AppConfig/Intune MAM, Keychain/Keystore secure storage, and AI coding agent usage (Cursor, Copilot, Claude Code) with some specificity around LLM-assisted debugging and test generation.

Senior Mobile Engineer — top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · ranked by composite score · ★NEW = first seen since 2026-10-01 00:00

#1 ★NEW · Joshua Lytle · Issaquah, Washington, United States · Principal Software Engineer · NomadGo, Inc. · score: 0.871

Greenhouse · LinkedIn | AI: 0.14 | GH fraud: strong — datacenter IP (+3 weak: proxy, bot-like IP, IP history)

Senior Mobile Engineer — top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · ranked by composite score · ★NEW = first seen since 2026-10-12 00:00

#1 · Joshua Lytle · Issaquah, Washington, United States · Principal Software Engineer · NomadGo, Inc. · score: 0.871

Greenhouse · LinkedIn | AI: 0.14 | GH fraud: strong — datacenter IP (+3 weak: proxy, bot-like IP, IP history)

Staff Full Stack Engineer — new in top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · first seen since 2026-10-01 00:00 · rank is within the role’s overall top 50

#1 ★NEW · Jonathan Pyers · Troutman, North Carolina, United States · Solution Architect and Team Lead - Senior Software Engineer, Full-Stack · RTS Labs · score: 0.859

Greenhouse · LinkedIn | AI: 0.07 | GH fraud: ok

Staff Full Stack Engineer — new in top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · first seen since 2026-10-12 00:00 · rank is within the role’s overall top 50

#1 ★NEW · Jonathan Pyers · Troutman, North Carolina, United States · Solution Architect and Team Lead - Senior Software Engineer, Full-Stack · RTS Labs · score: 0.859

Greenhouse · LinkedIn | AI: 0.07 | GH fraud: ok

Staff Full Stack Engineer — top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · ranked by composite score · ★NEW = first seen since 2026-10-01 00:00

#1 ★NEW · Jonathan Pyers · Troutman, North Carolina, United States · Solution Architect and Team Lead - Senior Software Engineer, Full-Stack · RTS Labs · score: 0.859

Greenhouse · LinkedIn | AI: 0.07 | GH fraud: ok

Staff Full Stack Engineer — top 50

Wed, 14 Oct 2026 00:00:00 +0000

2026-10-14 · ranked by composite score · ★NEW = first seen since 2026-10-12 00:00

#1 ★NEW · Jonathan Pyers · Troutman, North Carolina, United States · Solution Architect and Team Lead - Senior Software Engineer, Full-Stack · RTS Labs · score: 0.859

Greenhouse · LinkedIn | AI: 0.07 | GH fraud: ok

Mattermost Engineering Knowledge Base

PR Status Tracker

Sentry Fix PRs

Paid Search Attribution: Findings and Best Practices

1. Scope

From 62 Minutes to 14: How We Sharded Our Go Test Suite

Server Test Coverage Strategy

Current State

Website Forms and Tracking Audit

1. Summary

Workstream Tracker

1. Sentry Issue Triage & Fix

Summary (as of 2026-03-20)

Biggest wins by user impact

Next Steps

2. CI Improvements

Marketo and Salesforce Configuration Issues

CI Coverage Persistence — Current State & Options

Where We Are

The Numbers Problem

CI Pipeline Analysis v2: mattermost/mattermost

Progress Summary

What Shipped

New Issues Discovered

1. Workflow Inventory

PR Daily Status

2026-10-30 (Friday)

What Not to Do: Writing Go Tests That Survive Parallel Execution

Salesforce UTM Campaigns Over the Years

CI Pipeline Analysis: mattermost/mattermost

1. Workflow Inventory

2. Job Catalog

2.1 Server CI (server-ci.yml) — The Critical Path

Code Review Checklist

Before Writing Code

Production Code

Tests

CI / Shell Scripts

PR Submission

Proposed Plan: Rebooting Advertising, Lead Capture and Attribution

1. Why we are doing this

Server Test Sharding Analysis

Context: Why Are We Back at 60 Minutes?

mmworkbrain Workflow — Full Setup Guide

Overview

Code Coverage Investment Strategy

Current State

The Big Picture

Go 1.25 Race Condition: Deep Analysis

Timeline

Security Issue: GitHub Actions persist-credentials

Summary

mm-certification-platform: Security Findings Response

Summary

Mobile Observability Audit & Strategy v2

What Changed Since v1

Mobile Observability Audit & Strategy v3

What Changed Since v2

1. Who Built the Homegrown Telemetry Infrastructure

Large Scope PRs

Executive Summary

iOS SQLite Background Crash Analysis (0xdead10cc)

Summary

The Crash Mechanism: 0xdead10cc

GitHub Actions Cache & Queue Time Analysis

Executive Summary

Cache Analysis

mattermost/mattermost

Playbooks Improvements POC — PR #2222 Analysis

Size

Component Breakdown

Key Feature Areas

Hotspots (largest files)

Test Coverage

Wiki/Pages POC — PR #34607 Analysis

Size

Component Breakdown

ARM64 Docker Image Feasibility Analysis

Executive Summary

Team Email API Behavior Analysis

2.1 Server CI (`server-ci.yml`) — The Critical Path

The Crash Mechanism: `0xdead10cc`

🔴 Critical: Jest runs with `--runInBand` (6m 57s)

Active Projects (skip `-test` projects)