ZeroHedge — Comment Platform Migration at 40M+ Document Scale

Overview

ZeroHedge is a financial news site with a large comment archive. The comments collection in this migration held 40.8M documents, about 41.5 GB.

I owned the migration of the platform from a customized CoralTalk v4 installation to v7: investigation, migration tooling, infrastructure, rollout strategy, and the supporting delivery systems.

Scope of engagement

Full lifecycle ownership: migration planning, database export and schema-conversion tooling, Helm charts for dev and production, frontend integration and theming, feature-flag rollout planning, and payment/auth library upgrades along the way.

What I owned

• Wrote and tested the database export and schema-conversion pipeline for the CoralTalk v4-to-v7 migration of a 40.8M-document collection (~41.5GB).
• Drove an index-bloat fix during re-import that cut index size by more than half — importing data first and rebuilding indexes after, instead of paying the write-amplification cost.
• Built the Helm charts for both development and production-optimized deployments of the new platform on GKE.
• Designed the rollout around a feature flag to support controlled enablement and rollback.
• Upgraded the Stripe and Firebase integration libraries the community platform depends on, keeping subscriptions and auth working through the migration.
• Proposed and built ephemeral per-PR preview environments: a pull request gets its own namespace and an automatically posted preview URL, with cleanup on merge or close.

Caching for live market data

Separately from the migration, I recommended a caching approach for a live gold-price widget: Varnish in front of the API to cache the upstream price response for 15 seconds and reduce application requests. The reasoning came directly from operating high-traffic Next.js under load — the right tool placed at the right layer.

Operational care

Migration at this scale is mostly about not breaking things quietly: the plan included database copies validated in dev and prod, avatar handling moved onto owned infrastructure to reduce dependence on external URLs, and explicit client sign-off requested on the fork-versus-upstream decision because it changes long-term maintenance economics.

Outcome

Delivered the migration tooling, development and production Helm charts, frontend integration, and feature-flag rollout plan for CoralTalk v4 to v7. Drove the index rebuild from 60.8 GB to 28.4 GB, upgraded Stripe and Firebase libraries, and built per-PR previews for the delivery workflow.