Upgrade 1.6.x → 1.7.x

Two schema changes in 1.7 cannot apply themselves to data a 1.6.x deployment already holds. Everything else in 1.7 runs on 1.6.x data as is. Follow the steps in order; each command checks before it changes anything and is safe to run again.

Change	Since	Why it needs a step	Command
essential/sagas.session is no longer unique	1.7.2	Mongoose autoIndex only creates indexes, so the unique session_1 index stays. The 1.7 index build is refused, and a saga that reuses a pooled session id fails with E11000.	mongo migrate
Search text fields move from standard to text_analyzer (Persian normalization); messages gains content.$value	1.7.0, 1.7.6	Elasticsearch cannot change the analyzer of an existing field. elastic migrate alone skips those fields with needs a reindex, so old and new documents keep the old analysis. The watcher only syncs changes, so it cannot refill a recreated index.	elastic reindex, then elastic migrate

Which release

mongo migrate, elastic reindex and the --dry-run option ship in 1.7.9 or later. Run them from that release against your 1.6.x data, before the 1.7.x services start.

Before You Start

• Plan a maintenance window. Searches keep working throughout, but writes to the search indices pause, and with the essential service stopped no saga can start: any operation that runs inside a saga, financial transactions among them, waits for the window to end.
• Kafka must retain change events longer than the window: the watcher replays what it missed when it restarts.
• Elasticsearch needs free disk of at least twice the size of the largest index (the preflight checks it).
• Run every command from the 1.7.x release (1.7.9 or later), with the deployment's environment (its .env). A 1.6.x checkout, image or pod does not have these commands.
  • From source — check out the release, then in the platform directory: npm run command -- <command>.
  • Docker — name the release image explicitly, with the settings the compose services use: docker run --rm --network host --env-file .env wenex/platform:<1.7.x tag> npm run command -- <command>. docker compose run would use whatever wenex/platform:latest is.
  • Kubernetes — run the command in a one-off pod or Job from the 1.7.x image, with the environment the platform pods use.

A command that fails exits with status 1, so a script running these steps stops at the first failure. An unknown task, an -i/-c key that names nothing, and --dry-run on a task other than migrate or reindex are refused the same way, before anything changes.

1. Back Up

Gate

Do not continue until both backups verify.

Dump MongoDB — at least the essential database; drop --db to dump every database:

mongodump --uri "<mongo-uri>" --db <prefix>-essential --archive=pre-1.7.archive.gz --gzip
mongorestore --uri "<mongo-uri>" --dryRun -v --archive=pre-1.7.archive.gz --gzip

The dry run must list the collections it would restore, for example found collection <prefix>-essential.sagas bson to restore to <prefix>-essential.sagas.

Snapshot Elasticsearch. A shared-filesystem repository needs path.repo set on every node; any other repository type works the same way:

curl -XPUT "$ES/_snapshot/pre-1-7" -H 'Content-Type: application/json' \
  -d '{"type":"fs","settings":{"location":"pre-1-7"}}'
curl -XPUT "$ES/_snapshot/pre-1-7/before-upgrade?wait_for_completion=true" -H 'Content-Type: application/json' \
  -d '{"indices":"<prefix>-*","include_global_state":false}'
curl "$ES/_snapshot/pre-1-7/before-upgrade?filter_path=snapshots.state,snapshots.indices"

The last call must answer "state":"SUCCESS" and list the <prefix>-conjoint-messages, <prefix>-content-posts and <prefix>-career-products indices.

2. Preflight

Three dry runs report what each command would do and change nothing:

npm run command -- mongo migrate --dry-run
npm run command -- elastic migrate --dry-run
npm run command -- elastic reindex --dry-run

Verdict	Meaning	Handled by
replace	session_1 still carries unique	mongo migrate
conflict	an index on session that is neither the 1.6 nor the 1.7 shape	by hand — the command refuses it
reindex	a field or analyzer changed in a way Elasticsearch cannot apply in place	elastic reindex
in-place	only additions: new fields or new analysis components	elastic migrate
create	the index does not exist	elastic migrate
up-to-date	nothing to do	—

A 1.6.x deployment reports replace and three reindex verdicts. An excerpt, with the reasons listed under each index:

<prefix>-essential.sagas session_1: replace, session_1 carries {"unique":true}: since 1.7.2 `session` is not unique: it holds a pooled, recycled lsid

<prefix>-content-posts (index <prefix>-content-posts): reindex
  reindex  analysis.analyzer.html_analyzer char_filter: ["html_strip"] → ["html_strip","zero_width_non_joiner"]
  reindex  title analyzer: "standard" → "text_analyzer"
  reindex  title.fields.suggest analyzer: "simple" → "text_analyzer"
  reindex  slug analyzer: "standard" → "text_analyzer"
  add      analysis.analyzer.text_analyzer

<prefix>-content-posts (index <prefix>-content-posts): reindex
  would copy into <prefix>-content-posts-20261005155657, clone the original to <prefix>-content-posts-legacy-20261005155657

Any line starting with FAIL stops elastic reindex for that index before it writes anything: fix the cause and run the dry run again.

3. Stop the Writers

Stop the watcher and the essential service:

docker compose stop watcher essential
# Kubernetes:
kubectl scale deployment/<watcher> deployment/<essential> --replicas=0

Stop the watcher, do not just pause it

elastic reindex blocks writes to the index it copies. A running watcher would fail those writes and lose the events. Stopped, it leaves them in Kafka at its committed offset and replays them on restart.

4. Migrate MongoDB

npm run command -- mongo migrate

<prefix>-essential.sagas session_1: replace, session_1 carries {"unique":true}: since 1.7.2 `session` is not unique: it holds a pooled, recycled lsid
  captured {"v":2,"key":{"session":1},"name":"session_1","unique":true}; to restore it: db.getSiblingDB("<prefix>-essential").getCollection("sagas").dropIndex("session_1"); db.getSiblingDB("<prefix>-essential").getCollection("sagas").createIndex({"session":1}, {"name":"session_1","unique":true})
Mongo migrated ;)

It drops the unique session_1 index and creates session_1 again without unique. Documents are not touched: the command checks that the collection holds as many documents afterwards. Keep the printed to restore it statement; it rebuilds the 1.6 index exactly.

5. Reindex, Then Migrate Elasticsearch

npm run command -- elastic reindex

For each index with the verdict reindex, the command:

• blocks writes to the index, then copies every document, with its _id, into <index>-<timestamp>. The copy is built from the 1.7 schema, plus every field the original gained dynamically and the shard, replica and field-limit settings it was tuned with;
• checks that the copy holds as many documents as the original;
• clones the original, an exact copy kept read-only, to <index>-legacy-<timestamp> and checks its count too;
• in one atomic call, points the alias <index> at the copy and removes the original index. Elasticsearch applies both actions or neither, and nothing is removed before both copies verify.

  <prefix>-content-posts write-blocked
  copying <prefix>-content-posts → <prefix>-content-posts-20261005155845 (task NC8iiK0uS1m9jvp9NIZj2w:2921)
  <prefix>-content-posts-20261005155845 holds all 1000 documents
  <prefix>-content-posts-legacy-20261005155845 cloned from <prefix>-content-posts, 1000 documents
  <prefix>-content-posts → <prefix>-content-posts-20261005155845
<prefix>-content-posts reindexed: 1000 documents, alias → <prefix>-content-posts-20261005155845
  rollback: PUT <prefix>-content-posts-legacy-20261005155845/_settings {"index.blocks.write":null} then POST _aliases {"actions":[{"remove":{"index":"<prefix>-content-posts-20261005155845","alias":"<prefix>-content-posts"}},{"add":{"index":"<prefix>-content-posts-legacy-20261005155845","alias":"<prefix>-content-posts","is_write_index":true}}]}

Keep the printed rollback statement and the task id. The services and the watcher use the alias exactly as they used the index, so no configuration changes.

Then apply the remaining in-place changes. On the indices just rebuilt there is nothing left to apply. elastic migrate only adds what an index lacks, and names any change that still needs elastic reindex:

npm run command -- elastic migrate

6. Deploy 1.7.x and Restart

Deploy 1.7.x, start the services, and start the watcher last; it replays the events held in Kafka since step 3.

7. Verify

All three dry runs report up-to-date:

<prefix>-essential.sagas session_1: up-to-date, session_1 already migrated
<prefix>-conjoint-messages (alias <prefix>-conjoint-messages-20261005155844): up-to-date
<prefix>-content-posts (alias <prefix>-content-posts-20261005155845): up-to-date
<prefix>-career-products (alias <prefix>-career-products-20261005155847): up-to-date

Then check:

• GET <prefix>-content-posts/_count, and likewise for messages and products, matches the counts printed in step 5. Count each alias by name: <prefix>-* also counts the legacy clones.

• A word inside a compound written with a zero-width non-joiner now matches. For example, ترمز finds لنت‌ترمز, کیک finds كيك, and 1402 finds ۱۴۰۲:

  curl "$ES/<prefix>-content-posts/_count" -H 'Content-Type: application/json' \
    -d '{"query":{"match":{"title":"ترمز"}}}'

• A saga started through POST /essential/sagas/start answers with an id.

Recovery

Situation	State	What to do
A preflight check fails (FAIL, conflict, exit 1)	Nothing changed	Fix the cause, run the dry run again
elastic reindex fails before the alias call	The original index is untouched. The command cancelled its copy, removed its own copy and clone, and lifted the write block if it had set it	Fix the cause, run it again
The alias call itself fails (a timeout, a lost connection)	Nothing is deleted: the copy, the clone and the write block stay, because Elasticsearch may still apply the call	Run elastic migrate --dry-run: if the alias points at the copy, the cutover happened. Otherwise run elastic reindex again and delete the leftovers by name
The process is killed mid-copy	The original stays write-blocked; an unattached <index>-<timestamp> may still be filling, because Elasticsearch does not cancel a copy when its client dies	Cancel the copy with POST _tasks/<task id>/_cancel (the id is in the copying line), run elastic reindex again (it warns about the block and lists the leftover index), then delete the leftover by name. To abandon the upgrade instead, lift the block: PUT <index>/_settings {"index.blocks.write":null}
The new index is wrong after the alias call	The alias points at the copy; the legacy clone is intact and read-only	Run the printed rollback statement: it lifts the clone's write block and moves the alias back in one call
The session_1 index must be restored	—	Run the printed to restore it statement. It fails once two sagas share a session id, so restore 1.6.x from the step 1 dump instead
Anything else	—	Restore from the step 1 backups. An index can be restored under a new name, then aliased: POST _snapshot/pre-1-7/before-upgrade/_restore with "rename_pattern":"(.+)","rename_replacement":"restored-$1"

The legacy clones (<index>-legacy-<timestamp>) and the indices an alias no longer points at are your rollback points; elastic delete lists them but never deletes them. Delete them by name once the upgrade is settled.

What the Commands Guarantee

• Preflight first — every command runs its own dry run immediately before acting and stops before its first write if any check fails.
• A verified backup before anything is destroyed — the original index goes only after its clone and its copy both match its document count; the dropped session_1 definition is printed before the drop.
• Atomic cutover — the move from index to alias is one _aliases call that adds the alias and removes the index (remove_index).
• Safe to repeat — a second run over a migrated system changes nothing and says so.
• Only its own leftovers — before the alias call, a failure removes only what the command created (cancelling its copy first) and lifts only a block it set. From the alias call on, nothing is deleted automatically.