feat: infer memberOrganization stint dates from work-email activities (CM-1105)

[skwowet]

Context

When an activity comes in with a verified work email like jbeulich@suse.com, we already create a memberOrganizations row linking the member to SUSE, but with NULL dateStart/dateEnd. That causes two problems:

1. A @suse.com activity can lose to an unrelated dated enrichment row, because findAffiliation treats undated rows as last-resort fallback.
2. The affiliation timeline ignores the row entirely since it has no dates.

We want to use the activity timestamp as evidence that the person was at that company at that moment, and write that into dateStart/dateEnd. The catch: we can't write on every activity, active maintainers generate hundreds per day, we can't collapse a real multi-stint history like Google -> Apple -> Google into one wrong range, and we can't override user edits or enrichment data.

How it works

Hot path (data_sink_worker): When an activity arrives, we buffer orgId|date in a Redis set keyed by member, and add the member to a pending-members Redis set. This is done with Redis MULTI + SADD, so concurrent data-sink workers can safely write the same member/org/date without lost updates. Hundreds of same-day activities collapse to a single set member.

Cron (cron_service): Every 5 min, the service samples up to 500 pending members, reads each member's buffered set, loads their existing email-domain rows, and walks the buffered dates chronologically applying 4 rules:

1. Date already inside an existing stint -> no-op
2. No dated stint yet for this org -> stamp the undated placeholder row
3. Later than dateEnd -> extend forward, with a 30-day debounce and a multi-stint guard. If another org holds a 30+ day stint in the gap, insert a fresh stint instead of bridging.
4. Earlier than dateStart -> extend backward, same multi-stint guard, no debounce. This is mostly for historical re-ingestion.

After a member is processed successfully, cron atomically removes only the exact Redis set values it read. If new activity arrives while cron is processing, those new values remain buffered for the next run. If cron fails before acking Redis, the dates stay in Redis and are retried. The ack uses a small Lua script with chunked SREM calls to avoid large-argument unpack limits.

Walking all orgs together in chronological order is what makes multi-stint detection work: by the time the later Google event checks its gap, the Apple stint is already in the working copy and the guard can prevent incorrectly bridging Google -> Apple -> Google into one range.

findAffiliation gets two changes so email-domain rows start contributing meaningfully:

• Source priority tier (ui > email-domain > enrichment-* > other) is added to decidePrimaryOrganizationId. Once email-domain rows have dates, they beat enrichment on overlaps.
• Work-email bias: If the activity payload has a verified work email, findMemberWorkExperience pulls in the matching email-domain row as a candidate even when undated. This is the user-visible win that lands immediately: work-email activities resolve to the right org inline, without waiting for the cron to stamp dates.

Partial index on memberOrganizations("memberId") WHERE source = 'email-domain' AND deletedAt IS NULL backs the cron's per-member fetch so it's a single index seek.

Cleanup

• Dropped two outdated bin scripts (map-member-to-org.ts, map-tenant-members-to-org.ts).
• Renamed memberOrganization.ts -> member-organization.ts to match the folder's kebab-case convention.

---

Note

Medium Risk
Medium risk because it changes organization affiliation selection logic and introduces a new Redis-backed cron workflow that writes to memberOrganizations, which could affect timelines/affiliation attribution and add operational load if mis-tuned.

Overview
Adds a Redis-buffered pipeline to infer and persist memberOrganizations.dateStart/dateEnd for email-domain associations based on activity timestamps: data_sink_worker buffers per-member orgId|date evidence and a new cron_service job periodically applies computed inserts/updates to the DB.

Updates affiliation resolution to better respect user intent and source priority: frontend edits now force memberOrganizations.source to UI, and findAffiliation/decidePrimaryOrganizationId prefer higher-priority sources (UI > email-domain > enrichment-*) and can bias toward a verified work-email domain even when the email-domain row is undated.

Performance/ops changes include a new partial Postgres index for per-member email-domain lookups, an atomic Redis ackSetMembers Lua helper to safely clear processed evidence, and removal of two obsolete data-sink mapping scripts (plus a rename to member-organization.ts).

^{Reviewed by Cursor Bugbot for commit c2f76de. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Adds infrastructure to infer and persist memberOrganizations stint dates from verified work-email activities, so email-domain affiliations become timeline-aware and can compete with enrichment on overlaps.

Changes:

• Extend affiliation resolution to bias toward email-domain rows when a verified email domain is present, and add a source-priority tier in decidePrimaryOrganizationId.
• Buffer (memberId, orgId, YYYY-MM-DD) activity evidence in Redis on the ingestion hot path and introduce a cron job to infer stint insert/update operations from buffered dates.
• Add a partial Postgres index to speed up per-member fetches of email-domain memberOrganizations; remove legacy mapping scripts and rename the shared member-organization service file.

Reviewed changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
services/libs/types/src/organizations.ts	Adds shared types for buffered org-dates and inferred stint changes.
services/libs/data-access-layer/src/old/apps/data_sink_worker/repo/memberAffiliation.data.ts	Extends work-experience data shape to include source.
services/libs/data-access-layer/src/members/segments.ts	Adds optional email-domain candidate inclusion in findMemberWorkExperience.
services/libs/data-access-layer/src/members/organizations.ts	Adds fetchMemberOrganizationsBySource for cron’s targeted reads.
services/libs/common_services/src/services/memberOrganization.ts	Deleted (renamed).
services/libs/common_services/src/services/member/unmerge.ts	Updates import to new member-organization module path.
services/libs/common_services/src/services/member-organization.ts	New module: keeps unmerge helpers and adds stint inference logic + Redis key constants.
services/libs/common_services/src/services/index.ts	Re-exports renamed member-organization module.
services/libs/common_services/src/services/common.member.service.ts	Threads emailDomain through findAffiliation and adds source-priority selection logic.
services/apps/data_sink_worker/src/service/member.service.ts	Buffers per-member per-org activity dates in Redis and enqueues member IDs for cron.
services/apps/data_sink_worker/src/service/activity.service.ts	Extracts verified email domain from activity payload and passes it into affiliation lookup.
services/apps/data_sink_worker/src/bin/map-tenant-members-to-org.ts	Removed outdated script.
services/apps/data_sink_worker/src/bin/map-member-to-org.ts	Removed outdated script.
services/apps/data_sink_worker/package.json	Removes script entries for deleted bin scripts.
services/apps/cron_service/src/jobs/inferMemberOrganizationStintChanges.job.ts	New cron job to drain Redis buffers and compute stint changes (currently dry-run).
backend/src/database/migrations/V1776931245__member-organizations-email-domain-partial-index.sql	Adds partial index to support efficient per-member email-domain org reads.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 16 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 16 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 16 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 17 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[themarolt]

Remove log level trace here.

[skwowet]

Yeah, added it for debugging during tests. Will remove it after.

[themarolt]

Remove log level trace here.

[skwowet]

Same here

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bdce7d1. Configure here.}