-
Notifications
You must be signed in to change notification settings - Fork 50
Aul UI d2 5574 identity map v3 technical sample #915
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: staging-phase-2-identity-map-v3
Are you sure you want to change the base?
Changes from 4 commits
2c56c71
2d72ae2
b39b927
8ae1c19
8588213
0ead1ec
846610b
b67e514
af5fa1e
bb2191d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -58,6 +58,13 @@ DII refers to a user's normalized email address or phone number, or the normaliz | |
|
|
||
| <!-- diagram source: resource/advertiser-flow-endpoints-v3-mermaid.mermaid --> | ||
|
|
||
| ## Integration Example | ||
|
|
||
| For a complete demonstration of a working integration that includes all the recommended patterns, see the [UID2 Identity Map V3 Integration Example](https://github.com/IABTechLab/uid2docs/blob/main/static/examples/identity-map-integration-example). | ||
| While the sample is implemented using the Python SDK, the integration patterns are applicable to any SDK or direct API integration. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This isn't the main meaning for "while" so better not to use it, to avoid possible misunderstanding.. |
||
|
|
||
| For step-by-step setup instructions and to run the example, see the [Integration Example README](https://github.com/IABTechLab/uid2docs/blob/main/static/examples/identity-map-integration-example/README.md). | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For the link copy, use the title of the destination doc. In which case we don't need "the" also. |
||
|
|
||
| ### 1: Generate Raw UID2s from DII | ||
|
|
||
| | Step | Endpoint | Description | | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -60,6 +60,13 @@ The following table shows the implementation options that are available for adve | |
| | [5: Monitor for Raw UID2 Refresh](#5-monitor-for-raw-uid2-refresh) | Use the refresh timestamp (`r` field) returned from the [POST /identity/map](../endpoints/post-identity-map.md) endpoint to determine when to refresh Raw UID2s. | | ||
| | [6: Monitor for Opt-Out Status](#6-monitor-for-opt-out-status) | API call to the [POST /optout/status](../endpoints/post-optout-status.md) endpoint. | | ||
|
|
||
| ## Integration Example | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, it is and it is applicable to Snowflake. Gian is finishing changes to that doc now. This is not applicable to AWS Entity Resolution. I'll make changes you've suggested above across the board. |
||
|
|
||
| For a complete demonstration of a working integration that includes all the recommended patterns, see the [UID2 Identity Map V3 Integration Example](https://github.com/IABTechLab/uid2docs/blob/main/static/examples/identity-map-integration-example). | ||
| While the sample is implemented using the Python SDK, the integration patterns and applicable to any SDK or direct API integration. | ||
|
|
||
| For step-by-step setup instructions and to run the example, see the [Integration Example README](https://github.com/IABTechLab/uid2docs/blob/main/static/examples/identity-map-integration-example/README.md). | ||
|
|
||
| ## Integration Diagram | ||
|
|
||
| The following diagram outlines the steps that data collectors must complete to map DII to raw UID2s for audience building and targeting. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -350,6 +350,13 @@ If you're using server-side integration (see [Publisher Integration Guide, Serve | |
| } | ||
| ``` | ||
|
|
||
| ## Integration Example | ||
|
|
||
| For a complete demonstration of a working integration that includes all the recommended patterns, see the [UID2 Identity Map V3 Integration Example](https://github.com/IABTechLab/uid2docs/blob/main/static/examples/identity-map-integration-example). | ||
| While the sample is implemented using the Python SDK, the integration patterns are applicable to any SDK or direct API integration. | ||
|
|
||
| For step-by-step setup instructions and to run the example, see the [Integration Example README](https://github.com/IABTechLab/uid2docs/blob/main/static/examples/identity-map-integration-example/README.md). | ||
|
|
||
| >**Note:** The SDK automatically handles email normalization and hashing, ensuring that raw email addresses and phone numbers do not leave your server. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This note was at the end of the prior section -- and that's where it should be. The Integration Example section was inserted in the wrong place. |
||
|
|
||
| ### Usage Example | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -301,6 +301,12 @@ mixed_input = IdentityMapV3Input() | |
| mixed_response = client.generate_identity_map(mixed_input) | ||
| ``` | ||
|
|
||
| ### Integration Example | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If this is the identical copy, same edits and preferably a snippet... same as prior comment.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There's a subtle difference for Python SDK vs the others - in other places I say that the pattern is applicable despite the example using Python SDK, here I don't. Not sure it's worth keeping them separate though - happy to make it the same.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @aulme that makes sense, thx. But I do think we should have a snippet. I think it'd be OK to still have that line in the Python SDK doc... it's kind of unnecessary data but I don't think it sounds weird. It's just info. LMK if you want me to do that for you in the branch. |
||
|
|
||
| For a complete demonstration of a working integration that includes all the recommended patterns, see the [UID2 Identity Map V3 Integration Example](https://github.com/IABTechLab/uid2docs/blob/main/static/examples/identity-map-integration-example). | ||
|
|
||
| For step-by-step setup instructions and to run the example, see the [Integration Example README](https://github.com/IABTechLab/uid2docs/blob/main/static/examples/identity-map-integration-example/README.md). | ||
|
|
||
| ## Migration From Version Using v2 Identity Map | ||
|
|
||
| The following sections provide general information and guidance for migrating to the latest version of this SDK, which references `POST /identity/map` version 3, including: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Environment variables | ||
| .env | ||
|
|
||
| # Python cache | ||
| __pycache__/ | ||
| *.pyc | ||
|
|
||
| # Virtual environments (legacy and uv) | ||
| venv/ | ||
| .venv/ | ||
|
|
||
| # uv lock file | ||
| uv.lock | ||
|
|
||
| # Database files | ||
| *.db | ||
|
|
||
| .idea |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 3.13 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # UID2 Integration Technical Sample | ||
|
|
||
| **Complete UID2 integration example demonstrating Identity Map V3 flow.** | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. V3 > v3 |
||
|
|
||
| This sample shows a pattern for mapping email addresses and phone numbers to UID tokens, handling optouts, managing token refresh cycles, and performing a sample attribution analysis based on both current and previous UIDs. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the heading we have UID2 and now we have UID. Ideally keep it consistent at UID2 since this is in the context of the UID2 site, not a general resource. |
||
|
|
||
| ## Project Structure | ||
|
|
||
| ``` | ||
| identity-map-integration-example/ | ||
| ├── src/ # Python source code | ||
| │ ├── complete_demo.py # End-to-end demo workflow | ||
| │ ├── map_identities.py # Core UID2 mapping logic | ||
| │ ├── attribution_analysis.py # Attribution analysis example | ||
| │ ├── config.py # Configuration loading | ||
| │ ├── database.py # Database schema and utilities | ||
| │ ├── uid_client_wrapper.py # UID2 client with retry logic | ||
| │ └── populate_*.py # Test data generation scripts | ||
| ├── .env # UID2 credentials (create from .env.example) | ||
| ├── pyproject.toml # Project configuration | ||
| └── README.md # This file | ||
| ``` | ||
|
|
||
| ## Quick Start | ||
|
|
||
| ### 1. Install Dependencies | ||
| ```bash | ||
| # Install uv (Python package manager) | ||
| curl -LsSf https://astral.sh/uv/install.sh | sh | ||
|
|
||
| # Install project dependencies | ||
| uv sync | ||
| ``` | ||
|
|
||
| ### 2. Configure UID2 Credentials | ||
| ```bash | ||
| cp .env.example .env | ||
| # Edit .env with your UID2 integration credentials | ||
| ``` | ||
|
|
||
| Required `.env` format: | ||
| ``` | ||
| UID2_BASE_URL=operator-integ.uidapi.com | ||
| UID2_API_KEY=your_api_key_here | ||
| UID2_SECRET_KEY=your_secret_key_here | ||
| ``` | ||
|
|
||
| ### 3. Run Complete Demo | ||
| ```bash | ||
| # Full workflow: test data population → UID2 mapping → attribution analysis | ||
| uv run src/complete_demo.py | ||
| ``` | ||
|
|
||
| ### 4. Run Individual Components | ||
| ```bash | ||
| # Generate test data only | ||
| uv run src/populate_test_uid_mappings.py | ||
|
|
||
| # Run UID mapping only | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. UID > UID2, any instances you can in general copy.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Changed everything except for code |
||
| uv run src/map_identities.py | ||
|
|
||
| # Run attribution analysis only | ||
| uv run src/attribution_analysis.py | ||
| ``` | ||
|
|
||
| ## Core UID2 Integration Patterns | ||
|
|
||
| ### Identity Mapping Workflow | ||
|
|
||
| **Key Integration Points:** | ||
| 1. **Batch Processing** (`src/map_identities.py:build_uid2_input()`) - Process sequential batches of up to 5,000 DIIs per request | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not DIIs. In this context, email addresses and/or phone numbers. |
||
| 2. **Retry Logic** (`src/uid_client_wrapper.py:generate_identity_map_with_retry()`) - Exponential backoff for network resilience | ||
| 3. **Response Handling** (`src/map_identities.py:process_uid2_response()`) - Process mapped, opted-out, and invalid identifiers | ||
|
|
||
| ## Sample Database Schema | ||
|
|
||
| **Core `uid_mapping` table:** | ||
| ```sql | ||
| CREATE TABLE uid_mapping ( | ||
| uid_mapping_id INTEGER PRIMARY KEY, | ||
| dii TEXT NOT NULL, -- Email or phone (+E.164) | ||
| dii_type TEXT NOT NULL, -- 'email' or 'phone' | ||
| current_uid TEXT, -- Current UID2 token | ||
| previous_uid TEXT, -- Previous UID2 token (only available for 90 days after rotation, afterwards NULL) | ||
| refresh_from TIMESTAMP, -- When to refresh mapping | ||
| opt_out BOOLEAN DEFAULT FALSE -- The user has opted out, we shouldn't attempt to map them again | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. them > this user |
||
| ); | ||
| ``` | ||
|
|
||
| **Key business logic queries:** | ||
| ```sql | ||
| -- Records needing mapping (never mapped + refresh expired) | ||
| SELECT uid_mapping_id, dii, dii_type | ||
| FROM uid_mapping | ||
| WHERE opt_out = FALSE | ||
| AND (current_uid IS NULL OR refresh_from < datetime('now')); | ||
|
|
||
| -- Attribution joins using both current and previous UIDs | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. UIDs > UID2s |
||
| SELECT * FROM impressions imp | ||
| JOIN uid_mapping um ON (imp.uid2 = um.current_uid OR imp.uid2 = um.previous_uid) | ||
| WHERE um.opt_out = FALSE; | ||
| ``` | ||
|
|
||
| ## Script Reference | ||
|
|
||
| | Script | Purpose | Key Integration Concepts | | ||
| |--------|---------|-------------------------| | ||
| | `src/populate_test_uid_mappings.py` | Creates 100k test records | Database schema, DII formatting | | ||
| | `src/map_identities.py` | **Core UID2 mapping logic** | Batch processing, retry logic, response handling | | ||
| | `src/populate_test_conversions_impressions.py` | Attribution demo data | UID2 token usage in measurement | | ||
| | `src/attribution_analysis.py` | Attribution analysis | Cross-UID joins, measurement patterns | | ||
|
genwhittTTD marked this conversation as resolved.
Outdated
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. UID > UID2 |
||
| | `src/complete_demo.py` | End-to-end workflow | Full integration validation | | ||
|
|
||
| ## Production Integration Checklist | ||
|
|
||
| **Patterns for UID2 Integration:** | ||
|
|
||
| ✅ **Request Limits**: Maximum 5,000 DIIs per request | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As previous. Don't say DIIs anywhere. |
||
| ✅ **Sequential Processing**: No parallel requests to UID2 service | ||
| ✅ **Retry Logic**: Exponential backoff for network failures | ||
| ✅ **Optout Handling**: Permanent exclude opted out users from future processing | ||
| ✅ **Token Refresh**: Re-map tokens when they reach `refresh_from` timestamps | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it tokens, not raw UID2s?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry, I raw UID2s, used colloquially |
||
| ✅ **State Persistence**: Track mapping state | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| UID2_BASE_URL=https://operator-integ.uidapi.com | ||
| UID2_API_KEY=your_api_key_here | ||
| UID2_SECRET_KEY=your_secret_key_here |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| [project] | ||
| name = "identity-map-tech-sample-2" | ||
| version = "0.1.0" | ||
| description = "UID2 Identity Map V3 technical sample demonstrating email/phone to UID2 mapping with proper optout handling" | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. V3 > v3 optout handling > opt-out handling |
||
| requires-python = ">=3.13" | ||
| dependencies = [ | ||
| "python-dotenv>=1.0.0", | ||
| "uid2-client>=2.6.0", | ||
| ] | ||
|
|
||
| [dependency-groups] | ||
| dev = [ | ||
| "black>=23.0.0", | ||
| ] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| #!/usr/bin/env python3 | ||
| """ | ||
| Simple demo of joining impression and conversion data via current and previous UIDs | ||
| """ | ||
| import sqlite3 | ||
| import traceback | ||
| from database import get_connection | ||
|
|
||
|
|
||
| def attribution_analysis(conn: sqlite3.Connection) -> None: | ||
| """Run simple attribution analysis query""" | ||
| cursor = conn.cursor() | ||
|
|
||
| attribution_query = """ | ||
| SELECT | ||
| imp.impression_id, | ||
| conv.conversion_id, | ||
| conv.conversion_value, | ||
| imp.campaign_id, | ||
| um.dii, | ||
| um.current_uid | ||
| FROM impressions imp | ||
| JOIN uid_mapping um ON (imp.uid = um.current_uid OR imp.uid = um.previous_uid) | ||
| JOIN conversions conv ON (conv.uid = um.current_uid OR conv.uid = um.previous_uid) | ||
| WHERE um.opt_out = FALSE | ||
| ORDER BY RANDOM() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why do we want to order by random here?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just for the demo - we're showing 10 rows, if we don't order by random you'll just see a bunch of rows for the same impression. |
||
| LIMIT 10 | ||
| """ | ||
|
|
||
| cursor.execute(attribution_query) | ||
| results = cursor.fetchall() | ||
|
|
||
| print("Sample Attribution Results:") | ||
| print( | ||
| f"{'Impression':<12} {'Conversion':<12} {'Value':<10} {'Campaign':<12} {'DII':<40} {'UID':<15}" | ||
| ) | ||
| print("-" * 110) | ||
|
|
||
| for row in results: | ||
| imp_id, conv_id, value, campaign, dii, uid = row | ||
| print( | ||
| f"{imp_id:<12} {conv_id:<12} ${value:<9.2f} {campaign:<12} {dii:<40} {uid:<15}" | ||
| ) | ||
|
|
||
|
|
||
| def main(): | ||
| try: | ||
| conn = get_connection() | ||
| attribution_analysis(conn) | ||
| except Exception as e: | ||
| print(f"Attribution analysis failed: {e}") | ||
| traceback.print_exc() | ||
| finally: | ||
| if "conn" in locals(): | ||
| conn.close() | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| #!/usr/bin/env python3 | ||
| """ | ||
| Complete demo of UID Identity Mapping: | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. UID > UID2 Should be everywhere except code. Since this is for the UID2 site, not shared content. |
||
| - Creates a test database. | ||
| - Populates the database with test identity mapping data. | ||
| - Runs the UID mapping process. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. UID > UID2 |
||
| - Populates the database with test impression and conversion data. | ||
| - Runs a sample attribution analysis. | ||
| """ | ||
| import traceback | ||
|
|
||
| import map_identities | ||
| import populate_test_uid_mappings | ||
| import populate_test_conversions_impressions | ||
| import attribution_analysis | ||
| from database import get_connection | ||
|
|
||
|
|
||
| def complete_demo(): | ||
| conn = get_connection("uid_demo.db") | ||
| try: | ||
| print("Step 1: Populating UID mapping test data...") | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. UID > UID2 |
||
| populate_test_uid_mappings.populate_database(conn) | ||
|
|
||
| print("Step 2: Running UID mapping...") | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. UID > UID2 |
||
| map_identities.map_identities(conn) | ||
|
|
||
| print("Step 3: Populating attribution test data...") | ||
| populate_test_conversions_impressions.populate_attribution_data(conn) | ||
|
|
||
| print("Step 4: Running attribution analysis...") | ||
| attribution_analysis.attribution_analysis(conn) | ||
|
|
||
| print("Demo completed successfully!") | ||
|
|
||
| except Exception as e: | ||
| print(f"Failed with error: {e}") | ||
| traceback.print_exc() | ||
|
|
||
| finally: | ||
| conn.close() | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| complete_demo() | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| import os | ||
| import sys | ||
| from dataclasses import dataclass | ||
|
|
||
| from dotenv import load_dotenv | ||
|
|
||
|
|
||
| @dataclass | ||
| class Config: | ||
| uid_base_url: str | ||
| uid_api_key: str | ||
| uid_secret_key: str | ||
|
|
||
|
|
||
| def load_config() -> Config: | ||
| load_dotenv(override=True) # Override existing environment variables | ||
|
|
||
| uid_base_url = os.getenv("UID2_BASE_URL") | ||
| uid_api_key = os.getenv("UID2_API_KEY") | ||
| uid_secret_key = os.getenv("UID2_SECRET_KEY") | ||
|
|
||
| missing: list[str] = [] | ||
| if not uid_base_url: | ||
| missing.append("UID2_BASE_URL") | ||
| if not uid_api_key: | ||
| missing.append("UID2_API_KEY") | ||
| if not uid_secret_key: | ||
| missing.append("UID2_SECRET_KEY") | ||
|
|
||
| if missing: | ||
| print(f"Error: Missing required environment variables: {missing}") | ||
| sys.exit(1) | ||
|
|
||
| # At this point, we know all values are not None due to the validation above | ||
| assert uid_base_url is not None | ||
| assert uid_api_key is not None | ||
| assert uid_secret_key is not None | ||
|
|
||
| return Config(uid_base_url, uid_api_key, uid_secret_key) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
V3 > v3 -- could we do this in the destination please? Or say Version 3