Twitter Recommendation Platform

A high-throughput user-recommendation service for a Twitter-shaped dataset, end-to-end: PySpark ETL → MySQL → Go HTTP service → Kubernetes → managed AWS. Built to explore how one recommendation endpoint behaves as it moves from a self-managed cluster onto managed infrastructure.

Peak throughput hit ~5,000 RPS on a two-vCPU RDS instance behind a single EKS node — with most of the gain coming from fixing a silent CFS throttle, not scaling up.

---

What's in here

Layer	Tech
ETL	PySpark (RDD + DataFrame) on GCP Dataproc, denormalizes 5 tables → 2
Storage	MySQL 8 (in-cluster on EBS, then RDS on Graviton3)
Web tier	Go 1.24 + net/http, gRPC to auth service, in-process LRU caches
Orchestration	Kubernetes — kOps (self-managed) and EKS (Fargate + Managed NG)
Infra as code	Terraform (VPC + RDS + kOps state), Helm charts, eksctl
CI/CD	GitHub Actions: build → push ECR → Helm upgrade → smoke test

---

Repository layout

├── web/                        # Go HTTP service
│   ├── main.go                 # init, DB pool, metrics endpoints
│   ├── handlers.go             # /twitter, scoring orchestration
│   ├── db.go                   # MySQL queries (wide row, single round-trip)
│   ├── cache.go                # typed LRU with hit/miss counters
│   ├── scoring.go              # keyword + hashtag scoring on raw JSON
│   ├── auth.go                 # gRPC client for the auth sidecar
│   ├── proto/                  # auth.proto + generated Go
│   └── Dockerfile
│
├── etl/                        # PySpark pipelines
│   ├── etl_rdd.py              # Raw ETL, RDD API (baseline)
│   ├── etl_dataframe.py        # Raw ETL, DataFrame API (Catalyst + AQE)
│   ├── etl_denormalize.py      # 5-table → 2-table subsequent ETL
│   └── popular_hashtags.txt
│
├── helm/
│   ├── web-tier/               # Deployment + Service + Ingress + RDS Secret
│   ├── auth-service/           # gRPC token signer
│   └── mysql/                  # In-cluster MySQL (for kOps path only)
│
├── infra/
│   ├── terraform/
│   │   ├── eks-rds/            # VPC + RDS + parameter group (EKS path)
│   │   └── kops-state/         # S3 state bucket for kOps
│   └── k8s/
│       ├── kops/               # kOps cluster + IG manifests
│       └── eks/                # eksctl cluster configs (Fargate + Managed NG)
│
├── scripts/                    # Bring-up, tear-down, ETL submit, MySQL load
│
├── .github/workflows/
│   └── deploy.yml              # CI/CD pipeline
│
└── docs/
    └── engineering-notes/      # Architecture + perf deep-dives

---

How a request flows

   client
     │  GET /twitter?user_id=…&type=both&phrase=…&hashtag=…&timestamp=…
     ▼
  ALB (ingress)
     │
     ▼
  web-tier pod ──► auth-service (gRPC)       // parallel
     │
     └─► MySQL / RDS
             SELECT … FROM user_contacts WHERE user_id = ?   // single query,
                                                             // wide denorm row
   scoring in-process:
     interaction_score × hashtag_score × keyword_score(phrase, hashtag)
   sort; fetch user_info + tweet_texts; stream response.

The wide user_contacts row — a deliberate 5→2 table denormalization — is why one query is enough. Details: docs/engineering-notes/schema-denormalization.md.

---

Deep-dive notes

These are the pieces worth looking at:

• performance-optimizations.md — six iterative tweaks, each with diagnosis and before/after RPS. T4 (CFS throttle fix) is the single largest gain.
• bottleneck-analysis.md — why the limit was the connection-pool queue, not the DB.
• schema-denormalization.md — how 5 tables collapsed into 2 and what it bought.
• json-parsing-optimization.md — skipping json.Unmarshal in the hot path.
• mysql-performance-metrics.md — a snapshot of a healthy read-heavy MySQL at ~17k QPS and 98.3% buffer pool hit rate.

---

Running it

ETL

export GCP_PROJECT="my-project"
export GCS_BUCKET="gs://my-bucket"
./scripts/run_dataproc.sh

kOps path (self-managed K8s + in-cluster MySQL)

# 1. S3 state bucket (once)
cd infra/terraform/kops-state && terraform init && terraform apply
export KOPS_STATE_STORE="$(terraform output -raw kops_state_store_export | cut -d'=' -f2)"

# 2. Bring the cluster up
./scripts/start-cluster.sh

EKS + RDS path (managed services)

cd infra/terraform/eks-rds
cp terraform.tfvars.example terraform.tfvars   # set db_password
terraform init && terraform apply

export ACCOUNT_ID="<your AWS account id>"
export RDS_PASSWORD="<same as terraform.tfvars db_password>"
./scripts/bring_up_ng_cluster.sh               # Graviton Managed Node Group
# or
./scripts/bring_up_cluster.sh                  # Fargate

Smoke test once up:

HOST=$(kubectl get ingress web-tier-ingress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl "http://$HOST/health"
curl "http://$HOST/twitter?user_id=17&type=both&phrase=the&hashtag=cloud&timestamp=1710000000"

CI/CD

Open a PR to master. The workflow in .github/workflows/deploy.yml builds an ARM64 image, pushes to ECR, helm upgrades on the running cluster, then runs /health and /twitter smoke tests. Required secrets: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AWS_ACCOUNT_ID, RDS_DB_PASSWORD, SERVICE_OWNER, SERVICE_AWS_ACCOUNT.