📖 Guide 08 — Complete Event Lifecycle

What you’ll learn: The complete journey of a peak event from creation to post-peak descaling — every system involved, every code path triggered, every database write.

Section 1 — The Big Picture

A “Peak Event” in EPIC’s lifecycle has three phases:

╔══════════════════════════════════════════════════════════════════╗
║  PHASE 1: PRE-PEAK (Weeks before the event)                      ║
║  ─────────────────────────────────────────────────────────────   ║
║  Create Event → Gather Projections → Order Hardware →            ║
║  Confirm Delivery → Communicate TPM → Update Throttling          ║
╠══════════════════════════════════════════════════════════════════╣
║  PHASE 2: PEAK DAY (During the event)                            ║
║  ─────────────────────────────────────────────────────────────   ║
║  Monitor traffic → Handle emergent orders → Scale if needed      ║
╠══════════════════════════════════════════════════════════════════╣
║  PHASE 3: POST-PEAK (After the event)                            ║
║  ─────────────────────────────────────────────────────────────   ║
║  Gather Descale Projections → Place Descale Orders →             ║
║  Communicate Descale TPM → Update Descale Throttling →           ║
║  Final Sign-off                                                   ║
╚══════════════════════════════════════════════════════════════════╝

Timeline example for Prime Day 2024:

• T-8 weeks: Event created in EPIC
• T-6 weeks: Services submit traffic projections
• T-5 weeks: HOTW runs, places hardware orders (SPCO)
• T-4 weeks: Hardware delivered to data centers (FMC fulfillment)
• T-2 weeks: TPM communicated to downstream services
• T-1 week: Throttling updated in Gizmo/SDC
• T-0: Prime Day starts!
• T+1 week: Descale projections gathered
• T+2 weeks: Descale hardware orders placed
• T+3 weeks: Hosts returned to pool

Section 2 — Step 1: Event Creation

Who does it: EPIC team or service teams
Where: createEvent.jsx → backend API

User fills form in createEvent.jsx:
├── EventId: "PD2024"
├── EventName: "Prime Day 2024"
├── EventType: "Peak"
├── RegionList: ["us-east-1", "eu-west-1", "us-west-2"]
└── SPCOEventDates per region:
    ├── us-east-1: { spcoStart: "2024-05-15", spcoEnd: "2024-07-20" }
    └── eu-west-1: { spcoStart: "2024-05-15", spcoEnd: "2024-07-20" }
         ↓
POST /event
         ↓
Event.createEvent() in Node.js backend
         ↓
DynamoDB: Creates item in EventTable
{
    EventId: "PD2024",
    EventName: "Prime Day 2024",
    VersionId: 1,
    LatestVersionId: 1,
    RegionList: [...],
    // ...
}
         ↓
SNS publishes "EventCreated" notification
         ↓
SQS EventFleetCreationQueue receives message
         ↓
FleetReceiver.js processes:
    → Gets all registered services
    → For each service × fleet: creates EventPlan in DynamoDB
    EventPlan = {
        EventPlanId: "PD2024#RIPE-NA",
        EventMilestone: [
            { EventMilestoneId: "GatherProjectionFromUpstream", Status: "NotStarted" },
            { EventMilestoneId: "HardwareOrder", Status: "NotStarted" },
            { EventMilestoneId: "HardwareFulfillment", Status: "NotStarted" },
            { EventMilestoneId: "CommunicateTPMToDownstream", Status: "NotStarted" },
            { EventMilestoneId: "UpdateThrottlingInSDC", Status: "NotStarted" }
        ]
    }
         ↓
SQS EventTicketCreationQueue receives message
         ↓
EventTicketReceiver.js:
    → Creates SIM ticket for the event
    → Links ticket to EPIC dashboard

Section 3 — Step 2: Gather Projections

Who does it: Service owners (manually + via Axon/PMET automation)
Where: serviceDetails.jsx → Host Projections tab

Service owner opens serviceDetails.jsx for fleet "RIPE-NA" event "PD2024"
         ↓
Frontend calls:
    backend_api.getFleetDetails("RIPE-NA", "PD2024")
    apollo_api.getApolloData("RIPE_NA_PROD")
    axon_traffic_api.getTrafficData("RIPE-NA", "PD2024")
         ↓
Page shows:
    - Current hosts in Apollo: 150
    - Current TPM (from Axon): 35,000
    - CloudTune projection: 50,000 TPM at peak
         ↓
Service owner can:
    1. Accept auto-calculated projection
    2. Override Peak TPM (submitModalForOverridingInputTPM.jsx)
    3. Override specific ASG projections
         ↓
PUT /projection/{fleetId}/{eventId}
         ↓
Projection.updateProjection() saves to ProjectionsTable
         ↓
MilestoneWorkflow triggered:
    → GatherProjectionsMilestoneHandler checks all projections
    → If upstream projections verified → update status to "Completed"
    → Publish to MilestoneSNSTopic: { milestone: "HardwareOrder", fleetId, eventId }

What happens with upstream projections:

Service A sends traffic TO Service B
    ↓
Service A submits "downstream projection" for Service B
    ↓
Service B receives "upstream projection" in their upstreamProjectionsTab
    ↓
Service B verifies it (checks if the number makes sense)
    ↓
Once all upstream projections verified → GatherProjections milestone = Completed

Section 4 — Step 3: HOTW Hardware Ordering (The Core)

Who does it: Automated (HOTW system) + occasional manual trigger
When: Weekly cron job + triggered by Milestone SNS

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OPTION A: Scheduled Weekly Run
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Cron fires every week
         ↓
HotwHandler.handleSQSRequestForUpdateSpco()
         ↓
For each active event:
    For each registered service:
        For each fleet in correct region:
            → Send to validateAndUpdateSPCOSQSQueue
            Message: { FleetId, EventId, RunId }
         ↓
HotwHandler.validateAndUpdateSpco() (SQS consumer)
         ↓
For each message: calls HotwUpscalingHelper.handle(fleetId, eventId, runId)
         ↓
[The HOTW execution begins — see Section 5]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OPTION B: Manual Atomic Run (Single Fleet)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service owner opens hotwDashboard.jsx
    → Sees their fleet is "behind"
    → Clicks "Run HOTW" button
         ↓
atomicHotwModal.jsx opens
    → Shows previous vs current version
    → User confirms
         ↓
POST /hotw/atomic/{fleetId}/{eventId}
Body: { PreviousVersionId: "3", CurrentVersionId: "4" }
         ↓
HotwHandler.atomicHotwTrigger()
    → Creates RunId
    → Sends to atomicHOTWSQSQueue
         ↓
HotwHandler.atomicHotwExecutor() (SQS consumer)
    → Calls HotwUpscalingHelper.handleAtomic(fleetId, eventId, runId, prevVer, currVer)
         ↓
[The HOTW execution begins — see Section 5]

Section 5 — HOTW Execution Details

This is HotwUpscalingHelper.getResult() step by step:

┌─────────────────────────────────────────────────────────────────┐
│  HotwUpscalingHelper.getResult(fleetId, eventId, runId, vId)    │
└─────────────────────────┬───────────────────────────────────────┘
                          │
                    ┌─────▼──────┐
                    │  Step 1    │  Validate fleet
                    │ validateFleet│ ← throws if fleet invalid
                    └─────┬──────┘
                          │
                    ┌─────▼──────┐
                    │  Step 2    │  Get fleet data from EPIC Backend
                    │ getFleet() │ → Fleet: { apolloName, hostThroughputTPM, ctFactor... }
                    └─────┬──────┘
                          │
                    ┌─────▼──────┐
                    │  Step 3    │  Get event data
                    │ getEvent() │ → Event: { peakDate, regionList... }
                    └─────┬──────┘
                          │
                    ┌─────▼──────┐
                    │  Step 4    │  Get Apollo current host count
                    │ apolloHelper│ → apolloMaxHosts = 150
                    └─────┬──────┘
                          │
                    ┌─────▼──────┐
                    │  Step 5    │  Get FMC pending orders
                    │ fmcHelper  │ → pendingEpicFmcHostOrder = 20
                    └─────┬──────┘
                          │
                    ┌─────▼──────┐
                    │  Step 6    │  Get ASG details from EAP/ScalingPlanner
                    │ eapHelper  │ → For each ASG: EAP status, capacity override
                    └─────┬──────┘
                          │
                    ┌─────▼──────────────────────────────────────┐
                    │  Step 7  CALCULATE HOSTS NEEDED             │
                    │                                             │
                    │  Input TPM = 50,000 (from CloudTune/manual) │
                    │  CT Peak Factor = 1.5                       │
                    │  Buffer Factor = 1.1                        │
                    │  Host Throughput = 250 TPM/host             │
                    │                                             │
                    │  BAU TPM = 50000 / (1.5 × 1.1) = 30,303   │
                    │  Required Hosts = 50000 / 250 = 200         │
                    │  (× AZ Factor if multi-AZ)                  │
                    │                                             │
                    │  Hosts Needed = 200 - 150 - 20 = 30        │
                    │  (required - apollo - pending)              │
                    └─────┬──────────────────────────────────────┘
                          │
                    ┌─────▼──────────────────────────────────────┐
                    │  Step 8  IS ORDER NEEDED?                   │
                    │                                             │
                    │  hostsNeeded > 0?                           │
                    │  YES → place order                          │
                    │  NO → no order needed (log reason)          │
                    └─────┬──────────────────────────────────────┘
                          │
                    ┌─────▼──────────────────────────────────────┐
                    │  Step 9  IS THIS EMERGENT?                  │
                    │                                             │
                    │  Check: is today past emergent start date?  │
                    │  YES → create Sev2 SIM ticket               │
                    │  NO → standard SPCO order                   │
                    └─────┬──────────────────────────────────────┘
                          │
               ┌──────────▼──────────┐
               │ hostsNeeded > 0?    │
               └──────────┬──────────┘
                    ┌──────┘  └──────────────────────────┐
                    │ YES                                  │ NO
                    ▼                                     ▼
         ┌──────────────────────┐             ┌──────────────────────┐
         │ Step 10a             │             │ Step 10b             │
         │ fetchArnUpdateSPCO   │             │ Log "no order needed"│
         │  AndReturnSIMLink()  │             │ Update execution     │
         │                      │             │ details with reason   │
         │ → For each ASG:      │             └──────────────────────┘
         │   - Get ASG ARN      │
         │   - Enroll in EAP    │
         │     if not enrolled  │
         │   - Place SPCO order │
         │   - Open SIM ticket  │
         └──────────┬───────────┘
                    │
                    ▼
         ┌──────────────────────────────────────────────────────┐
         │ Step 11  UPDATE HOTW EXECUTION DETAILS               │
         │                                                       │
         │ POST /hotw/executionDetail                           │
         │ Body: {                                               │
         │   FleetId, EventId, RunId,                           │
         │   Status: "Success"/"Fail"/"PartialSuccess",         │
         │   HotwExecutionDetails: { peakTPM, bauTPM, ... },    │
         │   CapacityOverrideDetails: [...],                    │
         │   FulfillmentDetails: [...],                         │
         │   ASGDetails: [...]                                   │
         │ }                                                     │
         └──────────┬───────────────────────────────────────────┘
                    │
                    ▼
         ┌──────────────────────────────┐
         │ Step 12  PUBLISH TO SNS      │
         │                              │
         │ publishHardwareDetailsToSNS()│
         │ → SNS: hardware order summary│
         │ → triggers email to team     │
         └──────────┬───────────────────┘
                    │
          ┌─────────▼──────────────────┐
          │ Step 13  FINALLY BLOCK      │
          │ (ALWAYS RUNS)               │
          │                             │
          │ 1. Send to ApolloSQSQueue   │
          │    → refreshes Apollo data  │
          │                             │
          │ 2. Publish to MilestoneSNS  │
          │    → triggers milestone     │
          │      status update          │
          └─────────────────────────────┘

Section 6 — Step 4: Hardware Delivery Tracking

After orders are placed, FMC tracks delivery:

SPCO order placed (ASG ARN submitted)
         ↓
AWS hardware team processes order
         ↓
FMC order status: "Pending" → "InReview" → "Approved" → "Fulfilled"
         ↓
FmcTrigger cron fires (every few hours)
         ↓
FmcHandler fetches latest status from FMC API
         ↓
FmcHandler updates MySQL fulfillment_details table
         ↓
HardwareFulfillmentMilestoneHandler runs
         ↓
Checks: are all orders fulfilled?
    YES → Update HardwareFulfillment milestone to "Completed"
    NO  → Update with pending count
         ↓
Once all fulfilled:
    → Publish to MilestoneSNSTopic: "CommunicateTPM"

Frontend shows this as:

serviceDetails.jsx → Fleet Milestone Readiness tab:
┌────────────────────────────────────────────────────────┐
│ Milestone                   Status        Last Updated  │
├────────────────────────────────────────────────────────┤
│ Gather Projections          ✅ Completed  2024-06-01   │
│ Hardware Order              ✅ Completed  2024-06-15   │
│ Hardware Fulfillment        🟡 In Progress 2024-06-20  │
│   - 3 orders fulfilled                                  │
│   - 1 order pending approval                            │
│ Communicate TPM             ⚪ Not Started              │
│ Update Throttling           ⚪ Not Started              │
└────────────────────────────────────────────────────────┘

Section 7 — Step 5: TPM Communication

After hardware arrives:

HardwareFulfillment milestone completes
         ↓
CommunicateTPMMilestoneHandler triggered
         ↓
For each downstream service:
    → Creates/updates projection: "RIPE-NA will send X TPM to ServiceB"
    → ServiceB sees this in their upstreamProjectionsTab
    → ServiceB needs to verify/acknowledge
         ↓
Once all downstreams acknowledge:
    → CommunicateTPM milestone = "Completed"
    → Publish to MilestoneSNSTopic: "UpdateThrottling"

Section 8 — Step 6: Throttling Update

UpdateThrottlingMilestoneHandler triggered
         ↓
Reads throttling config from ThrottlingTable
         ↓
Sends to GizmoThrottlingUpdateQueue
         ↓
ThrottlingExecutor processes:
    → Calls Gizmo/SDC API with new throttling limits
    → Sets max TPS for this service during peak
         ↓
ThrottlingExecutor confirms update successful
         ↓
UpdateThrottlingInSDC milestone = "Completed"
         ↓
ALL 5 MILESTONES COMPLETE!
         ↓
Service shows 🟢 PEAK READY in serviceReadinessDashboard.jsx

Section 9 — Step 7: Peak Day

During the actual peak event:

Traffic spikes as customers start shopping
         ↓
Axon tracking real-time TPM for all services
         ↓
EPIC monitors:
    - Are actual TPMs within projected range?
    - Are hosts enough?
    - Any throttling issues?
         ↓
If emergency (actual > projected by variance threshold):
    → VarianceExceededHandler fires
    → Alerts team via SIM ticket
    → EPIC operators can manually trigger emergent HOTW
         ↓
Emergent HOTW (if needed):
    HotwUpscalingHelper detects emergent=true
    → Creates Sev2 SIM ticket
    → Places urgent SPCO order
    → FLO scales up hosts as fast as possible

EmergentAnnouncementFlashbar.jsx shows red banner:

🚨 EMERGENT SCALING IN PROGRESS
Fleet RIPE-NA requires 50 additional hosts urgently.
Order placed: SIM-123456 | FMC Order: SPCO-789012

Section 10 — Step 8: Descaling (Post-Peak)

After the peak ends, hosts need to be returned:

Event end date passes
         ↓
Descale milestones become active:
    1. GatherDescaleProjections
    2. DescaleHardwareOrder
    3. CommunicateDescaleTPM
    4. UpdateDescaleThrottling
    5. DescaleCompletion
         ↓
Service owners open descaleFleetConfigurations.jsx
    → Review descale recommendation from DescaleHostRecommendationHelper
    → Recommendation: "You can safely return 50 hosts from AZ 1a"
         ↓
Owner confirms descale
         ↓
DescaleRecommendationHandler.handle(fleetId, eventId)
    → Greedy algorithm determines which ASGs to remove from
    → For each ASG to descale:
        → Place SPCO descale order
        → FMC tracks fulfillment
         ↓
BAUScalingHandler takes over:
    → Computes BAU capacity needed
    → Returns any hosts above BAU level
         ↓
DescaleCompletion milestone = "Completed"
         ↓
Event lifecycle complete! ✅

Descale recommendation algorithm (from DescaleHostRecommendationHelper.java):

// Greedy approach: always remove from ASG with MOST hosts first
// Goal: get to target host count as evenly as possible

private Map<String, Integer> descaleHostRecommendation(Fleet fleet, int hostsToRemove) {
    Map<String, Integer> asgCurrentHosts = getCurrentHostsPerASG(fleet);
    Map<String, Integer> toRemovePerASG = new HashMap<>();
    
    while (hostsToRemove > 0) {
        // Find ASG with most hosts
        String maxAsg = asgCurrentHosts.entrySet().stream()
            .max(Map.Entry.comparingByValue())
            .get().getKey();
        
        // Remove one host from that ASG
        asgCurrentHosts.merge(maxAsg, -1, Integer::sum);
        toRemovePerASG.merge(maxAsg, 1, Integer::sum);
        hostsToRemove--;
    }
    
    return toRemovePerASG;
}
// Example: [100, 80, 70] hosts needed → recommend removing from 100-host ASG first

Section 11 — Exception Flow (When Normal Rules Don’t Apply)

Sometimes a service needs more buffer than standard:

Standard buffer factor = 1.1 (10% safety margin)
But Service XYZ says: "Our traffic is unpredictable, we need 1.3"
         ↓
Service owner opens createException.jsx:
    → Fills exception request:
        ExceptionType: "BufferFactor"
        RequestedValue: 1.3
        BusinessReason: "Our traffic has 30% variance due to external APIs"
        BusinessDrivers: ["ThirdPartyDependency", "SeasonalVariance"]
    → Submits
         ↓
POST /exception
ExceptionOperations stores in ExceptionTable
         ↓
EPIC team sees in approveException.jsx
    → Reviews business justification
    → Approves or rejects
         ↓
ApprovalsHandler.approveOrReject()
    → If approved: updates fleet's bufferFactor field to 1.3
    → Sends email notification
    → HOTW will use 1.3 as buffer factor for this fleet going forward

Section 12 — BAU (Business As Usual) Flow

BAU is different from peak — it’s ongoing, not event-specific:

BAU capacity = what you need for normal (non-peak) operations

Every week, BAUScalingHandler runs:
    1. Gets BAU TPM from Axon (last week's average)
    2. Applies BAU buffer factor
    3. Calculates BAU hosts needed = BAU TPM / Host Throughput / AZ Factor
    4. Compares with current hosts in Apollo
    5. If needed > current: places BAU SPCO order
    6. If needed < current (excess hosts): logs for potential release
         ↓
Service owners see in bauServiceOwnerDashboard.jsx:
    ┌──────────────────────────────────────────────────────────────┐
    │ Service: RIPE                                                │
    │ Fleet: RIPE-NA   BAU TPM: 30,000   Required Hosts: 120      │
    │ Current Hosts: 115   Status: ORDERING 5 HOSTS               │
    └──────────────────────────────────────────────────────────────┘

Section 13 — Onboarding Flow (New Service)

When a new service joins EPIC:

Service owner opens serviceOnboarding.jsx
    → Fills in service details:
        ServiceId: "MyNewService"
        Owner: "owner@amazon.com"
        ServiceType: "Registered"
        Fleets: [{ region: "us-east-1", fleetIds: ["MyNewService-NA"] }]
         ↓
POST /service
Service.createService() creates in ServiceTable
         ↓
SNS notification sent → Email to EPIC team
         ↓
OnboardingChecklist shows steps:
    ✅ Service registered
    ✅ Fleet created
    ⚪ Connect Axon metrics
    ⚪ Configure PMET links
    ⚪ Set BAU host throughput
    ⚪ Complete onboarding
         ↓
Each step has specific UI + backend integration:
    - Axon: configure which metric tracks TPM
    - PMET: link CloudWatch metric to fleet
    - Host Throughput: how many TPM per host type
         ↓
Once all steps complete:
    → Service is onboarded
    → HOTW will start managing it
    → Shows in readiness dashboards
    → Gets included in weekly HOTW runs

Section 14 — Key Formulas Reference

AZ Factors (from OtherConstants.js):

AZ Factor ensures each AZ has enough hosts even if one AZ fails.

Section 15 — Notification Chain

When HOTW completes for a fleet, this notification chain fires:

1. HotwUpscalingHelper publishes to SNS (hardware order details)
   → Email sent to: service owner, fleet owner, EPIC team
   Template: hardwareOrderDetails.html.ts
   
2. Published to MilestoneSNSTopic
   → HotwHandler.handleRequest() receives it
   → Updates HardwareOrder milestone status
   
3. ApolloSQSQueue receives message
   → ApolloHandler refreshes Apollo data for this fleet
   → If Apollo now shows enough hosts → updates dashboard
   
4. hotwDashboard.jsx polls every 30 seconds
   → Shows updated status for this fleet

Section 16 — State Machine: Milestone Status Transitions

          ┌──────────────┐
          │  NotStarted  │
          └──────┬───────┘
                 │ Previous milestone completed
                 ▼
          ┌──────────────┐
          │  InProgress  │◄──────────── External system updates
          └──────┬───────┘
                 │ All checks pass
                 ▼
          ┌──────────────┐
          │   Completed  │
          └──────────────┘
          
          Also possible:
          ┌──────────────────┐
          │  PendingOnEPIC   │ ← EPIC automation needs to act
          └──────────────────┘
          ┌──────────────────────┐
          │ PendingOnServiceOwner│ ← Human needs to act
          └──────────────────────┘
          ┌──────────────┐
          │    Blocked   │ ← Something went wrong
          └──────────────┘

Section 17 — Data Flow Diagram

                         EPIC Frontend
                         ┌──────────┐
                         │  React   │
                         │  Redux   │
                         └────┬─────┘
                              │ REST
                    ┌─────────▼──────────┐
                    │   API Gateway       │
                    └─────────┬──────────┘
                              │
                ┌─────────────▼─────────────────────────┐
                │          EPICBackend (Node.js)          │
                │  Fleet  Service  Event  EventPlan  HOTW │
                └─────────────┬─────────────────────────┘
                              │
                    ┌─────────┴──────────┐
                    │                    │
               ┌────▼────┐        ┌─────▼──────┐
               │DynamoDB │        │   MySQL    │
               │         │        │  (Aurora)  │
               │Fleet    │        │            │
               │Service  │        │hotw_run    │
               │Event    │        │hotw_exec   │
               │EventPlan│        │asg_details │
               │Projections│      │capacity_   │
               │Schema   │        │  override  │
               └────┬────┘        └────────────┘
                    │
           DynamoDB Streams
                    │
                    ▼
         ┌──────────────────────┐
         │ EPICBackendTriggers  │
         │    (Java Lambda)     │
         │                      │
         │  ┌────────────────┐  │
         │  │  SQS Queues    │  │
         │  │  40+ queues    │  │
         │  └────────┬───────┘  │
         │           │          │
         │  ┌────────▼───────┐  │
         │  │   Handlers     │  │
         │  │  HOTW/Apollo/  │  │
         │  │  FMC/Milestone/│  │
         │  │  BAU/Throttling│  │
         │  └────────┬───────┘  │
         └───────────┼──────────┘
                     │
         ┌───────────▼───────────────────────────┐
         │      Amazon Internal Systems           │
         │                                        │
         │ Apollo    FMC     SIM     Gizmo         │
         │ (config)  (order) (ticket) (throttle)  │
         │                                        │
         │ ScalingPlanner  CloudTune  Axon         │
         │ (EAP)           (ML proj)  (traffic)   │
         └────────────────────────────────────────┘