📗 Layer 5 — Full System Flow - Obsidian Developer Docs

The Big Picture: How all 14 files work together

1. The HOTW Big Picture

╔══════════════════════════════════════════════════════════════════════════╗
║                     HOTW SYSTEM — COMPLETE FLOW                        ║
╚══════════════════════════════════════════════════════════════════════════╝

  TRIGGER: Weekly CloudWatch Event (every Monday morning)
  OR
  TRIGGER: DynamoDB Stream (when fleet version is updated)
                │
                ▼
  ┌─────────────────────────────────────────────────────┐
  │           HotwHandler.java (Lambda entry point)     │
  │                                                     │
  │  1. Parse event (which fleets to process)           │
  │  2. Create HotwUpscalingHelper(context)             │
  │  3. For each fleet: call handle() or handleAtomic() │
  └──────────────────────┬──────────────────────────────┘
                         │
                         ▼
  ┌─────────────────────────────────────────────────────┐
  │         HotwUpscalingHelper.handle()                │
  │                                                     │
  │  Step 1: Get latest fleet from EPIC API             │
  │  Step 2: validateFleetData() — must have:           │
  │    ✓ FleetProjection[0] with InputTPM > 5           │
  │    ✓ HostThroughputTPM > 1                          │
  │    ✓ FleetType = "Registered"                       │
  │    ✓ TotalInputTPMLinkVerifiedByCustomer = true      │
  │  Step 3: updateFMCAndApolloDetails()                │
  │  Step 4: sleep(5000ms)                              │
  │  Step 5: getResult() ─────────────────────────────► │
  └──────────────────────┬──────────────────────────────┘
                         │ HOTWHelperModel returned
                         ▼
  ┌─────────────────────────────────────────────────────┐
  │  If hostNeeded > 0 AND not differentialOrder:       │
  │    publishHardwareDetailsToSNS()                    │
  │    → Sends email to service owner + EPIC team       │
  └─────────────────────────────────────────────────────┘

2. The `getResult()` Method Flow in Detail

getResult(fleetId, eventId, runId, versionId)
│
├─ GET FLEET DATA ─────────────────────────────────────────────────────
│   fleet = callGetLatestEventFleet(fleetId, eventId)
│   projectedHost = fleet.getFleetProjection()[0].getProjectedHosts()
│   (or specific version if versionId provided)
│
├─ CLASSIFY HOSTS ─────────────────────────────────────────────────────
│   hostTypeWithAsg = { "r5.xlarge": ["asg-1a", "asg-1b"] }
│   sortedMap = { "r5.xlarge": 150 }  (count per type, sorted desc)
│   hostType = "r5.xlarge"  (dominant type)
│   asgList = ["asg-1a", "asg-1b"]
│
├─ CALCULATE HOSTS NEEDED ─────────────────────────────────────────────
│   maxHostInApollo = 180
│   totalPendingHost = 10  (pending approval + pending delivery)
│   hostNeeded = projectedHost(200) - maxHostInApollo(180) - pending(10) = 10
│
│   hostNeededWithPending = 10 + pending(10) = 20
│   (full SPCO should be for 20, but delta is only 10)
│
├─ GET PER-ASG RECOMMENDATIONS ────────────────────────────────────────
│   asgToHostRecommendationMap = {
│       "asg-1a": 12,   (distribute 20 hosts across ASGs)
│       "asg-1b": 8
│   }
│
├─ HANDLE PREFERRED ASG CHANGES ───────────────────────────────────────
│   preferredASGList = [latest, previousLatest]
│   If ASGs were removed from preference:
│       - Those ASGs get BAU count (from asgProperties.hostCount)
│       - Remaining preferred ASGs get new recommendations
│       - Maps are MERGED
│
├─ CHECK EMERGENT ─────────────────────────────────────────────────────
│   Is today past EmergentScalingStartDate AND before SPCO end date?
│   │
│   ├─ YES → checkAndCreateSev2TicketIfNeeded()
│   │           Check for existing sev2 ticket
│   │           If none → createSev2PendingTicketOnEPICCTI()
│   │           sev2TicketId = "T-12345"
│   │
│   └─ NO → sev2TicketId = ""
│
├─ PLACE SPCO (HARDWARE ORDERS) ────────────────────────────────────────
│   fetchArnUpdateSPCOAndReturnSIMLink()
│   │
│   ├─ For each ASG in asgToHostRecommendationMap:
│   │   │
│   │   ├─ scalingPlannerHelper.getAsgInfoResponse() → asgArn, EAP status
│   │   │
│   │   ├─ IF EAP Enabled:
│   │   │     asgArn[asgTag] = asgScalingPlannerInfo  (store for later)
│   │   │
│   │   └─ IF EAP Disabled:
│   │         asgDisableOrNotOnboarded = true  (will fail below)
│   │
│   ├─ IF ALL ASGs EAP enabled:
│   │   │
│   │   ├─ getSIMLink() → existing or new SIM ticket
│   │   │
│   │   └─ For each ASG:
│   │       ├─ Check lower bound
│   │       ├─ ScalingStrategyFactory.getScalingStrategy()
│   │       │     (how to gradually scale up: gradual vs immediate)
│   │       └─ scalingPlannerHelper.getScalingPlanOverrideV2ResultResponse()
│   │             → ACTUAL SPCO PLACED IN AWS! 🎯
│   │
│   └─ IF ANY ASG not EAP enabled:
│         → createAndInsertHotwExecutionDetailsForFailExecution()
│         → throw IllegalStateException("Some ASGs are disabled on EAP")
│
├─ UPDATE FMC ORDERS ────────────────────────────────────────────────────
│   fmcHelper.updateFMCorders() → sync FMC status from ScalingPlanner
│
├─ UPDATE SIM TICKET ────────────────────────────────────────────────────
│   updateHardwareOrderSimDetails()
│   │
│   ├─ IF fleet already in SIM:
│   │   ├─ Get HOTW execution history
│   │   ├─ IF reqHosts changed: update SIM with new data + add comment
│   │   └─ IF reqHosts same: no SIM update needed
│   │
│   └─ IF fleet new to SIM:
│       ├─ Get SIM description
│       ├─ Append new table row with fleet data
│       ├─ Update SIM description
│       ├─ Print FMC links in SIM thread
│       └─ Create ticket entry in EPIC DB
│
├─ SAVE AUDIT RECORDS ───────────────────────────────────────────────────
│   createHotwExecutionDetails(hotwExecutionAllDetailsInput)
│   updateHotwRunDetails(runId, eventId, orderType)
│
├─ BUILD RETURN VALUE ───────────────────────────────────────────────────
│   HOTWHelperModel {
│       hostNeeded: 10,
│       service: service,
│       hardwareOrderSimLink: "https://sim.amazon.com/issues/T-12345",
│       isDifferentialOrder: true/false,
│       fleet: fleet,
│       emergentDetails: { isEmergent: false, ... }
│   }
│
└─ [FINALLY] ────────────────────────────────────────────────────────────
    sqs.sendMessage → Apollo queue (trigger Apollo config refresh)
    publishToMilestoneSNSTopic → trigger HardwareOrder milestone completion

3. File Relationships Diagram

Files in HotwHelper and what they use each other:

HotwUpscalingHelper.java  ◄──── MAIN ORCHESTRATOR ────────────────────────
    │
    ├── uses ──► HardwareOrdersUtil
    │               │
    │               ├── uses ──► TableUtil
    │               │               └── uses ──► TableConstants
    │               │
    │               ├── builds ──► HotwAllDetails
    │               │                  └── HotwAllDetails.FleetDetails (inner)
    │               │
    │               └── builds ──► HotwExecutionAllDetailsInput
    │                                  ├── HotwExecutionDetails (inner)
    │                                  ├── CapacityOverrideDetails (inner)
    │                                  ├── FulfillmentDetail (inner)
    │                                  └── StatusOfHOTW enum (inner)
    │
    ├── uses ──► EPICBackendHotwApiCallsCommon
    │               ├── calls HTTP ─► EPIC Backend APIs
    │               └── returns ──► PreferredASG (list)
    │
    ├── builds ──► HOTWHelperModel
    │                  └── EmergentDetails (inner class)
    │
    ├── uses ──► AtomicHOTWRequestModel (input for handleAtomic)
    │
    └── uses ──► CreateHotwRunDetailsInput (for run record updates)


DescaleHostRecommendationHelper.java  ◄── POST-EVENT DESCALING
    │
    ├── uses ──► EPICBackendHotwApiCallsCommon
    │
    └── builds ──► DescaleRecommendationWithClb (result per ASG)


OrderDetails.java  ◄── SIMPLE DATA CARRIER (used in other reports/tables)

4. What Each Java Concept Means in This Code

Classes = Real-world things

Java Class	Real-world thing
`Fleet`	A server group (e.g., FORTRESSService-X1-NA)
`Service`	An Amazon service (e.g., FORTRESSService)
`Event`	A peak event (Prime Day 2026)
`HOTWHelperModel`	Result of one fleet’s HOTW processing
`HotwAllDetails`	Summary of a fleet’s hardware situation
`PreferredASG`	A preference record for which ASGs to use
`CapacityOverrideDetails`	One SPCO (hardware order request)
`FulfillmentDetail`	Status of one hardware delivery

Methods = Actions

Java Method	Real-world action
`handle()`	“Run HOTW for this fleet”
`handleAtomic()`	“Re-run HOTW because fleet data changed”
`validateFleetData()`	“Check fleet is ready for HOTW”
`getResult()`	“Calculate needs and place orders”
`calculateBauTpm()`	“Work backwards to find BAU traffic”
`buildHOTWDetails()`	“Gather all data about fleet’s orders”
`descriptionFormatter()`	“Format data as a markdown table row”
`publishHardwareDetailsToSNS()`	“Email the service owner”
`createSev2PendingTicketOnEPICCTI()`	“Create emergency SIM ticket”

Exceptions = Things that can go wrong

Exception thrown	When it’s thrown
`IllegalArgumentException`	Fleet data is invalid/incomplete
`IllegalStateException`	Business rule violation (e.g., >1000 hosts, ASG not on EAP)
`Exception e` re-thrown	Unexpected error — propagates up to caller

5. The Emergent Ordering Flow

Normal order timeline:
  Feb: Initial hardware order
  Apr: Hardware readiness date
  Jul 14: Peak Event starts

Emergent timeline (close to peak):
  Jun: EmergentScalingStartDate  ← after this, orders are "emergent"
  Jul: SPCO End Date
  Jul 14: Peak Event starts

checkIfCurrentDayPastEmergentScaleStartDate():
    Returns true if:
    currentDate > EmergentScalingStartDate
    AND
    currentDate < lastSpcoEndDate (still time to order)

If emergent:
    1. Create sev2 ticket (or reuse existing one)
    2. Place SPCO with sev2 ticket as justification
    3. Update ticket with comment about what was ordered
    4. HOTWHelperModel.emergentDetails.isEmergent = true
    5. SNS notification includes emergent details

6. Key Formulas Used

Peak TPM (total input TPM):
    = Σ(InputTraffic sources for this fleet)
    Source types: Self, Upstream, CloudTune

BAU TPM:
    = Peak TPM ÷ (CT Peak Factor × Buffer Factor)
    Example: 10,000 ÷ (2.0 × 1.3) = 3,846 TPM

Required Hosts:
    = Peak TPM ÷ Host Throughput TPM × AZ Factor
    Example: 10,000 ÷ 570 × 1.125 = 19.7 → ceil → 20 hosts

Hosts Needed (to ORDER):
    = Required Hosts - (Current Hosts in Apollo + Pending Orders)
    Example: 20 - (14 + 3) = 3 more to order

SPCO Value per ASG:
    = ScalingPlanner recommendation for this ASG
    (distributed across multiple ASGs by RecommendDistribution algorithm)

Normal HOTW notification (`publishHardwareDetailsToSNS`):

{
    "FleetId": "FORTRESSService-X1-NA",
    "ServiceId": "FORTRESSService",
    "EventId": "PrimeDay26",
    "SimLink": "https://sim.amazon.com/issues/T-12345"
}
MessageAttributes:
    NotificationName = "HARDWARE_ORDER_DETAILS"
    ServiceEmailId = ["team@amazon.com", "owner@amazon.com", "poc@amazon.com"]

Atomic HOTW notification (`publishAtomicDetailsToSNS`):

{
    "FleetId": "FORTRESSService-X1-NA",
    "ServiceId": "FORTRESSService",
    "EventId": "PrimeDay26",
    "LatestVersionId": "v3",
    "PreviousVersionId": "v2",
    "SIMLink": "https://sim.amazon.com/issues/T-12345",
    "Owner": "johndoe",
    "POC": "janedoe",
    "EmergentDetails": {
        "IsEmergent": false,
        "EventName": "Prime Day 2026",
        "DaysToPeakGameDays": 15,
        "IsEmergentCloseToPeak": false
    }
}
MessageAttributes:
    NotificationName = "ATOMIC_HARDWARE_ORDER_DETAILS"
    ServiceEmailId = ["team@amazon.com", "owner@amazon.com", "poc@amazon.com"]

Message: "Sending SNS trigger for HardwareOrder milestone"
MessageAttributes:
    FleetId = "FORTRESSService-X1-NA"
    EventId = "PrimeDay26"
    WorkflowName = "HardwareOrder"

→ This triggers the MilestoneWorkflowHandler to mark HardwareOrder milestone as completed.

8. SIM Ticket Structure (What Gets Created)

The SIM ticket description looks like this in markdown:

We have created this SIM for PrimeDay26 hardware orders. EPIC has used this SIM to 
automatically place hardware orders for the above mentioned service.
Here is a [link](https://console.harmony.a2z.com/epic/Event/PrimeDay26/Dashboards) to EPIC.

| Fleet Id | Peak TPM | BAU TPM | CT Peak Factor | Buffer% | Required Hosts | Hosts Present in Apollo | Capacity Override Value | Pending Hosts Ordered By EPIC | FMC Order Details |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [FORTRESSService-X1-NA](https://console.harmony.a2z.com/epic/Event/PrimeDay26/...) | 10000 | 3846 | 2.08 | 1.3 | 20 | 14 | 17 | 3 | [FMC LINK](https://fmc.amazon.com/?...) |
|    |    |    |    |    |    |    |    |    |    |
| [OtherService-X2-EU](link) | 5000 | 2000 | 1.8 | 1.2 | 12 | 10 | 10 | 0 | [FMC LINK](url) |
|    |    |    |    |    |    |    |    |    |    |

9. Common Questions & Answers

Q: Why does the code sleep(5000)?

Apollo is a distributed config system. When EPIC calls updateApolloDetails, the data propagates across multiple nodes. Sleeping 5 seconds gives Apollo time to update, so the NEXT call to get fleet data returns fresh Apollo host counts.

Q: What is EAP?

EAP = Emergency Access Protocol (or similar). In the context of HOTW, it means the ASG (Auto Scaling Group) is enrolled in the ScalingPlanner system and can accept capacity override requests. If EAP is not enabled, HOTW can’t place orders for that ASG automatically.

Q: Why Math.ceil() for host counts?

Safety. If calculation says you need 19.3 hosts, you order 20 (round up). Never round down — you’d be short on capacity!

Q: What is “differential order”?

When the fleet was already in the SIM ticket and the host count DIDN’T change, it’s a “differential” = no new order needed. When isDifferentialOrder = true, the code skips sending the SNS notification.

Q: Why does getResult() have a finally block?

Even if HOTW fails partway through, the Apollo queue message and milestone SNS must still be sent:

Apollo queue: triggers Apollo to re-read fleet configs (even on failure, this cleans up)

Milestone SNS: the MilestoneWorkflowHandler uses this to know HOTW ran (success or fail)

Q: What is StringUtils.isNotBlank()?

From Apache Commons library. isNotBlank("hello") = true. isNotBlank("") = false. isNotBlank(" ") = false (spaces don’t count). isNotBlank(null) = false. It’s safer than !string.isEmpty() because it also handles null.

Q: Why two different API proxies (epicBackendApiProxy and hotwApiProxy)?

They point to different endpoints:

epicBackendApiProxy: main EPIC backend (fleet, service, event data)

hotwApiProxy: HOTW-specific endpoint (preferred ASG data, HOTW dashboard) Having separate proxies lets them potentially run on different infrastructure.

10. Internship Tips — Things to Focus On

What to understand deeply:

The HOTW formula: hostNeeded = projected - (inApollo + pending)
Why finally is critical: Apollo queue and milestone must always run
The builder pattern: Every model uses .builder()....build()
Lombok saves thousands of lines: @Data, @Builder replace manual getters/setters
Jackson maps JSON ↔ Java: @JsonProperty maps key names

What questions to ask your mentor:

“How is getRecommendedHostForEachAsg() in ScalingPlannerHelper implemented?”
“What happens in MilestoneWorkflowHandler after receiving the milestone SNS?”
“How is the HOTW triggered — is it CloudWatch Events or DynamoDB Streams?”
“What is the difference between FMC and SPCO orders?”
“How does the gradual scaling strategy work vs immediate?”

How to navigate the code:

To understand how HOTW is triggered:
    → Look at HotwHandler.java (in lambda/handler/ folder)

To understand ScalingPlanner:
    → Look at lambda/scalingplanner/ package

To understand milestone completion:
    → Look at lambda/milestone/ package

To understand Apollo updates:
    → Look at lambda/apollo/ package

To understand FMC orders:
    → Look at lambda/fmc/ package

11. Quick Reference Card

╔═══════════════════════════════════════════════════════════════════════╗
║                    HOTWHELPER QUICK REFERENCE                        ║
╠═══════════════════════════════════════════════════════════════════════╣
║                                                                       ║
║  DATA MODELS:                                                         ║
║    HOTWHelperModel        → Result of one fleet's HOTW               ║
║    HotwAllDetails         → Fleet TPM + order summary                ║
║    HotwExecutionAllDetails → Full audit record with ASG data         ║
║    CreateHotwRunDetails   → Create/update run record                 ║
║    AtomicHOTWRequestModel → Version IDs for atomic trigger          ║
║    PreferredASG           → Preferred ASG preference record          ║
║    DescaleRecommendation  → Post-event descale per ASG              ║
║    OrderDetails           → Simple per-fleet order summary          ║
║    TableConstants         → Column name strings                      ║
║                                                                       ║
║  UTILITIES:                                                           ║
║    HardwareOrdersUtil     → Calculate orders, build details          ║
║    TableUtil              → Build markdown tables for SIM            ║
║                                                                       ║
║  API CLIENT:                                                          ║
║    EPICBackendHotwApiCallsCommon → REST API calls to EPIC           ║
║                                                                       ║
║  CORE LOGIC:                                                          ║
║    HotwUpscalingHelper    → Main upscaling orchestrator             ║
║    DescaleRecommendation  → Post-event descaling recommendations    ║
║                                                                       ║
║  KEY FORMULA:                                                         ║
║    hostNeeded = projected - (inApollo + pendingOrders)               ║
║                                                                       ║
║  KEY ANNOTATIONS:                                                     ║
║    @Data        → getters + setters + equals + toString              ║
║    @Builder     → .builder()....build() pattern                      ║
║    @Jacksonized → @Builder + Jackson JSON deserialization            ║
║    @JsonProperty("Name") → map field to JSON key                    ║
║    @JsonIgnoreProperties → ignore unknown JSON fields               ║
║    @JsonInclude(NON_NULL)→ skip null fields in JSON output          ║
║                                                                       ║
╚═══════════════════════════════════════════════════════════════════════╝

🎉 Congratulations!

You have now read through:

✅ Layer 1: Java basics — classes, variables, methods, loops
✅ Layer 2: Java intermediate — exceptions, streams, lambdas, enums, builder pattern
✅ Layer 3: Libraries — Lombok, Jackson, AWS SDK
✅ Layer 4: Deep dive into every file in HotwHelper
✅ Layer 5: Full system flow and big picture

You are now equipped to:

Read and understand any Java file in this codebase
Explain what each class and method does
Navigate the dependency relationships between files
Ask intelligent questions during your internship
Start making small modifications with confidence

Good luck with your internship! 🚀