📕 Layer 4 — HotwHelper Deep Dive - Obsidian Developer Docs

Every file, every method, every line explained

The 14 Files — Quick Map
Data Model Files (9 files)
Utility Files
API Client File
Core Logic — HotwUpscalingHelper
Core Logic — DescaleHostRecommendationHelper
How Data Flows Through the Code

1. The 14 Files — Quick Map

HotwHelper/
│
├── ─── DATA MODELS (blueprints for data objects) ──────────────────────
│
├── TableConstants.java              → String constants for column names
├── AtomicHOTWRequestModel.java      → Input: version IDs for atomic HOTW
├── CreateHotwRunDetailsInput.java   → Input: create/update a HOTW run
├── HOTWHelperModel.java             → Output: result of running HOTW
├── HotwAllDetails.java              → Data: fleet details + order summary
├── HotwExecutionAllDetailsInput.java → Full execution record with ASG/SPCO data
├── OrderDetails.java                → Per-fleet order summary (for table)
├── PreferredASG.java                → Database record: preferred ASGs per fleet
├── DescaleRecommendationWithClb.java → Per-ASG descale recommendation + lower bound
│
├── ─── UTILITIES (helper logic) ───────────────────────────────────────
│
├── TableUtil.java                   → Build markdown tables for SIM tickets
├── HardwareOrdersUtil.java          → Build HOTW details, calculate orders, FMC
│
├── ─── API CLIENT ─────────────────────────────────────────────────────
│
├── EPICBackendHotwApiCallsCommon.java → Call EPIC backend HOTW REST APIs
│
├── ─── CORE LOGIC (main workflow) ─────────────────────────────────────
│
├── HotwUpscalingHelper.java         → MAIN: upscaling workflow orchestrator
└── DescaleHostRecommendationHelper.java → Post-event descaling recommendations

Inner Class	Stores
`HotwExecutionDetails`	TPM numbers, hosts needed, hosts in Apollo, SPCO sum
`CapacityOverrideDetails`	One SPCO (Capacity Override) placed
`FulfillmentDetail`	Hardware fulfillment status from AWS
`StatusOfHOTW` enum	SUCCESS / FAIL / PARTIAL_SUCCESS

A fleet may have many ASGs (Auto Scaling Groups) across different availability zones. Engineers can mark which ASGs to prefer for ordering — e.g., “order from the US-EAST-1a ASG first because it has better hardware available.”

How it’s used in HotwUpscalingHelper:

// Get the list of preferred ASGs (most recent first):
List<PreferredASG> preferredASGList =
    epicHotwApiCallsCommon.getPreferredASGList(fleetId, eventId);

if (preferredASGList.size() > 1) {
    // Compare latest preferred ASG set vs previous preferred ASG set
    String preferredASG = preferredASGList.get(0).getAsgs();         // latest
    String penUltimatePreferredASG = preferredASGList.get(1).getAsgs(); // second latest
    
    // Find ASGs that were in previous set but NOT in latest set
    // These are "decommissioned" ASGs — set them to BAU host count
    List<String> diffASGList = new ArrayList<>(penUltimatePreferredASGList);
    diffASGList.removeAll(latestPreferredASGList);
    // diffASGList = ASGs removed from preference (should go back to BAU level)
}

2.9 `DescaleRecommendationWithClb.java`

Purpose: Carries descale recommendation + lower bound for one ASG.

@Data @Jacksonized @Builder
@JsonInclude(JsonInclude.Include.NON_NULL)
@JsonIgnoreProperties(ignoreUnknown = true)
public class DescaleRecommendationWithClb {

    @JsonProperty("AsgCapacityLowerBound")
    Integer asgCapacityLowerBound;
    // ↑ Minimum hosts this ASG must always have (even after descaling)
    // Cannot go below this! Safety floor.
    
    @JsonProperty("AsgDescaleRecommendationHost")
    Integer asgDescaleRecommendationHost;
    // ↑ Recommended target host count after descaling
    // Will be ≥ asgCapacityLowerBound
}

Example:

Current hosts: 200
Required descale: remove 50 hosts
AsgCapacityLowerBound: 120

Recommended host count = 200 - 50 = 150
Since 150 > 120 (lower bound) → safe to descale to 150 ✅

If: Required descale: remove 100 hosts
   Recommended = 200 - 100 = 100
   But 100 < 120 (lower bound) → cannot do this descale! ❌
   → Report as failure

3. Utility Files

3.1 `TableUtil.java` — Building Markdown Tables

Purpose: Creates properly-formatted markdown tables for SIM ticket descriptions.

What is a markdown table?

| Fleet Id | Peak TPM | BAU TPM | Required Hosts |
| --- | --- | --- | --- |
| [FleetA-NA](https://epic-link) | 10000 | 4807 | 200 |
|    |    |    |    |

This renders as a formatted table in SIM (Amazon’s ticket system).

Method 1: getDescriptionHeader(String eventId)

public static StringBuilder getDescriptionHeader(String eventId) {
    // Creates the main SIM ticket description with a table header
    
    String header = "We have created this SIM for " + eventId + " hardware orders..."
    + " Here is a [link](...) to EPIC ...\n\n";
    
    // Returns: header text + table column headers + separator row
}

Method 2: getCommentHeader(String fleetId, String eventId)

public static StringBuilder getCommentHeader(String fleetId, String eventId) {
    // Used when adding a COMMENT to an existing SIM (update to existing ticket)
    String header = "Here is the updated SPCO and FMC order details for **FleetA-NA**:\n\n";
    // Returns: header text + table column headers + separator row
}

Method 3: getTableBuilder(String header, String eventId) (private)

private static StringBuilder getTableBuilder(String header, String eventId) {
    StringBuilder tableBuilder = new StringBuilder();
    tableBuilder.append(header).append("\n");
    
    // Build column headers using StringJoiner:
    StringJoiner headerJoiner = new StringJoiner(" | ", "| ", " |");
    // prefix="| "  separator=" | "  suffix=" |"
    
    headerJoiner
        .add(TableConstants.FLEET_ID)         // "Fleet Id"
        .add(TableConstants.PEAK_TPM)         // "Peak TPM"
        .add(TableConstants.BAU_TPM)          // "BAU TPM"
        .add(TableConstants.CT_Peak_Factor)   // "CT Peak Factor"
        .add(TableConstants.Buffer_PERCENTAGE)// "Buffer%"
        .add(TableConstants.REQUIRED_HOSTS)   // "Required Hosts"
        .add(TableConstants.HOSTS_PRESENT_IN_APOLLO)  // "Hosts Present in Apollo"
        .add(TableConstants.SPCO_VALUE);      // "Capacity Override Value"
    
    // HVE events (special events) have fewer columns:
    if (!CommonUtil.isHveEvent(eventId)) {
        headerJoiner
            .add(TableConstants.PENDING_HOSTS_ORDERED_BY_EPIC)
            .add(TableConstants.FMC_ORDER_DETAILS);
        columnCount = 11;
    }
    
    // Build separator row: | --- | --- | --- | ...
    StringJoiner separatorJoiner = new StringJoiner(" | ", "| ", " |");
    for (int i = 0; i < columnCount; i++) {
        separatorJoiner.add(TableConstants.DASH_SEPARATOR);  // "---"
    }
    
    tableBuilder.append(headerJoiner.toString()).append("\n");
    tableBuilder.append(separatorJoiner.toString());
    return tableBuilder;
}

Result of getTableBuilder:

| Fleet Id | Peak TPM | BAU TPM | CT Peak Factor | Buffer% | Required Hosts | Hosts Present in Apollo | Capacity Override Value | Pending Hosts Ordered By EPIC | FMC Order Details |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

Method 4: descriptionFormatter(StringBuilder, HotwAllDetails, String)

// Adds ONE DATA ROW to the table:
public static void descriptionFormatter(
    StringBuilder tableBuilder,  // table being built
    HotwAllDetails details,      // fleet data
    String eventId) {
    
    // Build the row:
    String row = String.join(" | ",
        details.getFleetDetails().getFleetId(),          // "[FleetA-NA](link)"
        String.valueOf(details.getFleetDetails().getPeakTpm()),     // "10000"
        String.valueOf(details.getFleetDetails().getBauTpm()),      // "4807"
        String.valueOf(details.getFleetDetails().getCtFactor()),    // "2.08"
        String.valueOf(details.getFleetDetails().getBufferFactor()), // "1.3"
        String.valueOf(details.getFleetDetails().getHostRequired()), // "200"
        String.valueOf(details.getFleetDetails().getMaxHostsInApollo()), // "180"
        details.getSpcoValue()                           // "20"
    );
    
    if (!CommonUtil.isHveEvent(eventId)) {
        row = String.join(" | ", row,
            details.getPendingHardwareOrdered(),  // "5"
            fmcOrderDetails                       // "[FMC LINK](url)"
        );
    }
    
    tableBuilder.append("| ").append(row).append(" |\n");
    // adds empty row for spacing:
    tableBuilder.append("| ").append("    | ".repeat(columnCount - 1)).append("   |\n");
}

Result — the full table with data:

| [FleetA-NA](https://...) | 10000 | 4807 | 2.08 | 1.3 | 200 | 180 | 20 | 5 | [FMC LINK](url) |
|    |    |    |    |    |    |    |    |    |    |

Method 5: getFmcLink(...) — build FMC URL

public static String getFmcLink(
    String fleetApolloName,   // "FORTRESSService/NA/X1/Prod"
    String regionName,        // "us-east-1"
    Map<String, List<String>> hardwareOrderTags) {  // event-specific tags
    
    List<String> taglist = hardwareOrderTags.get(REGIONS_NAME_MAPPING.get(regionName));
    
    Map<String, List<String>> queryParams = new HashMap<>();
    queryParams.put("tab", List.of("ec2"));                    // tab=ec2
    queryParams.put("tags[]", taglist);                         // tags[]=PrimeDay26_NA
    queryParams.put("environment[]", List.of(fleetApolloName)); // environment[]=FORTRESSService/NA/X1/Prod
    
    return generateDynamicURL(EndpointConstants.FMC_URL, queryParams);
    // Result: https://fmc.amazon.com/?tab=ec2&tags[]=PrimeDay26_NA&environment[]=FORTRESSService/NA/X1/Prod
}

Method 6: generateDynamicURL(...) — build URL with query params

public static String generateDynamicURL(
    String baseUrl,
    Map<String, List<String>> queryParams) {
    
    // Convert Map to URL query string using streams:
    String queryString = queryParams.entrySet()
        .stream()
        .flatMap(entry -> {
            String key = entry.getKey();
            List<String> values = entry.getValue();
            // For each value: create "key=value" string
            return values.stream().map(value -> key + "=" + value);
        })
        .collect(Collectors.joining("&"));
    //                               ↑ join all with "&"
    
    URI uri = new URI(baseUrl + "?" + queryString);
    // Result: "https://fmc.url?tab=ec2&tags[]=tagA&environment[]=apolloName"
    
    return uri.toString();
}

3.2 `HardwareOrdersUtil.java` — Hardware Order Calculations

Purpose: The computational workhorse — calculates hardware order numbers, builds HOTW details, interacts with ScalingPlanner.

Key Design — Singleton-like Factory:

@AllArgsConstructor
public class HardwareOrdersUtil {
    private LambdaLogger logger;
    private static Map<String, ScalingPlannerApiServiceProxy> scalingPlannerApiServiceProxyByRegion;
    //              ↑ static — shared across all instances
    
    // Creates one proxy per AWS region (us-east-1, eu-west-1, etc.)
    private static void initialize(LambdaLogger logger) {
        scalingPlannerApiServiceProxyByRegion = new HashMap<>();
        scalingPlannerApiServiceProxyByRegion.put(US_EAST_1,
            LambdaEnvironment.getScalingPlannerApiServiceProxy(logger, US_EAST_1));
        scalingPlannerApiServiceProxyByRegion.put(EU_WEST_1,
            LambdaEnvironment.getScalingPlannerApiServiceProxy(logger, EU_WEST_1));
        // ... more regions ...
    }
    
    // Factory method — always initializes before returning instance:
    public static HardwareOrdersUtil getInstance(LambdaLogger logger) {
        initialize(logger);             // set up region proxies
        return new HardwareOrdersUtil(logger);
    }
}

Method: getFleetDetails(Fleet, Event, Service)

public HotwAllDetails.FleetDetails getFleetDetails(Fleet fleet, Event event, Service service) {
    FleetProjectionItem projection = fleet.getFleetProjection()[0];
    FleetConfiguration config = fleet.getFleetConfiguration();
    
    // Step 1: Get total peak TPM
    int totalInputTpm = projection.getInputTPM() != null ? projection.getInputTPM() : 0;
    
    // Step 2: Get buffer factor used
    double bufferUsed = getBufferUsed(projection);
    // getBufferUsed reads "BufferUsed" from the projection's TPM properties
    
    // Step 3: Get CT (CloudTune) peak factor for this region
    double ctFactor = CommonUtil.getPreferredCTPFForService(
        service,
        REGIONS_NAME_MAPPING.get(fleet.getFleetConfiguration().getRegion()),
        event
    );
    
    // Step 4: Calculate BAU TPM (reverse from peak)
    double bauTpm = calculateBauTpm(totalInputTpm, ctFactor, bufferUsed);
    // bauTpm = peakTpm / (ctFactor * bufferUsed)
    
    // Step 5: Get projected hosts (already calculated by EPIC)
    double projectedHosts = projection.getProjectedHosts();
    double hostThroughputTPM = config.getHostThroughputTPM();
    double maxHostCount = config.getMaxHostCount() != null ? config.getMaxHostCount() : 0.0;
    
    // Step 6: Build and return FleetDetails object
    return HotwAllDetails.FleetDetails.builder()
        .peakTpm(totalInputTpm)
        .bauTpm((int) bauTpm)
        .bufferFactor(bufferUsed)
        .ctFactor(ctFactor)
        .maxHostsInApollo((int) maxHostCount)
        .hostRequired((int) projectedHosts)
        .hostThroughPutTPM((int) hostThroughputTPM)
        .build();
}

Method: buildHOTWDetails(Fleet, Event, Service, ...)

This is the BIG method that aggregates all data for one fleet:

public HotwAllDetails buildHOTWDetails(
    Fleet fleet, Event event, Service service,
    ScalingPlannerHelper scalingPlannerHelper,
    Map<String, List<String>> hardwareOrderTags,
    HotwExecutionAllDetailsInput hotwExecutionAllDetailsInput) {
    
    // 1. Get numerical fleet details (TPM, hosts):
    HotwAllDetails.FleetDetails fleetDetails = getFleetDetails(fleet, event, service);
    
    // 2. Build HotwAllDetails with just fleet details initially:
    HotwAllDetails hotwAllDetails = HotwAllDetails.builder().fleetDetails(fleetDetails).build();
    
    // 3. Set the fleetId as a clickable markdown link:
    String fleetLink = buildFleetLink(event.getEventId(), service.getServiceId(), fleet.getFleetId());
    hotwAllDetails.getFleetDetails().setFleetId("[" + fleetId + "](" + fleetLink + ")");
    // Result: "[FleetA-NA](https://console.harmony.a2z.com/epic/Event/PrimeDay26/...)"
    
    // 4. Build the FMC link:
    String fmcLink = getFmcLink(fleet, hardwareOrderTags);
    
    // 5. Query ScalingPlanner for total hardware ordered and SPCO values:
    Map<String, Integer> totalOrderMap = getTotalHardwareOrderForEvent(
        scalingPlannerHelper, fleet.getFleetId(), event.getEventId(),
        asgList, scalingPlannerProxy, hotwExecutionAllDetailsInput
    );
    
    // 6. Set the remaining fields:
    hotwAllDetails.setFmcOrderDetails("[FMC LINK](" + fmcLink + ")");
    hotwAllDetails.setSpcoValue(String.valueOf(totalOrderMap.get(TableConstants.SPCO_VALUE)));
    hotwAllDetails.setPendingHardwareOrdered(
        String.valueOf(totalOrderMap.get(TableConstants.PENDING_HOSTS_ORDERED_BY_EPIC)));
    
    // 7. Update the audit record:
    updateHotwExecutionAllDetails(hotwExecutionAllDetailsInput, ...);
    
    return hotwAllDetails;
}

Method: getTotalHardwareOrderForEvent(...)

Queries ScalingPlanner API to count how many hosts have been ordered:

// For each ASG in the fleet:
for (AutoScalingGroupDataModel asg : asgList) {
    
    // Get ScalingPlanner info for this ASG:
    AsgScalingPlannerInfo asgInfo = scalingPlannerHelper.getAsgInfoResponse(fleetId, eventId, asg.getAsgTag());
    
    // Build the capacity override name: "asgName_PrimeDay26"
    String capacityOverrideName = extractAsgName(asgInfo.getAsgInfo().getArn()) + "_" + eventId;
    
    // Fetch all capacity overrides and their fulfillment details:
    List<CapacityOverrideWithFulfillmentDetails> fulfillmentDetails =
        getCapacityOverrideWithFulfillmentDetails(proxy, asgInfo, capacityOverrideName);
    
    for (CapacityOverrideWithFulfillmentDetails override : fulfillmentDetails) {
        spcoValue += override.getCapacityOverride().getOverrideValue();  // add to SPCO total
        
        for (CapacityOverrideFulfillmentDetails fulfillment : override.getFulfillmentDetails()) {
            totalHardwareOrdered += fulfillment.getFulfillmentValue();   // actual hosts ordered
            pendingOrders += calculatePendingHardwareOrder(fulfillment); // pending orders
        }
    }
}

Method: calculatePendingHardwareOrder()

private int calculatePendingHardwareOrder(
    CapacityOverrideFulfillmentDetails fulfillment) {
    
    // Is the order still pending (not yet approved/fulfilled)?
    if (PENDING.equals(fulfillment.getFulfillmentStatus())
        || PENDING_BUSINESS_APPROVAL.equals(fulfillment.getFulfillmentStatus())
        || PENDING_FINANCIAL_APPROVAL.equals(fulfillment.getFulfillmentStatus())
        || PENDING_REGIONAL_APPROVAL.equals(fulfillment.getFulfillmentStatus())) {
        
        return fulfillment.getFulfillmentValue();  // count these hosts as "pending"
    }
    return 0;  // not pending (completed or cancelled)
}

4. API Client File

`EPICBackendHotwApiCallsCommon.java`

Purpose: Makes HTTP API calls to the EPIC backend for HOTW-related operations.

Pattern used — Private Constructor + Static Factory:

@AllArgsConstructor(access = AccessLevel.PRIVATE)  // private constructor
public class EPICBackendHotwApiCallsCommon {
    private final EPICBackendApiProxy epicBackendApiProxy;  // HTTP client
    private final LambdaLogger logger;
    
    // Public factory method:
    public static EPICBackendHotwApiCallsCommon getInstance(
        EPICBackendApiProxy proxy, LambdaLogger logger) {
        return new EPICBackendHotwApiCallsCommon(proxy, logger);
    }
}

API call methods:

Method	HTTP	Path	Purpose
`createOrUpdateHotwRunDetails()`	POST	`/hotw/runDetail`	Create HOTW run record
`updateHotwRunDetails()`	PUT	`/hotw/runDetail`	Update run status
`createHotwExecutionDetails()`	POST	`/hotw/executionDetail`	Save execution details
`createHotwDashboardWithAsgDetails()`	POST	`/hotw/dashboardDetail`	Save dashboard data
`getAsgDetails()`	GET	`/hotw/asgDetail/{fleetId}`	Get ASG details
`getHOTWExecutionHistoryForFleetId()`	GET	`/hotw/executionDetail/{eventId}/{fleetId}`	Get history
`getPreferredASGList()`	GET	`/hotw/asgsPreference/{eventId}/{fleetId}`	Get preferred ASGs

How each method works:

public void createHotwExecutionDetails(
    HotwExecutionAllDetailsInput input) {
    try {
        // 1. Convert Java object to JSON string:
        String jsonInput = JsonUtil.toJson(input);
        
        logger.log("Create Input Json: " + jsonInput);
        
        // 2. Call EPIC backend API:
        String response = this.epicBackendApiProxy
            .apigPostCall("/hotw/executionDetail", jsonInput);
        //                ↑ path              ↑ JSON body
        
        logger.log("Response: " + response);
        
    } catch (Exception e) {
        e.printStackTrace();
        logger.log("Unable to create Hotw execution details entry");
        throw new IllegalStateException("The erroneous response received", e);
        //         ↑ wrap original exception and re-throw
    }
}

Special method: getLatestHOTWExecutionDetailsForEventFleet()

// This one uses a SQL QUERY via the RDS execute endpoint!
public List<HotwExecutionDetailsWithRunId> getLatestHOTWExecutionDetailsForEventFleet(
    Integer fleetIndexId, Integer eventIndexId) {
    
    // Build the SQL query:
    String sql = QueryUtil.queryForFetchingLastHOTWExecutionDetailsForEventFleet(
        fleetIndexId, eventIndexId);
    // SQL looks like: SELECT * FROM hotw_execution WHERE fleet_id = ? AND event_id = ?
    
    // Wrap it in a request object:
    ExecuteQueryRequest queryRequest = ExecuteQueryRequest.builder()
        .query(sql)
        .build();
    
    // Call the executeQuery endpoint (which runs SQL on RDS):
    String response = this.epicBackendApiProxy
        .apigPutCall("/rds/executeQuery", JsonUtil.toJson(queryRequest));
    
    // Deserialize response to list of objects:
    return JsonUtil.parseJson(response,
        new TypeReference<List<HotwExecutionDetailsWithRunId>>() {});
}

5. Core Logic — `HotwUpscalingHelper.java`

This is the most complex file — the main orchestrator for HOTW upscaling.

Constructor — Setting Up Dependencies

public HotwUpscalingHelper(Context context) {
    // Step 1: Get the logger (for CloudWatch logs)
    this.logger = context.getLogger();
    
    // Step 2: Initialize environment (reads Lambda env variables)
    LambdaEnvironment.initialize();
    stage = LambdaEnvironment.getSTAGE();   // "beta", "gamma", "prod"
    region = LambdaEnvironment.getREGION(); // "us-east-1"
    
    // Step 3: Create API proxy clients
    this.epicBackendApiProxy = LambdaEnvironment.getEpicBackendApiProxy(context);
    this.hotwApiProxy = LambdaEnvironment.getHOTWApiProxy(context);
    this.fmcApiServiceProxy = LambdaEnvironment.getFMCApiServiceProxy(logger, stage);
    
    // Step 4: Create business logic helpers (all are factory-method style)
    this.epicBackendFleetApiCallsCommon = 
        EPICBackendFleetApiCallsCommon.getInstance(epicBackendApiProxy, logger);
    this.epicBackendApiCallsCommon =
        EPICBackendApiCallsCommon.getInstance(epicBackendApiProxy, logger);
    this.milestoneCommons = new MilestoneCommons(logger);
    this.epicBackendApiCallsMilestone =
        EPICBackendApiCallsMilestone.getInstance(epicBackendApiProxy, logger);
    this.recommendDistribution = new MinChangeWeightsDistribution();
    this.scalingPlannerHelper = ScalingPlannerHelper.getInstance(
        logger, epicBackendApiCallsCommon, recommendDistribution);
    this.fmcHelper = FMCHelper.getInstance(logger, epicBackendApiProxy, epicBackendApiCallsCommon);
    this.apolloHelper = ApolloHelper.getInstance(logger, epicBackendApiProxy, epicBackendApiCallsCommon);
    this.epicBackendServiceApiCallsCommon =
        EPICBackendServiceApiCallsCommon.getInstance(epicBackendApiProxy, logger);
    this.epicBackendTicketApiCallsCommon =
        EPICBackendTicketApiCallsCommon.getInstance(epicBackendApiProxy, logger);
    
    // Step 5: Create AWS SDK clients
    this.maxisAPIServiceProxy = LambdaEnvironment.getMaxisApiServiceProxy(logger);
    this.ticketyAPIServiceProxy = LambdaEnvironment.getTicketyAPIServiceProxy(logger);
    this.hardwareOrdersUtil = HardwareOrdersUtil.getInstance(logger);
    this.sns = AmazonSNSClientBuilder.defaultClient();
    this.sqs = AmazonSQSClientBuilder.defaultClient();
    
    // Step 6: Create HOTW-specific API client
    this.epicBackendHotwApiCallsCommon =
        EPICBackendHotwApiCallsCommon.getInstance(epicBackendApiProxy, logger);
    this.epicHotwApiCallsCommon =
        EPICBackendHotwApiCallsCommon.getInstance(hotwApiProxy, logger);
    //  ↑ Note: uses hotwApiProxy (different base URL) — for reading preferred ASGs
    
    // Step 7: Sleep strategy and SQS queue
    this.sleepStrategy = LambdaEnvironment.getSleepStrategy();
    this.queueUrl = sqs.getQueueUrl(APOLLO_QUEUE_NAME).getQueueUrl();
}

Public Method 1: `handle()` — Normal Weekly HOTW

public void handle(String fleetId, String eventId, String runId) {
    logger.log("HOTW upscaling workflow invoked for fleetId: " + fleetId);
    try {
        // Step 1: Get latest fleet data
        Fleet fleet = epicBackendFleetApiCallsCommon.callGetLatestEventFleet(fleetId, eventId);
        
        // Step 2: Validate fleet data is complete
        validateFleetData(fleet);
        
        // Step 3: Refresh FMC and Apollo data
        updateFMCAndApolloDetails(fleetId, eventId);
        
        // Step 4: Wait for Apollo data to update
        this.sleepStrategy.sleep(5000);   // sleep 5 seconds
        
        // Step 5: Run the main workflow (calculate + order)
        HOTWHelperModel result = getResult(fleetId, eventId, runId, null);
        
        // Step 6: Send SNS notification (only if hosts are needed and not differential)
        if (!result.isDifferentialOrder() && result.getHostNeeded() > 0) {
            publishHardwareDetailsToSNS(
                result.getService(), fleetId, eventId, result.getHardwareOrderSimLink()
            );
        }
        
    } catch (Exception e) {
        logger.log("Error in HOTW workflow for FleetId:" + fleetId + " Error:" + e);
        e.printStackTrace();
    }
}

Public Method 2: `handleAtomic()` — Triggered by Fleet Version Change

public void handleAtomic(String fleetId, String eventId, String runId,
    String previousVersionId, String currentVersionId) {
    try {
        // Step 1: Get the event — skip if it's BAU (not a peak event)
        Event event = epicBackendApiCallsCommon.callGetEvent(eventId);
        if (EVENT_TYPE_BAU.equals(event.getEventType())) {
            logger.log("Skipping HOTW for BAU event");
            return;   // ← early return, skip everything else
        }
        
        // Step 2: Get BOTH versions of the fleet
        Fleet previousFleet = epicBackendFleetApiCallsCommon
            .callGetFleetObjectWithVersion(fleetId, previousVersionId);
        Fleet currentFleet = epicBackendFleetApiCallsCommon
            .callGetFleetObjectWithVersion(fleetId, currentVersionId);
        
        // Step 3: Calculate the DELTA (change in required hosts)
        int deltaRequiredHost =
            currentFleet.getFleetProjection()[0].getProjectedHosts().intValue()
            - previousFleet.getFleetProjection()[0].getProjectedHosts().intValue();
        // deltaRequiredHost = current - previous
        // Positive → need more hosts
        // Negative → need fewer hosts
        // Zero → no change
        
        // Step 4: Validate current fleet data
        validateFleetData(currentFleet);
        
        // Step 5: Update FMC and Apollo data
        updateFMCAndApolloDetails(fleetId, eventId);
        this.sleepStrategy.sleep(5000);
        
        // Step 6: Run main workflow
        HOTWHelperModel result = getResult(fleetId, eventId, runId, currentVersionId);
        
        // Step 7: Only notify if there was an actual change in hosts needed
        if (deltaRequiredHost != 0) {
            publishAtomicDetailsToSNS(
                result.getService(), fleetId, eventId,
                previousVersionId, currentVersionId,
                result.getHardwareOrderSimLink(),
                result.getEmergentDetails()
            );
        }
        
    } catch (Exception e) {
        logger.log("Error in atomic HOTW for FleetId:" + fleetId);
        e.printStackTrace();
    }
}

Private Method: `getResult()` — The Core Logic

This is the heart of HOTW. Let me walk through it step by step:

private HOTWHelperModel getResult(String fleetId, String eventId, String runId, String versionId)
    throws Exception {
    try {
        // ══════ STEP 1: GET FLEET DATA ══════
        Fleet fleet = epicBackendFleetApiCallsCommon.callGetLatestEventFleet(fleetId, eventId);
        Fleet versionfleet = null;
        int projectedHost;
        String user = AUTO;  // "AUTO" = HOTW ran automatically
        
        if (versionId != null) {
            // Atomic HOTW: use specific version
            versionfleet = epicBackendFleetApiCallsCommon
                .callGetFleetObjectWithVersion(fleetId, versionId);
            projectedHost = versionfleet.getFleetProjection()[0].getProjectedHosts().intValue();
            user = versionfleet.getAuditMetadata().getUser();  // who made the change
        } else {
            // Normal HOTW: use latest projection
            projectedHost = fleet.getFleetProjection()[0].getProjectedHosts().intValue();
        }
        
        // ══════ STEP 2: CLASSIFY HOST TYPES ══════
        // Find what type of hosts the fleet uses (e.g., r5.xlarge, c5.2xlarge)
        Map<String, ArrayList<String>> hostTypeWithAsg = new HashMap<>();
        // hostTypeWithAsg = { "r5.xlarge": ["asg-1a", "asg-1b"], "c5.2xlarge": ["asg-2a"] }
        
        Map<String, Integer> sortedMap = CommonUtil.getHostType(fleet, hostTypeWithAsg);
        // sortedMap = { "r5.xlarge": 150, "c5.2xlarge": 30 }  (sorted by count, desc)
        
        // Get the MOST COMMON host type:
        Map.Entry<String, Integer> maxEntry = sortedMap.entrySet().iterator().next();
        String hostType = maxEntry.getKey();   // "r5.xlarge" — the dominant host type
        
        // Get the ASG list for the dominant host type:
        List<String> asgListOfRequiredHostType = hostTypeWithAsg.get(hostType);
        // ["asg-1a", "asg-1b"]
        
        // ══════ STEP 3: CALCULATE HOSTS NEEDED ══════
        Integer maxHostInApollo = fleet.getFleetConfiguration().getMaxHostCount().intValue();
        // Current hosts running in Apollo (e.g., 180)
        
        // Get pending host orders (already ordered but not delivered):
        Optional<HostOrderStatus> hostOrderStatusOpt =
            Arrays.stream(fleet.getHostOrderStatuses()).findFirst();
        Integer totalPendingHost =
            getPendingApprovalHostOrders(hostOrderStatusOpt.get().getHostsPendingApproval())
            + hostOrderStatusOpt.get().getHostsPendingDelivery();
        // totalPendingHost = pending approval + pending delivery (e.g., 10)
        
        // KEY FORMULA: how many more hosts do we need to order?
        Integer hostNeeded = projectedHost - (maxHostInApollo + totalPendingHost);
        // Example: 200 projected - (180 in Apollo + 10 pending) = 10 more needed
        
        // Safety check — don't order 1000+ at once:
        if (hostNeeded >= 1000) {
            throw new IllegalStateException("More than 1000 hosts ordering for:" + fleetId);
        }
        
        // Including pending orders in total recommendation:
        Integer hostNeededWithPending = hostNeeded
            + getPendingApprovalHostOrders(hostOrderStatusOpt.get().getHostsPendingApproval())
            + hostOrderStatusOpt.get().getHostsPendingDelivery();
        
        // ══════ STEP 4: GET PER-ASG RECOMMENDATIONS ══════
        // Ask ScalingPlanner: "how to distribute hostNeededWithPending across ASGs?"
        Map<String, Integer> asgToHostRecommendationMap =
            scalingPlannerHelper.getRecommendedHostForEachAsg(
                fleetId, eventId, hostType, hostNeededWithPending, asgListOfRequiredHostType);
        // Result: { "asg-1a": 8, "asg-1b": 12 }  (distribute 20 hosts across 2 ASGs)
        
        // ══════ STEP 5: HANDLE PREFERRED ASG CHANGES ══════
        // Check if preferred ASGs changed recently (some ASGs removed from preference):
        List<PreferredASG> preferredASGList = epicHotwApiCallsCommon.getPreferredASGList(fleetId, eventId);
        
        if (preferredASGList.size() > 1) {
            // Compare latest vs previous preferred ASG list
            // Removed ASGs should go back to BAU host count
            // New/preferred ASGs get the new recommendations
            // (complex logic to merge these two maps)
            // ... (see full code for details) ...
        }
        
        // ══════ STEP 6: HANDLE HOST TYPE MIGRATION ══════
        // Some fleets are migrating from old host type to new host type:
        if (FLEET_TO_PREFERRED_HOST_TYPE_MAP.containsKey(fleetId)) {
            // Old host type: keep at BAU count (remove extra from old type)
            // New host type: give all the recommendations
            // (merge both maps)
        }
        
        // ══════ STEP 7: CHECK EMERGENT ORDERING ══════
        // Is today past the emergent scaling start date?
        Boolean isStuckInPendingApproval = fmcHelper.getIfFMCOrderNotCompletedInAsgList(...);
        
        String sev2TicketId = checkAndCreateSev2TicketIfNeeded(
            event,
            checkIfDeltaHostsPositive(hostNeeded, totalPendingHost),
            isStuckInPendingApproval
        );
        // sev2TicketId = "" if not emergent, or actual ticket ID if emergent
        
        // ══════ STEP 8: PLACE CAPACITY OVERRIDES (SPCO) ══════
        String hardwareOrderSimLink = fetchArnUpdateSPCOAndReturnSIMLink(
            fleet, event, service, asgToHostRecommendationMap, ...
        );
        // This places the actual SPCO orders in AWS ScalingPlanner
        // Returns the SIM ticket link for tracking
        
        // ══════ STEP 9: UPDATE FMC ORDERS ══════
        fmcHelper.updateFMCorders(fleetId, eventId, fmcApiServiceProxy,
            StringUtils.isNotBlank(sev2TicketId));  // isEmergent flag
        
        // ══════ STEP 10: UPDATE SIM TICKET ══════
        boolean isDifferentialOrder = updateHardwareOrderSimDetails(
            service, fleet, hardwareOrderSimLink, hostNeeded, totalPeakFmcOrders, event, ...
        );
        
        // ══════ STEP 11: SAVE AUDIT RECORDS ══════
        epicBackendHotwApiCallsCommon.createHotwExecutionDetails(hotwExecutionAllDetailsInput);
        epicBackendHotwApiCallsCommon.updateHotwRunDetails(
            CreateHotwRunDetailsInput.builder()
                .eventId(eventId)
                .runId(Integer.parseInt(runId))
                .orderType(StringUtils.isNotBlank(sev2TicketId) ? "Emergent" : "Standard")
                .build()
        );
        
        // ══════ BUILD AND RETURN RESULT ══════
        HOTWHelperModel.EmergentDetails emergentDetails = HOTWHelperModel.EmergentDetails.builder()
            .isEmergent(StringUtils.isNotBlank(sev2TicketId))
            .eventName(event.getEventName())
            .build();
        setEmergentDetailsRemainingParameters(emergentDetails, event);
        
        return HOTWHelperModel.builder()
            .hostNeeded(hostNeeded)
            .service(service)
            .hardwareOrderSimLink(hardwareOrderSimLink)
            .isDifferentialOrder(isDifferentialOrder)
            .fleet(fleet)
            .emergentDetails(emergentDetails)
            .build();
        
    } catch (Exception e) {
        e.printStackTrace();
        throw e;  // re-throw so caller knows it failed
        
    } finally {
        // ══════ ALWAYS RUNS (even if exception occurred) ══════
        // Push to Apollo queue (refreshes Apollo config):
        SendMessageRequest apolloMessage = ApolloHelper.getMessageRequestForApolloQueue(
            fleetId, eventId, queueUrl, 600  // 600 second delay
        );
        sqs.sendMessage(apolloMessage);
        
        // Trigger milestone check (mark HardwareOrder milestone as done):
        this.publishToMilestoneSNSTopic(fleetId, eventId);
    }
}

Private Method: `validateFleetData()`

private void validateFleetData(Fleet fleet) {
    // NULL CHECKS — exit if any critical data is missing:
    if (fleet == null                                               // no fleet at all?
        || fleet.getFleetProjection() == null                      // no projection?
        || fleet.getFleetProjection().length == 0                  // empty array?
        || fleet.getFleetProjection()[0] == null                   // null first element?
        || fleet.getFleetConfiguration() == null                   // no config?
        || fleet.getFleetConfiguration().getHostThroughputTPM() <= 1  // invalid TPM?
        || fleet.getFleetProjection()[0].getInputTPM() <= 5        // near-zero traffic?
        || fleet.getFleetProjection()[0].getProjectedHosts() == null  // no host projection?
        || fleet.getFleetProjection()[0].getTotalInputTPMProperties() == null) {
        
        throw new IllegalArgumentException(
            "Invalid fleet data, fix fleet details for fleetID: " + fleet.getFleetId()
        );
    }
    
    // BUSINESS RULES:
    if (!REGISTERED.equals(fleet.getFleetType())) {
        // Only "Registered" fleets can have hardware orders
        throw new IllegalStateException("FleetType is not Registered for: " + fleet.getFleetId());
    }
    
    // Customer must have verified the traffic link:
    if (!Boolean.TRUE.equals(
        fleet.getFleetProjection()[0].getTotalInputTPMProperties()
            .getTotalInputTPMLinkVerifiedByCustomer())) {
        throw new IllegalStateException("TotalInputTPMLink not verified by customer");
    }
}

Private Method: `fetchArnUpdateSPCOAndReturnSIMLink()`

This places the actual SPCO (hardware orders) in AWS ScalingPlanner:

// For each ASG and its recommended host count:
for (Map.Entry<String, Integer> entry : asgToHostRecommendationMap.entrySet()) {
    String asgTag = entry.getKey();        // "asg-1a"
    Integer hostForThisAsg = entry.getValue(); // 8 (hosts to order for this ASG)
    
    // Skip if 0 hosts needed:
    if (hostForThisAsg == 0) continue;
    
    // Check lower bound — can't go below minimum:
    int asgCapacityLowerBound = asgScalingPlannerInfo.getAsgCapacityLowerBound();
    if (hostForThisAsg < asgCapacityLowerBound) {
        // Can't place order below lower bound!
        isLowerBoundIssue = true;
        continue;
    }
    
    // Build the scaling strategy (gradual vs immediate):
    ScalingStrategy scalingStrategy = ScalingStrategyFactory.getScalingStrategy(...);
    List<CapacityOverrideDataModel> overrides = scalingStrategy.getCapacityOverrideDataModels();
    
    // Build SPCO request:
    PutCapacityOverrideInput spcoRequest = PutCapacityOverrideInput.builder()
        .arn(asgScalingPlannerInfo.getAsgInfo().getArn())  // AWS ARN of the ASG
        .justification(hardwareOrderSimLink)                // SIM ticket link
        .overrides(overrides)                              // scaling plan
        .placeOrderIfRequired("true")                      // yes, order hardware!
        .requester("EPIC")                                 // who is requesting
        .build();
    
    // Send to ScalingPlanner API:
    String response = scalingPlannerHelper.getScalingPlanOverrideV2ResultResponse(spcoRequest);
    logger.log("SPCO response for ASG " + asgTag + ": " + response);
}

Private Method: `checkAndCreateSev2TicketIfNeeded()`

private String checkAndCreateSev2TicketIfNeeded(
    Event event, Boolean isDeltaHostsPositive, Boolean isStuckInPendingApproval) {
    
    // Only create emergent ticket if past the emergent start date:
    if (checkIfCurrentDayPastEmergentScaleStartDate(event)) {
        
        if (isDeltaHostsPositive || isStuckInPendingApproval) {
            // Check if a sev2 ticket already exists:
            List<Ticket> existingTickets = epicBackendTicketApiCallsCommon.callGetOpenTicket(
                event.getEventId(), EPIC_SERVICE, EPIC_SERVICE, EMERGENT_ORDERING_PURPOSE
            );
            String ticketId = existingTickets.isEmpty() ? "" : existingTickets.get(0).getTicketId();
            
            if (StringUtils.isBlank(ticketId)) {
                // No existing ticket → create new sev2:
                return createSev2PendingTicketOnEPICCTI(event);
            } else {
                return ticketId;  // use existing ticket
            }
        }
    }
    return "";  // not emergent → empty string
}

6. Core Logic — `DescaleHostRecommendationHelper.java`

Purpose: After a peak event ends, recommends which hosts to remove (descale) per ASG.

`handle(String fleetId, String eventId)`

public Map<String, DescaleRecommendationWithClb> handle(String fleetId, String eventId) {
    // Returns: { "asg-1a": {recommendedHosts: 80, lowerBound: 50} }
    
    Map<String, Integer> asgToDescaleMap = new HashMap<>();
    Map<String, DescaleRecommendationWithClb> result = new HashMap<>();
    
    try {
        // Step 1: Refresh Apollo data
        Integer maxHostInApollo =
            apolloHelper.fetchAndUpdateApolloDetails(apolloLambdaFunction, fleetId, eventId);
        
        // Step 2: Get fleet
        Fleet fleet = epicBackendFleetApiCallsCommon.callGetLatestEventFleet(fleetId, eventId);
        
        // Step 3: Get required descale hosts (how many to remove)
        Optional<FleetProjectionItem> projectionOpt =
            Arrays.stream(fleet.getFleetProjection()).findFirst();
        Integer requiredDescaleHosts = projectionOpt.get().getRequiredDescaleHosts();
        
        // Step 4: Validate
        if (requiredDescaleHosts == null || requiredDescaleHosts == 0) {
            return result;  // nothing to descale
        }
        if ((maxHostInApollo - requiredDescaleHosts) < 0) {
            return result;  // can't descale more than we have
        }
        
        // Step 5: Calculate per-ASG descale recommendation
        // Target: maxHostInApollo - requiredDescaleHosts
        // Example: 200 current - 50 to remove = 150 target
        asgToDescaleMap = descaleHostRecommendation(fleet, maxHostInApollo - requiredDescaleHosts);
        
        // Step 6: Get CLB (Capacity Lower Bound) values from EPIC backend
        List<EAPInputDetails.ASGDetail> eapDetails =
            epicBackendHotwApiCallsCommon.getAsgDetails(fleetId);
        
        // Step 7: For each ASG, combine recommendation + lower bound:
        for (EAPInputDetails.ASGDetail asgDetail : eapDetails) {
            String asgTag = asgDetail.getAsgTag();
            int clb = asgDetail.getAsgCapacityLowerBound();
            
            if (asgToDescaleMap.containsKey(asgTag)) {
                int recommendation = asgToDescaleMap.get(asgTag);
                result.put(asgTag, DescaleRecommendationWithClb.builder()
                    .asgDescaleRecommendationHost(recommendation)
                    .asgCapacityLowerBound(clb)
                    .build());
            }
        }
        
        return result;
        
    } catch (Exception e) {
        logger.log("Error in descale recommendation for:" + fleetId);
        e.printStackTrace();
        return result;  // return empty map — don't crash the caller
    }
}

`descaleHostRecommendation(Fleet, Integer)` — The Algorithm

private Map<String, Integer> descaleHostRecommendation(Fleet fleet, Integer targetHostCount) {
    // targetHostCount = maxHostsInApollo - hostsToRemove
    // We need to distribute these remaining hosts across ASGs
    
    Map<String, Integer> asgWithCount = new HashMap<>();
    Map<String, Integer> asgToDescale = new HashMap<>();
    
    // Start with current host counts per ASG:
    for (AutoScalingGroupDataModel asg : fleet.getFleetConfiguration().getAsgProperties()) {
        asgWithCount.put(asg.getAsgTag(), asg.getHostCount());
    }
    // asgWithCount = { "asg-1a": 100, "asg-1b": 60, "asg-1c": 40 }
    
    int hostsToRemove = getTotalHosts(asgWithCount) - targetHostCount;
    // Remove hosts one by one from the largest ASG each time
    // (greedy algorithm: always pick the largest to keep things balanced)
    
    if (hostsToRemove > 0) {
        while (hostsToRemove > 0) {
            // Find the ASG with the most hosts:
            String maxKey = "";
            int maxValue = 0;
            for (Map.Entry<String, Integer> entry : asgWithCount.entrySet()) {
                if (entry.getValue() > maxValue) {
                    maxValue = entry.getValue();
                    maxKey = entry.getKey();
                }
            }
            
            if (maxValue == 0) break;  // no more hosts to remove
            
            // Remove one host from largest ASG:
            asgWithCount.put(maxKey, asgWithCount.get(maxKey) - 1);
            hostsToRemove--;
            asgToDescale.put(maxKey, asgWithCount.get(maxKey));
        }
    }
    
    return asgToDescale;
    // Result: { "asg-1a": 75, "asg-1b": 45 }  (recommended counts after descale)
}

Example walkthrough:

Before: asg-1a=100, asg-1b=60, asg-1c=40  (Total=200)
Target: 150 hosts (remove 50)

Round 1: Remove from asg-1a (largest 100) → asg-1a=99
Round 2: Remove from asg-1a (still largest 99) → asg-1a=98
...
Round 41: Remove from asg-1a (60=asg-1b) → now asg-1a=60, asg-1b=60
Round 42: Remove from asg-1a (tie, picks first alphabetically) → asg-1a=59
...eventually balances to: asg-1a=67, asg-1b=50, asg-1c=33  (Total=150)

7. How Data Flows Through the Code

HotwHandler.java (Lambda entry point — outside HotwHelper)
    │
    │  Creates HotwUpscalingHelper(context)
    ▼
HotwUpscalingHelper.handle(fleetId, eventId, runId)
    │
    ├─── validateFleetData(fleet)
    │       checks nulls, registered type, customer verification
    │
    ├─── updateFMCAndApolloDetails(fleetId, eventId)
    │       └── fmcHelper.updateFMCorders(...)
    │       └── apolloHelper.fetchAndUpdateApolloDetails(...)
    │
    ├─── sleep(5000ms)  [wait for Apollo update to propagate]
    │
    └─── getResult(fleetId, eventId, runId, null)
            │
            ├── Get fleet + calculate projectedHost
            ├── CommonUtil.getHostType() → sortedMap + hostTypeWithAsg
            ├── Calculate hostNeeded = projected - (inApollo + pending)
            ├── scalingPlannerHelper.getRecommendedHostForEachAsg()
            │       → asgToHostRecommendationMap
            ├── Handle PreferredASG changes (if any)
            ├── Handle host type migration (if any)
            ├── checkAndCreateSev2TicketIfNeeded()
            │       → sev2TicketId (empty or actual ticket ID)
            ├── fetchArnUpdateSPCOAndReturnSIMLink()
            │       ├── For each ASG: check EAP status
            │       ├── If all EAP enabled:
            │       │       ├── getSIMLink() → hardware order SIM link
            │       │       └── For each ASG: place PutCapacityOverride (SPCO)
            │       └── Return simLink
            ├── fmcHelper.updateFMCorders()
            ├── updateHardwareOrderSimDetails()
            │       ├── If fleet already in SIM: add comment with updated data
            │       └── If fleet new to SIM: update SIM description + create ticket entry
            ├── createHotwExecutionDetails() → save audit record
            ├── updateHotwRunDetails() → update run record
            ├── Build HOTWHelperModel and return
            │
            └── [FINALLY ALWAYS RUNS]:
                    ├── sqs.sendMessage → Apollo queue (trigger Apollo refresh)
                    └── publishToMilestoneSNSTopic → trigger milestone check

Back in handle():
    └── publishHardwareDetailsToSNS() → notify service owner via SNS/email

→ Continue to Layer 5: Full System Flow

📕 Layer 4 — HotwHelper Deep Dive

Every file, every method, every line explained

Table of Contents

1. The 14 Files — Quick Map

2. Data Model Files

2.1 `TableConstants.java`

2.2 `AtomicHOTWRequestModel.java`

2.3 `CreateHotwRunDetailsInput.java`

2.4 `HOTWHelperModel.java`

2.5 `HotwAllDetails.java`

2.6 `HotwExecutionAllDetailsInput.java`

2.7 `OrderDetails.java`

2.8 `PreferredASG.java`