# AIVA: Enhanced Downscale Timer Observability and Control

This PRD focuses on adding robust logging, monitoring, and control mechanisms for the downscale timer, building upon the existing downscale logic.

### GPM-057: Implement Downscale Timer Logging
**Goal:** Log key events related to the downscale timer, providing insights into its operation.
**Acceptance Criteria:**
- A log message is generated when the downscale timer is started, including the configured `downscale_duration`.
- A log message is generated when the downscale timer expires and triggers a downscale action.
- A log message is generated when the downscale timer is reset (e.g., due to increased resource utilization).
- Log messages include relevant context, such as the current number of replicas and resource utilization metrics.
**Files:** `aiva/autoscaler.py`, `aiva/utils/logging.py` (if needed)
**Spawns:** Add metrics for downscale timer events to a monitoring system.

### GPM-058: Expose Downscale Timer as a Metric
**Goal:** Expose the downscale timer's remaining time as a metric to a monitoring system (e.g., Prometheus).
**Acceptance Criteria:**
- A metric is exposed that represents the remaining time on the downscale timer.  The metric should be named clearly (e.g., `aiva_downscale_timer_remaining_seconds`).
- The metric is updated regularly (e.g., every second) to reflect the current timer value.
- The metric is accessible through the configured monitoring endpoint.
**Files:** `aiva/autoscaler.py`, `aiva/metrics.py`
**Spawns:** Create alerts based on the downscale timer metric.

### GPM-059: Implement Downscale Timer Reset API Endpoint
**Goal:** Provide an API endpoint to manually reset the downscale timer.
**Acceptance Criteria:**
- A new API endpoint (e.g., `/api/v1/downscale/reset`) is created.
- A POST request to the endpoint resets the downscale timer to its initial `downscale_duration`.
- The API endpoint requires authentication/authorization.
- The API endpoint returns a success status code (e.g., 200 OK) upon successful reset.
- Logging occurs on successful and failed attempts to reset the timer.
**Files:** `aiva/api.py`, `aiva/auth.py`
**Spawns:** Implement a UI element to trigger the downscale timer reset.

### GPM-060: Add UI indication of Downscale Timer Status
**Goal:** Display the downscale timer status in the AIVA UI.
**Acceptance Criteria:**
- A visual indicator (e.g., a progress bar or countdown timer) is added to the AIVA UI to show the remaining time on the downscale timer.
- The indicator is updated in real-time to reflect the current timer value.
- The indicator is clearly labeled and easy to understand.
**Files:** `aiva/ui/src/components/AutoscalerDashboard.js` (example path, adjust to your project)
**Spawns:** Allow configuration of `downscale_duration` from the UI.

### GPM-061: Implement Graceful Shutdown During Downscale
**Goal:** Ensure a graceful shutdown of instances during the downscaling process.
**Acceptance Criteria:**
- Before terminating an instance during downscaling, AIVA sends a SIGTERM signal to the instance.
- AIVA waits for a configurable grace period (e.g., `shutdown_grace_period`) for the instance to shut down gracefully.
- If the instance does not shut down within the grace period, AIVA forcibly terminates it.
- The `shutdown_grace_period` is configurable via the application configuration.
**Files:** `aiva/autoscaler.py`
**Spawns:** Implement pre-downscale hooks to allow for custom actions before instance termination.

### GPM-062: Add Configuration Option for Downscale Timer Behavior
**Goal:** Allow configuration of the downscale timer's behavior (e.g., whether it restarts on small utilization fluctuations).
**Acceptance Criteria:**
- A new configuration option is added to control whether the downscale timer restarts on minor resource utilization fluctuations below a certain threshold.
- The configuration option defaults to a reasonable value (e.g., restart only if utilization drops significantly).
- The configuration option is clearly documented.
**Files:** `aiva/config.py`, `aiva/autoscaler.py`
**Spawns:** Implement a rolling restart strategy to minimize downtime during configuration changes.

### GPM-063: Implement Alerting on Downscale Timer Events
**Goal:** Configure alerts based on the downscale timer metric.
**Acceptance Criteria:**
- An alert is configured to trigger when the downscale timer is nearing expiration (e.g., 1 minute remaining).
- An alert is configured to trigger when the downscale timer expires and triggers a downscale action.
- Alerts are sent to the configured notification channels (e.g., Slack, email).
- Alert messages include relevant context, such as the current number of replicas and resource utilization metrics.
**Files:** `aiva/alerts.py`, `aiva/config.py`
**Spawns:** Implement automated rollback of downscale actions based on specific alert conditions.
