Zwing Docs

Staging Cluster Scheduled Downtime

Overview

To optimize resource usage and reduce costs, staging clusters are automatically scaled down during off-hours and scaled back up during business hours. This document explains the scheduled downtime activities and how to manage service exclusions.

Scheduled Activities

Scale Down (Suspension)

  • Schedule: Daily at 10:30 PM IST (17:00 UTC)
  • Action: Sets global.suspend: true for all enabled services
  • Workflow: Scheduled Scale Down
  • Effect: Services are suspended and scaled down to zero replicas

Scale Up (Resume)

  • Schedule: Daily at 8:30 AM IST (02:30 UTC)
  • Action: Sets global.suspend: false for all enabled services
  • Workflow: Scheduled Scale Up
  • Effect: Services are resumed and scaled back to their normal replica counts

Service Exclusion

If your service needs to remain available during scheduled downtime, you can exclude it from the automatic suspension by modifying the ops/suspend-targets.yaml file.

How to Exclude a Service

  1. Locate your service in ops/suspend-targets.yaml
  2. Set enabled: false for your service entry
  3. Create a Pull Request with your changes
  4. Wait for approval and merge

Example

To exclude the crm-backend/users service from downtime:

- name: crm-backend/users
  file: gitops-infra/values/staging/crm-backend/users.yaml
  key: global.suspend
  enabled: false # Changed from true to false

Important Notes

  • Services with enabled: false will be skipped during both scale-down and scale-up operations
  • The service will remain in its current state (suspended or active) regardless of the scheduled actions
  • Make sure to coordinate with your team before excluding services, as this affects resource usage

Manual Trigger

You can manually trigger the scale-up or scale-down actions at any time using GitHub Actions.

How to Manually Trigger

  1. Go to the Actions tab in the GitHub repository
  2. Select the workflow you want to run:
    • Scheduled Scale Down - to suspend services
    • Scheduled Scale Up - to resume services
  3. Click Run workflow button
  4. Select the branch (usually main or master)
  5. Click Run workflow to start the action

Use Cases for Manual Trigger

  • Emergency scale-up: If you need services available outside business hours for testing or debugging
  • Early scale-down: If you want to suspend services earlier than the scheduled time
  • Recovery: If scheduled actions failed and you need to retry

Understanding the Configuration

The ops/suspend-targets.yaml file contains a list of all services that participate in the scheduled downtime. Each entry has the following structure:

- name: service-name/component-name # Descriptive name for the service
  file: path/to/values/file.yaml # Path to the Helm values file
  key: global.suspend # YAML key to modify (usually global.suspend)
  enabled: true # Whether this service is included in scheduled actions

Current Services

The following services are currently configured for scheduled downtime:

  • Zwing services (console, butler, common, item-master, pos)
  • CRM backend services (apps, butler, cdp, common, dataout, fm, gatekeeper, gv, users, wallet)
  • CRM frontend
  • DIP backend services (app-connector, egress, inbound, ingress, outbound, pos-offline-sync, webhook)
  • DIP frontend
  • SSO backend services (auth, bulk-actions, common, enterprise, go-apps, license, queue-worker, users)
  • SSO frontend services (admin, customer)
  • Various other services (laravel-api, media-compression-service, notification services, etc.)

See the full list in ops/suspend-targets.yaml.

Troubleshooting

Service Still Suspended After Scale-Up Time

  1. Check if the service has enabled: false in suspend-targets.yaml - if so, it won't be automatically resumed
  2. Verify the GitHub Action ran successfully in the Actions tab
  3. Check the ArgoCD sync status to ensure changes were applied
  4. Manually trigger the scale-up workflow if needed

Service Suspended When It Shouldn't Be

  1. Verify the service entry in suspend-targets.yaml has enabled: false
  2. Check if your PR to exclude the service was merged
  3. Ensure the service wasn't manually suspended in the values file

Need Help?

  • Review the workflow logs in the GitHub Actions tab
  • Check the suspend-targets.yaml file for your service configuration
  • Contact the infrastructure team for assistance
Edit on GitHub

Setting Environment Variables

Previous Page