Guides
API ReferenceMCP Guides
Start typing to search…

Kubernetes (K8S) Cluster Onboarding - Ingestion & Storage (Agent Based)

This tutorial guides you through onboarding your Kubernetes cluster.

Introduction

The Kubernetes Cluster Onboarding process in CoreStack enables users to connect, monitor, and manage their Kubernetes environments for cost visibility and governance. This agent-based onboarding integrates cluster metrics with CoreStack’s FinOps capabilities to deliver accurate cost insights.

During onboarding, users configure cluster settings, choose a storage type (platform-managed or user-specific), and set up data ingestion. A CoreStack Kubernetes Agent is deployed within the cluster to collect utilization and performance metrics using Prometheus. These metrics are periodically uploaded to object storage (such as S3), from where CoreStack processes and visualizes cost and usage data.

Onboarding Pre-requisites

Before onboarding your Kubernetes cluster to CoreStack, ensure the following pre-requisites are met:

  1. Cluster and Access Requirements
    1. A functioning Kubernetes (K8s) cluster (e.g., AWS EKS) is available and accessible.
    2. The user performing onboarding has the required permissions to access the cluster and associated cloud account in CoreStack.
  2. Prometheus Configuration
    1. Prometheus must be running in the client environment to export node, pod, and container metrics.
    2. The Prometheus endpoint must be reachable from within the Kubernetes cluster by the K8s agent.
    3. Only one Prometheus instance per cluster is required to collect metrics.
    4. If security policies apply, ensure Prometheus certification details (path and certificate) are available for configuration.
    5. The following version list must be adhered to:
      • Prometheus v2.x or higher (Verified - v2.39.1)
      • Kube-State-Metrics: v2.9.0 or later (preferably v2.10.x)
      • cAdvisor: No separate installation needed — it is bundled with the Kubelet. Any Kubernetes version ≥1.20 provides a compatible cAdvisor endpoint.
  3. Network and Security Setup
    1. Whitelisting between the CoreStack agent cluster and the Prometheus endpoint must be completed to allow secure communication.
    2. Outbound connectivity from the cluster to CoreStack storage endpoints (e.g., S3) must be enabled for metric uploads.
  4. Storage Configuration
    1. Choose one of the following storage options during onboarding:
      • Platform-managed storage: Uses CoreStack’s shared S3 bucket.
      • User-specific storage: Uses customer-managed cloud storage with proper credentials.
  5. Kubernetes Agent Deployment
    1. The CoreStack Kubernetes Agent YAML file must be downloaded and deployed in the cluster.
  6. Temporary Storage Requirement in the Cluster
    1. The Kubernetes agent uses the /tmp directory as a temporary workspace to download data from Prometheus before uploading it to Amazon S3. This disk-based buffering mechanism helps prevent spikes in memory and CPU usage, even when the volume of data retrieved from Prometheus varies.
    2. Agent logs are also written temporarily to the same location. To manage disk usage, a predefined storage limit is enforced during each synchronization cycle. The agent validates the total log size and automatically removes older logs if the size exceeds 500 MB.
    3. Ensure that the /tmp directory is writable and has sufficient disk space to support temporary data files and logs.
  7. Additional Setup
    1. Have cluster details ready (Cluster ID, Cluster Type, Cloud Provider, Region).
    2. Define cost weightage for CPU, Memory, and GPU (total must equal 100%). [This is optional]
    3. Confirm all credentials and endpoints before proceeding with onboarding.

Kubernetes Cluster Onboarding Steps

Perform the following steps to onboard a Kubernetes cluster:

1. Navigate to "Governance"

On the left navigation pane, click Governance.

2. Click "Account Governance"

Click Account Governance.

3. Select "Container Services"

Click Container Services to access Kubernetes container management options.

Select 'Container Services'

4. Click "Onboard Cluster"

To onboard an AWS EKS cluster account, click Onboard Cluster.

Click 'Onboard Cluster'

5. Click "Onboard"

Click Onboard to proceed with cluster onboarding.

Click 'Onboard'

6. Alternative Step

Alternatively, on the Container Platform Accounts screen that displays the list of AWS EKS Clusters, under the Onboarding Status column, click Onboard corresponding to an appropriate cluster name.

Alternative Step

7. Select "Cloud Account"

In the Cloud Account drop-down list, click to select an account and click Ok.

Select 'Cloud Account'

8. Select a "Cluster"

In the Cluster drop-down list, click to select a cluster and then click Ok.

Select a 'Cluster'

9. View "Cluster ID" and "Cluster Type"

In the Cluster ID field, users can view the cluster they are accessing and in the Cluster Type field, they can view the cluster type.

View 'Cluster ID' and 'Cluster Type'

10. Click "Next"

Click Next to go to the next step.

Click 'Next'

11. Select and Manage Products

In the Select and Manage Products step, the Active Product(s) field shows FinOps. For the FinOps product to be active, ensure that the drop-down at left shows the option Active.

Select and Manage Products

12. Click "Next"

Click Next to go to the next step.

13. Storage Access Step

The Storage Access Step helps to configure the storage space from where the platform can access all the metrics that is uploaded by the agent. In the Storage Access step, in the Select Storage Access Type field, select one of the following options:

  • Platform managed storage: Use this option to store the metrics in the platform's S3 bucket. This is a shared storage.
  • User specific storage: Use this option if the user doesn't want to use the shared storage and wants to store the metrics details in a specific location.
Storage Access Step

14. Select "Platform managed storage"

If you select Platform managed storage, then no other information needs to be filled and you can proceed to the next step.

15. Select "User specific storage"

If you select this option User specific storage, then you need to select an appropriate option from the following:

  • Select the Cloud Account Onboarded with Product checkbox.
  • In the Select Authentication Protocol field, select either Assume Role or Access Key.
Select 'User specific storage'

16. Select "Cloud Account Onboarded with Product"

The Cloud Account Onboarded with Product checkbox can be used by users when the account is already onboarded with products. While selecting this option, users can specify the storage path where the metrics data will be stored.

If you select the option Cloud Account Onboarded with Product, then fill the following fields:

  • In the Select Cloud Account drop-down list, click to select a cloud account.
  • In the Select Storage Bucket drop-down list, click to select the storage bucket.
  • In the File Path box, enter the storage file path.
  • Click Save & Validate.
Select 'Cloud Account Onboarded with Product'

17. Select "Assume Role"

You can select the Assume Role field and fill the fields that appears. This option is used for temporary and secure access.

Select 'Assume Role'

18. Fill Assume Role Credentials

Fill details in the following fields and click Save & Validate.

  • Role ARN
  • External ID
  • In MFA Enabled field, select either True or False.
  • Select Storage Bucket
  • File Path
Fill Assume Role Credentials

19. Select "Access Key"

If you select the authentication protocol type as Access Key, then access key or secret key is used for authentication. The users must fill the following access key credentials and then click Save & Validate.

  • Access Key
  • Secret Key
  • Select Storage Bucket
  • File Path
Select 'Access Key'

20. Click "Next"

Click Next to proceed to the next step.

21. Enter "Prometheus Endpoint"

In the Prometheus Endpoint box, enter the path where metrics will be uploaded. The endpoint should be accessible by the K8s agent that will be deployed within the cluster.

Enter 'Prometheus Endpoint'

22. Select Certification Confirmation

If Prometheus certification is required, then select the "Prometheus certification is required for accessing Prometheus endpoint." checkbox and in the Path of the Certificate box, enter the path to the certificate.

Select Certification Confirmation

23. Install Kubernetes Agent

In the Install Kubernetes Agent section, you can click Download YAML to get the YAML code.

The YAML code is used to install the Kubernetes Agent for registering your Kubernetes cluster. The Kubernetes Agent is deployed at the cluster using the YAML code. After this, Prometheus (kube-state-metrics exporter) is used for metrics collection and the data is uploaded to S3 bucket periodically. Next, CoreStack collects the data from the storage, processes the details, and presents the cost insights in FinOps Dashboard for the end-users.

Install Kubernetes Agent

24. YAML Code

The YAML code consists of  docker image that is deployed as a container in the end-user’s Kubernetes cluster. After this code is downloaded in the end-user’s environment, it runs and communicates with CoreStack to fetch details that are to be uploaded. A Kubernetes cluster onboarded into CoreStack has a single Kubernetes container account and that in turn has a unique service account that helps the agent to communicate with CoreStack about the end-user configurations.

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-kube-agent-config
  namespace: default
  labels:
    app: custom-kube-agent
data:
  PROMETHEUS_ENDPOINT: "{{ PROMETHEUS_ENDPOINT }}"
  CLUSTER_SERVICE_ACCOUNT: "{{ CLUSTER_SERVICE_ACCOUNT }}"
  MANAGEMENT_ENDPOINT: "{{ MANAGEMENT_ENDPOINT }}"
---
apiVersion: v1
kind: Secret
metadata:
  name: custom-kube-agent-secret
  namespace: default
  labels:
    app: custom-kube-agent
data:
  CLUSTER_API_KEY: "{{ CLUSTER_API_KEY_BASE64 }}"
  PROMETHEUS_CERTIFICATE: "{{ PROMETHEUS_CERTIFICATE_BASE64 }}"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-kube-agent
  namespace: default
  labels:
    app: custom-kube-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom-kube-agent
  template:
    metadata:
      labels:
        app: custom-kube-agent
    spec:
      containers:
        - name: custom-kube-agent-container
          image: cscorestack/cs-kubernetes-agent:v1
          imagePullPolicy: Always
          ports:
            - containerPort: 9090
          env:
            - name: PROMETHEUS_ENDPOINT
              valueFrom:
                configMapKeyRef:
                  name: custom-kube-agent-config
                  key: PROMETHEUS_ENDPOINT
            - name: CLUSTER_SERVICE_ACCOUNT
              valueFrom:
                configMapKeyRef:
                  name: custom-kube-agent-config
                  key: CLUSTER_SERVICE_ACCOUNT
            - name: MANAGEMENT_ENDPOINT
              valueFrom:
                configMapKeyRef:
                  name: custom-kube-agent-config
                  key: MANAGEMENT_ENDPOINT
            - name: CLUSTER_API_KEY
              valueFrom:
                secretKeyRef:
                  name: custom-kube-agent-secret
                  key: CLUSTER_API_KEY
            - name: PROMETHEUS_CERTIFICATE
              valueFrom:
                secretKeyRef:
                  name: custom-kube-agent-secret
                  key: PROMETHEUS_CERTIFICATE
          livenessProbe:
            exec:
              command:
                - cat
                - /tmp/healthy

25. Kubernetes Agent Pre-requisites

  • Prometheus should be running in the client environment to export details about node, pod, and container metrics.
  • The endpoint should be accessible by the K8s agent that will be deployed within the cluster.
  • Each cluster requires a single Prometheus for it to fetch cost insights.
  • Whitelisting is required between the agent cluster and Prometheus to enable secure communication that allows the agent to fetch metrics from Prometheus and upload to S3.

26. Confirm Kubernetes Agent Installation

After the Kubernetes agent is installed, select the "I have installed Kubernetes Agent" checkbox. Note that the Kubernetes Agent can only be deployed by the Technical Support Team of CoreStack.

Confirm Kubernetes Agent Installation

27. Click "Next"

Click Next to go to the next step.

28. Advance Settings

In the Advance Settings step, fill the following details:

  • In the Cluster Description text box, type the cluster description.
  • In the Cost Resolution Frequency drop-down list, select an appropriate frequency.

29. Add Cost Weightage

In the Cost Weightage section, enter cost weightages for the following:

  • CPU Weight (%)
  • Memory Weight (%)
  • GPU Weight (%)

The total cost weightage must always be 100%. Cost is calculated based on the weightage percentages defined in this step.

Add Cost Weightage

30. Click "Finish"

Click Finish to complete the onboarding of a new Kubernetes cluster.

Click 'Finish'

The newly onboarded cluster will now appear on the Container Platform Accounts page.

Container Platform Accounts Page

The Container Platform Accounts page shows the Kubernetes cluster details. The top card on this page shows -- Accounts with Invalid Credentials, Deactivated Accounts, Not Onboarded Accounts, and Active and Governed Accounts. Count of each of these account categories are also shown next to it and users can also click the link provided at right side of each category to view the list of specific accounts.

Container Platform Accounts Page

Users can view the AWS EKS Cluster details in tabular format. The columns under which details are being displayed are -- Cluster Name, Cluster Type, Cloud Account, Cloud Provider, Region, Onboarding Status, Agent Status, Cluster Status in Cloud, Onboarded By, Created Date, and Actions.

View Cluster Details

Additional Actions

  • Users can use the Search box to search for specific details.
  • Use the Filter icon on the right side of the screen to hide/display the ADD+ filter. To apply a filter, click ADD+, select an option, and select values for it to apply them.
  • Users can also use the Download icon located on the right side of the screen to download cluster details.
Additional Actions

Actions on Cluster

Users can click the ellipses under the Actions column and take the following actions on an onboarded cluster:

  • Edit Configuration
  • View Configuration
  • Delete
  • Deactivate
Actions on Cluster

Edit Configuration

To edit or update onboarded cluster details, click the ellipses under the Actions column and select Edit Configuration.

Edit Configuration

The Edit Amazon EKS Container page appears and users can update the relevant details. Users can click Next to go to the next step and make updates. Users must ensure to click Finish to save the updates.

Edit Amazon EKS Container

View Configuration

To view cluster details, under the Actions column, click the ellipses and select View Configuration.

Select 'View Configuration'

The Details tab appears by default and users can go through the Basic Details, Storage Access, Deployment, and Advance Settings sections to view cluster-related details. You can also go to the FinOps tab to view the cost processing details.

View 'Details' Tab

Delete

To delete a cluster, under the Actions column, click the ellipses and select Delete.

Select 'Delete'

A dialog box appears and users can select Yes to delete the cluster account, else they can select No.

Select 'Yes' to Delete Cluster Account

Deactivate

To deactivate a cluster, under the Actions column, click the ellipses and select Deactivate.

Select 'Deactivate'

A dialog box appears and users can select Yes to deactivate the cluster account, else they can select No.

Select 'Yes' to Deactivate Cluster Account

You have successfully onboarded your Kubernetes cluster using CoreStack.

Updated 1 day ago