Overview

In this section, we'll help guide you in onboarding a Google Cloud Platform (GCP) Billing Account into CoreStack.

Pre-onboarding

GCP Projects can be onboarded as a Billing Account in CoreStack. Onboarding a Billing Account allows you to discover the cost information of all its linked GCP Projects.

However, before your GCP project can be onboarded into CoreStack, there are certain prerequisites that need to be set up.

Set Up Cloud Billing Data Export to BigQuery:

Please ensure, you need to have the billing data exported to bigquery dataset atleast before 24hrs.

To enable the Cloud Billing data export to BigQuery, refer to this GCP configuration guide.

GCP Project Permissions

The following permissions must be configured in your GCP Project before onboarding.

User Account Permissions:

• A user account must be created in GCP with the following permissions:
• Project Editor (View and Modify).
• Security Command Center Access: Either *Security Center Admin* or *Security Center Admin Viewer* role is required for security, vulnerability, and compliance.
• Operations Governance: Logging Admin & Pub/Sub Admin.
• Billing Account Admin (This is required for exporting the billing data to big query dataset. If the data is already exported this permission is not required)

API access:

• In GCP, enable API Access for the Cloud Resource Manager API, Recommender API, Cloud Billing API, and Security Command Center API in the API & Services – Library screen.

Service Account Permissions:

• A service account must be created in GCP with the following permissions:
• For Assessment: Project Viewer (Read only).
• For Assessment + Governance: Project Editor (View and Modify).
• Security Command Center Access: Either *Security Center Admin* or *Security Center Admin Viewer* role is required for security, vulnerability, and compliance.
• Operations Governance: Logging Admin & Pub/Sub Admin.

Billing Account Prerequisites:

• Schedule queries in the GCP BigQuery console.
• Create a Bucket for a BigQuery data transfer (under the same GCP Project where BigQuery exists).

📘

Note:

The Bucket, Dataset and the Scheduled Query created should be in the same location for the schedule query to execute successfully

📘

Note:

If threats and advanced security health analytics policies are required then the Security Command Center premium tier needs to be enabled.

Retrieving Onboarding Information from the GCP Console

Based on the authentication protocol being used in CoreStack (refer to the options below for guidance), certain information must be retrieved from the GCP console.

OAuth2 protocol:

The following values must be generated/copied from your GCP Project and configured in CoreStack.

Client ID & Client Secret:

1. Login to the GCP console.
2. Navigate to the Credentials screen.
3. Click Create credentials and select OAuth client ID.
4. Select Web application in the Application type field.
5. Specify the following URI in the Authorized redirect URIs by clicking the Add URI button:
URI

https://corestack.io/

6. Click the Create button. The Client ID and Client secret values will be displayed.

Scope:

The OAuth 2.0 scope information for a GCP project can be found at: https://www.googleapis.com/auth/cloud-platform.

Project ID:

The project ID is a unique identifier for a project and is used only within the GCP console.

1. Navigate to the Projects screen in the GCP console.
2. The Project ID will be displayed next to your GCP project in the project list.

Redirect URI:

The following redirect URI that is configured while creating the Client ID and Client Secret must be used:

Redirect URI

https://corestack.io/

Authorization Code:

The authorization code must be generated with user consent and required permissions.

1. Construct a URL in the following format:
Contructed URL

https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<Client ID>&redirect_uri=<Redirect URI>&scope=https://www.googleapis.com/auth/cloud-platform&prompt=consent&access_type=offline

2. Open a browser window in private mode (e.g. InPrivate, Incognito) and use it to access the above URL.
3. Login using your GCP credentials.
4. The page will be redirected to the Redirect URI, but the address bar will have the Authorization Code specified after code=.

📘

Note:

The values retrieved in the earlier steps can be used instead of <Client ID> and <Redirect URI> specified in the URL format.

Copy these details and provide them while onboarding your GCP Project into CoreStack when using the OAuth2 option.

Service Account protocol:

A service account must be created in your GCP Project. Then, you need to create a service account key and download it as a JSON file. Also, the Project ID must be retrieved from your GCP Project.

How to Download the Credentials File (JSON):

1. Navigate to the Credentials screen.
2. Click Create credentials and select Service account. The Create service account page appears.
3. Provide the necessary details to create a service account: Name, ID, and Description.
4. Click Create button.
5. Click Select a role to select the required roles.
6. Click the Continue button.
7. Click Create key.
8. Select JSON as the Key type.
9. Click the Create button. A JSON key file will be downloaded.
10. Click Done.

Project ID:

Refer to the steps in the Project ID topic of the OAuth2 Based section above.

Provide the JSON and Project ID while onboarding the GCP Project in CoreStack when using the Service Account option.

Additional Values from the Parent Billing Account:

In addition to the authentication protocol prerequisites explained previously, there are a few additional values that must be generated/copied from your parent GCP billing account and configured in CoreStack.

Bucket Name:

1. Login to the GCP console.
2. Navigate to the Storage - Browser screen.
3. Click Create bucket. The Create bucket screen appears.
4. Provide a unique value in the Name your bucket field along with the other details required to create the bucket.
5. Click the Create button.
6. Copy the value provided in the Name your bucket field.

Parent Billing Account ID:

1. Login to the GCP console.
2. Navigate to the Manage Billing Accounts screen.
3. Click My Projects. The list of projects will be displayed.
4. Copy the Billing Account ID for the required projects.

Provide these details in CoreStack for your Billing Account onboarding, along with either the OAuth2 or Service Account information explained above based on your Authentication Protocol selection.

Schedule Queries in GCP

Next, you need to schedule some queries in GCP. Navigate to the Schedule Query Page in the GCP console then follow the steps below:

1. Login to the GCP console.
2. Navigate to the GCP BigQuery console.
3. On the left menu, click Schedule Queries. The schedule queries list appears.



4. Click Create Schedule Queries located at the top of the page.



5. Copy each schedule query (Daily Schedule Query, Monthly Schedule Query, and On-demand Schedule Query).
6. For the Daily Schedule Query, schedule it on an hourly basis.



7. For the Monthly Schedule Query, schedule it on the fifth of every month.



8. For On-demand Schedule Query, run it in real time.



9. Select the Data location for the query to be executed.



📘

Note:

The Data location selected should be the same location same as selected for Bucket and the Dataset.



🚧

Note:

For the code snippets below, you need to insert your table id and bucket name.

Daily Schedule Query

Daily Schedule Query

DECLARE
unused STRING;
DECLARE
current_month_date DATE DEFAULT DATE_SUB(@run_date, INTERVAL 0 MONTH);
DECLARE
cost_data_invoice_month NUMERIC DEFAULT EXTRACT(MONTH
FROM
current_month_date);
DECLARE
cost_data_invoice_year NUMERIC DEFAULT EXTRACT(YEAR
FROM
current_month_date);
EXPORT DATA
OPTIONS ( uri = CONCAT('gs://<your bucket name>/', CAST(cost_data_invoice_year AS STRING), '-', CAST(current_month_date AS STRING FORMAT('MM')), '/', CAST(DATE(CURRENT_DATE()) as STRING FORMAT('YYYY-MM-DD')), '/*.csv'),
format='JSON',
overwrite=False) AS
SELECT
*, (SELECT STRING_AGG(display_name, '/') FROM B.project.ancestors) organization_list
FROM
`<complete table id>` as B
WHERE
B.invoice.month = CONCAT(CAST(cost_data_invoice_year AS STRING), CAST(current_month_date AS STRING FORMAT('MM'))) AND B.cost != 0.0

Monthly Schedule Query

Monthly Schedule Query

DECLARE
unused STRING;
DECLARE
current_month_date DATE DEFAULT DATE_SUB(@run_date, INTERVAL 1 MONTH);
DECLARE
cost_data_invoice_month NUMERIC DEFAULT EXTRACT(MONTH
FROM
current_month_date);
DECLARE
cost_data_invoice_year NUMERIC DEFAULT EXTRACT(YEAR
FROM
current_month_date);
EXPORT DATA
OPTIONS ( uri = CONCAT('gs://<your bucket name>/', CAST(cost_data_invoice_year AS STRING), '-', CAST(current_month_date AS STRING FORMAT('MM')), '_backfill/*.csv'),
format='JSON',
overwrite=True) AS
SELECT
*, (SELECT STRING_AGG(display_name, '/') FROM B.project.ancestors) organization_list
FROM
`<complete table id>` as B
WHERE
B.invoice.month = CONCAT(CAST(cost_data_invoice_year AS STRING), CAST(current_month_date AS STRING FORMAT('MM')))
AND B.cost != 0.0

On-demand Scheduled Query

On-demand Scheduled Query 1On-demand Scheduled Query 2On-demand Scheduled Query 3

DECLARE
unused STRING;
DECLARE
current_month_date DATE DEFAULT DATE_SUB(@run_date, INTERVAL 1 MONTH);
DECLARE
cost_data_invoice_month NUMERIC DEFAULT EXTRACT(MONTH
FROM
current_month_date);
DECLARE
cost_data_invoice_year NUMERIC DEFAULT EXTRACT(YEAR
FROM
current_month_date);
EXPORT DATA
OPTIONS ( uri = CONCAT('gs://<your bucket name>/', CAST(cost_data_invoice_year AS STRING), '-', CAST(current_month_date AS STRING FORMAT('MM')), '/*.csv'),
format='JSON',
overwrite=True) AS
SELECT
*, (SELECT STRING_AGG(display_name, '/') FROM B.project.ancestors) organization_list
FROM
`<Your complete Table Id goes here>` as B
WHERE
B.invoice.month = CONCAT(CAST(cost_data_invoice_year AS STRING), CAST(current_month_date AS STRING FORMAT('MM')))
AND B.cost != 0.0

DECLARE
unused STRING;
DECLARE
current_month_date DATE DEFAULT DATE_SUB(@run_date, INTERVAL 2 MONTH);
DECLARE
cost_data_invoice_month NUMERIC DEFAULT EXTRACT(MONTH
FROM
current_month_date);
DECLARE
cost_data_invoice_year NUMERIC DEFAULT EXTRACT(YEAR
FROM
current_month_date);
EXPORT DATA
OPTIONS ( uri = CONCAT('gs://<your bucket name>/', CAST(cost_data_invoice_year AS STRING), '-', CAST(current_month_date AS STRING FORMAT('MM')), '/*.csv'),
format='JSON',
overwrite=True) AS
SELECT
*, (SELECT STRING_AGG(display_name, '/') FROM B.project.ancestors) organization_list
FROM
`<your Complete Table id goes here>` as B
WHERE
B.invoice.month = CONCAT(CAST(cost_data_invoice_year AS STRING), CAST(current_month_date AS STRING FORMAT('MM')))
AND B.cost != 0.0

DECLARE
unused STRING;
DECLARE
current_month_date DATE DEFAULT DATE_SUB(@run_date, INTERVAL 3 MONTH);
DECLARE
cost_data_invoice_month NUMERIC DEFAULT EXTRACT(MONTH
FROM
current_month_date);
DECLARE
cost_data_invoice_year NUMERIC DEFAULT EXTRACT(YEAR
FROM
current_month_date);
EXPORT DATA
OPTIONS ( uri = CONCAT('gs://<your bucket name>/', CAST(cost_data_invoice_year AS STRING), '-', CAST(current_month_date AS STRING FORMAT('MM')), '/*.csv'),
format='JSON',
overwrite=True) AS
SELECT
*, (SELECT STRING_AGG(display_name, '/') FROM B.project.ancestors) organization_list
FROM
`Your complete Table Id goes here` as B
WHERE
B.invoice.month = CONCAT(CAST(cost_data_invoice_year AS STRING), CAST(current_month_date AS STRING FORMAT('MM')))
AND B.cost != 0.0

Onboarding your GCP Billing Account

The following steps need to be performed to onboard a GCP Billing Account into CoreStack:

1. Click the Add New button in the CoreStack dashboard and select Single Account.
2. Click Start Now.
3. Select the GCP option in the Public Cloud field.
4. Click the Get Started button.
5. Select the required option in the Access Type field. The options are: Assessment and Assessment + Governance.
6. Select the Billing Account option in the Account Type field.
7. Select the required option in the Authentication Protocol field. The options are: OAuth2 and Service Account.
8. Click Next.
9. Provide the necessary details explained in the Pre-onboarding section above based on the option selected in the Authentication Protocol field (Client ID, Client Secret, Scope, Project ID, Redirect URI, Authorization Code, Bucket Name, Billing Account ID, and Dataset ID OR Bucket Name, Billing Account ID, Dataset ID, Project ID, and Credentials File (JSON)).
10. Click the Validate button.
11. The Advanced Settings section will be displayed with additional fields (Name and Scope).
12. Modify the pre-populated name of the account in the Name field, if required.
13. Select the required option in the Scope field. The options are: Account, Private, and Tenant.
14. Click the I'm Done button.

After completing the previous steps, the GCP Project will be onboarded successfully into CoreStack. Any relevant insights and information about the resources available in the GCP Project will be populated under each of the cloud governance pillars available in CoreStack.

Monitoring and Logging Agent Metrics

When you install Cloud Monitoring and Cloud Logging agents on your virtual machine (VM) instances, they transmit data for the metric types listed below for monitoring:

Redis

• clients/blocked
• clients/connected
• commands/calls
• commands/total_time
• commands/usec_per_call
• keyspace/avg_ttl
• keyspace/keys
• keyspace/keys_with_expiration
• persistence/rdb/bgsave_in_progress
• replication/master/slaves/lag
• stats/memory/maxmemory
• stats/memory/system_memory_usage_ratio
• stats/memory/usage
• stats/memory/usage_ratio

Load_Balancing

• https/backend_latencies
• https/backend_request_bytes_count
• https/backend_request_count
• https/backend_response_bytes_count
• https/frontend_tcp_rtt
• https/internal/backend_latencies
• https/internal/request_bytes_count
• https/internal/request_count
• https/internal/response_bytes_count
• https/internal/total_latencies
• https/request_bytes_count

Cloud_Filestore

• nfs/server/average_read_latency
• nfs/server/average_write_latency
• nfs/server/free_bytes

To view Redis metrics, click Operations, then Monitoring & Alerts.



To view Load_Balancing metrics, click Operations, then Monitoring & Alerts.



To view Cloud_Filestore metrics, click Operations, then Monitoring & Alerts.