PierCloud FinOps Platform - Lighthouse Configurations

Configuration Process for Lighthouse Product

Welcome to Pier Cloud!

Elaborated to provide clear and precise guidance, this guide covers the configuration of the AWS Lighthouse module, aiming to facilitate the understanding and execution of the necessary steps to start using it.

Lighthouse Configuration Flow

  1. How to extract roles from the Pier Platform

  2. Integrating with AWS

  3. Creating the Billing Connector on the Pier Platform

  4. Best Practices for Billing Data Formatting

1. Creating the PierCloud Role

How to extract roles from the Pier Platform:

To configure the PierCloud role in the AWS console, you’ll need a YAML file, which can be downloaded directly from the Pier Cloud platform.

Once the file is saved to your desktop, follow the steps below.

This step must be performed on all existing accounts for the billing role.

On the Pier Cloud Platform:

  1. Within the platform, under the Amazon Web Services context, access the user icon, as shown below:

  1. Select Settings, as illustrated. The first step is to download the YAML files available on the Pier Platform:

  1. In this step, click on AWS Roles in the bottom-left menu, then find the desired role — in this case, we will select the “lighthouse-piercloud” role — and click on the icon on the right to download the role, as shown:

This file must be saved on the client’s computer for later use in the AWS environment.

In PoC cases, this role will be provided by the sales team to assist with initial configuration.

If you want to rename the role, simply click the pencil icon next to the role name.

2. Integrating with AWS

In the AWS environment:

  1. From the Services menu, choose CloudFormation.

  1. Click on Stacks.

  1. Click Create Stack.

  1. Choose the option: Template is ready, then Upload a template file. Locate the .yml file, downloaded from the Pier Cloud platform in step 1, and click Next after upload.

  1. Enter the name registered in Lighthouse for the stack and, under parameters, enter the bucket name, then click Next.

  1. If you want to assign any tags to this role, you can do so on this page. Otherwise, scroll to the end and click Next.

  1. Check the box to confirm resource creation and click Create Stack.

  1. CloudFormation will begin creating the PierCloud role. After a few minutes, you should see CREATE_COMPLETE in the status column.

  1. That's it, the configuration is complete.

Below is a summary of the information that must be sent to Pier Cloud:

  • Role: billing-clientname

    • S3 Bucket Name

    • Organization Account ID

    • Path – where the files are stored

    • File type in the bucket – csv/parquet

Note: This role must be configured in the Organization Account

CUR 1.0 Configuration

This step should only be performed if CUR has not yet been enabled in your organization.

Creating AWS Cost and Usage Reports (CUR)

AWS Cost and Usage Reports monitor your usage and provide estimated charges associated with your account.

The reports contain line items for each unique combination of AWS services, usage type, and operation used by your AWS account. You can customize the AWS Cost and Usage Reports to aggregate the information by hour, day, or month.

Considerations before configuration:

If CUR (Cost and Usage Report) has already been configured, there's no need to redo it. However, if the current configuration is not in Parquet format or not hourly, we recommend reconfiguring it.

The bucket structure for objects must follow the format below:

  • First layer of folders named with the year number, e.g., year=2024

  • Second layer of folders named with the month number, e.g., month=5

For configuration, follow these steps:

  1. Log in to the Billing and Cost Management Console at: Console AWS Billing

  2. In the navigation pane, select Cost and Usage Reports.

  3. Choose Create Report

  4. In Report name, choose the name of your report..

  5. In Report additional content, select Include resource IDs to include the IDs of each individual resource in the report

Including resource IDs creates individual line items for each of your resources. This can significantly increase the size of your cost and usage report files, based on your AWS usage.

  1. Select Split Cost Allocation Data to include detailed cost and usage of shared resources (Amazon ECS only).

Including split cost allocation data creates individual line items for each of your resources (i.e. ECS tasks). This can significantly increase the size of your cost and usage report files, based on your AWS usage.

For data update settings, select whether you want AWS cost and usage reports to be updated if AWS applies refunds, credits, or support fees to your account after your bill is finalized. When a report is updated, a new report is sent to Amazon S3.

  1. Click Next

  2. Under S3 bucket, click Configure.

  3. In the Configure S3 Bucket dialog box, do the following:

    1. Select an existing bucket.

    2. Select Create a bucket, enter a bucket name, and choose the region where you want to create a new bucket.

  4. Review the bucket policy, select The following default policy will be applied to your bucket, and choose Save.

  5. In "Report path prefix", inform the report path prefix that you want to add before the report name.

  6. For Time granularity, choose one of the following options

    1. Hourly (By Hour) if you want the line items in the report to be aggregated by hour.

    2. Daily (by day) if you want the line items in the report to be aggregated by day.

    3. Monthly if you want the line items in the report to be aggregated by month.

  7. Para Report versioning (Versionamento de relatórios), escolha se deseja que cada versão do relatório seja substituída pela versão mais recente do relatório, ou seja, entregue além das versões anteriores.

  8. For Report versioning, choose whether you want each report version to be replaced by the latest report version or to be delivered in addition to previous versions.

Replacing reports can save on Amazon S3 storage costs. Delivering new versions of reports can improve the auditability of billing data over time.

  1. In Enable reporting data integration for, select whether you want to enable your cost and usage reports to integrate with Amazon Athena, Amazon Redshift, or Amazon QuickSight. The report is packaged in the following formats:

    1. Athena: parquet format

    2. Amazon Redshift ou Amazon QuickSight: compressão.gz

  2. Choose Next.

  3. After reviewing your report settings, choose Review and Complete.

Note: The report is accessed through the lighthouse-piercloud role. Once complete, the data loading and processing flow is accessed by Pier Cloud, and then it will be possible to create billing and consumption views through the Pier Cloud platform. Read the official AWS documentation here.

CUR 2.0 Configuration

Only required if upgrading to CUR 2.0

  1. In the AWS console, search for CUR and select Cost and Usage Reports.

  1. The system will display the "Cost and Usage Reports" screen. In the side menu, select the Data Exports option, as shown below:

  1. On the Exports and Dashboards screen, click Create, as illustrated below:

  1. In the Create Export screen:

  • Under Export Details, select Standard data export

  • Export name: Define your name

  • Under Data table content settings, choose CUR 2.0

  • Under Additional export content, select:

    • Include resource ID

    • Split cost allocation data

    • Time granularity: Hourly

  • Under Data export delivery options, select:

    • Compression type and file format: Parquet - Parquet

    • File versioning: Overwrite existing data export file

  • In Data export storage settings, select the options:

    • In S3 bucket: When selecting the configure option, the system will display the "Configure S3 bucket" screen, which must be completed, as illustrated below:

      • In Create a bucket or select an existing bucket, select "Create a bucket" if you don't have a bucket created. If you already have one, you can choose "Select existing bucket". In this case, we'll use "Select existing bucket."

      • In S3 bucket name, select the name of the bucket you want to use.

      • Check the "I agree to overwrite my S3 bucket policy" box.

      • Then click "Select bucket".

The system will return to the main Create export screen, where you must enter the name of your data export, according to AWS name restrictions:

  • S3 path prefix: Enter the name you want to assign to the path, according to existing naming restrictions.

Finally, click on the Create option, so that your export is created:

  1. On the Data Exports screen, the system will display the created CUR:

  1. By clicking on the bucket name, as illustrated above, the system will direct you to the bucket where a folder will be created containing the name of the Path used in "S3 path prefix".

It can take up to 24 hours for the folder to be created within the bucket and for CUR to begin populating the bucket with billing data. If retroactive data needs to be populated, you'll need to contact AWS.

3. Creating the Billing Connector on the Pier Platform

On the Pier Cloud Platform:

  1. Click the User Profile icon and select Settings:

  1. In the side menu, select the "Billing Connectors" option:

  1. The system will display the Billing Connector main screen, where you can add a new Billing connector by clicking on the "Add Connector" option:

  1. The screen for registering new Connectors will be displayed, where you will need to enter the following information:

  • Type: Select the default option

  • Cloud Provider: Select the AWS (Amazon Web Services) provider

  • Billing Connector Identification: Inform the name that identifies the billing connector

  • AWS Account ID: Account identification ID contains AWS billing information

  • Data source for daily query:

    • Name of the S3 Bucket that contains the billing data information: Enter the name of the bucket used in the S3 Bucket configuration when creating the CUR.

    • Path containing the billing data: Enter the path that contains the billing data within the AWS S3 bucket.

    • Enter the AWS CUR version: Select the CUR 2.0 version.

    • Type of file in bucket: Enter the type of file stored in the S3 bucket, specified when creating the CUR.

  1. Finally, after filling in all the requested information, clicking the Save button, the system will return to the Billing Conector main screen, where you will be able to view the registered Billing.

4. Best Practices for Billing Data Formatting

For data to be sent correctly from the CUR, it is necessary to configure the reports with the information formatted below.

  1. Layer of folders named with year number, e.g., year=2022

  1. Second layer of month folders named with month number, e.g., month=5

  1. .Parquet file for that month with the data for the entire month, even if partitioned into as many parts as necessary.

That’s it! You’ve completed all the steps to configure Lighthouse using your AWS provider.

PreviousPierCloud FinOps Platform - CCA ConfigurationsNextPierCloud FinOps Platform - Spot Configurations

Last updated