AI & Data
Communications & Media

Data Lakehouse: Enhanced Compliance & Capabilities

Enabled data intelligence across the enterprise with critical cloud architecture and compliance enhancements to the existing data lakehouse, which Trility helped deploy in a previous engagement. This project expanded capabilities beyond the data intelligence team, and allowed other areas of business to fully leverage the data lakehouse.

Problem Statement

The client had a contractual obligation to provide specific reporting capabilities to a customer. The development of those capabilities also offered a valuable opportunity to enhance analytics across the enterprise.

The first engagement with Trility required deploying the data lakehouse in an air-gapped environment due to a key government customer with classified information.

The client had capacity and expertise gaps and hired Trility to return and securely deliver these high-priority capabilities and enable future efforts.

Solution Approach

Using a strike team approach, Trility collaborated with stakeholders to identify, define, and prioritize the backlog. Key aspects included:

  • Enabling data intelligence features through data lake architecture

  • Enhancing tagging enforcement capabilities for enterprise standards

  • Centralizing authentication/authorization for data lake environments

  • Deploying data lakes and setting up GeoServers

  • Migrating Artifactory deployments to S3

  • Completing Terraform upgrades, rolling out admin and access roles, and updating AMQ IaC and deployment

Outcomes

With the enhancements to the Data Lakehouse, teams across the enterprise could now access, trust, and leverage the information and improve decision-making and strategies.

  • Data lake architecture standardized data, making it readable and user-friendly

  • Enforced standardization instilled trust in the data

  • Teams can load, access, and leverage data quickly through centralized authentication

  • Enhanced compliance with automated patterns for developer teams to leverage

Project Attributes

  • Reduced Risk
  • Increased Capabilities
  • Increased Security
  • Reduced COO
  • Reusable Patterns
  • Increased Uptime
  • Increased Scalability
  • Increased Automation
  • Documentation
  • Paired Programming

Technologies Used

  • AWS Simple Storage Service (S3)
  • Terraform
  • Trino
  • Apache Airflow
  • OpenFGA
  • Javascript
  • Apache Superset
  • Python
  • Bash
  • AWS Kubernetes Service (EKS)
  • Helm