Kloud9 - Lead Data Engineer - Google Cloud Platform

Company : Kloud9

Location : Bangalore

About Us :

Founded in 2016 and headquartered in New York, Kloud9 is one of the leading technology services and solutions providers specializing in Data Science, Artificial Intelligence (AI) and Machine Learning (ML). At Kloud9, we work closely with our customers across Retail, Consumer Goods, Healthcare & Manufacturing to achieve a satisfying customer experience using modern technologies in AI, ML, Cloud and Automation, that transform Data into action-driven insights.

As a leading provider of cutting-edge cloud solutions, Kloud9 empowers businesses to harness the full potential of cloud technology, enabling them to scale, innovate, and thrive in the digital era. With a strong commitment to excellence and a customer-centric approach, Kloud9 is redefining the way organizations leverage the power of the cloud to drive growth and efficiency.

Through our customer-centric approach and relentless commitment to excellence, we have helped numerous organizations achieve remarkable results in their cloud journey. Our success stories highlight the transformative impact of our cloud solutions, including cost savings, increased productivity, improved scalability, and enhanced business agility. With our strong technology expertise, we understand & visualize a new world with an AI-First Approach. Kloud9 is constantly on the lookout for top-tier talent to join us in this exciting journey

Role Overview :

We're seeking an experienced Lead GCP Data Engineer who can build cloud analytics platform to meet ever expanding business requirements with speed and quality using lean Agile practices. You will work on analysing and manipulating large datasets supporting the enterprise by activating data assets to support Enabling Platforms and Analytics in the Google Cloud Platform (GCP).

You will be responsible for designing the transformation and modernization on GCP, as well as landing data from source applications to GCP. Experience with large scale solution and operationalization of data warehouses, data lakes and analytics platforms on Google Cloud Platform or other cloud environment is a must.

We are looking for candidates who have a broad set of technology skills across these areas and who can demonstrate an ability to design right solutions with appropriate combination of GCP and 3rd party technologies for deploying on Google Cloud Platform.

In a nutshell :

- Deep dive into complex performance issues to improve efficiency and reduce customer incidents.

- Google Certified Professional Cloud Architect with experience automating and orchestrating workloads on GCP or other Public Clouds.

- Automation experience with at least one configuration management system such as Chef, Puppet, Ansible, Salt, or other such tools.

- Experience working at least one of the following languages: Python, Go Proficient with git and git workflows.

- Proficient in demonstrating CI and CD tools (Jenkins, TeamCity, Spinnaker) to automate testing and deployment.

Key Responsibilities :

Data Architecture Design :

- As a Lead GCP Data Engineer, your primary responsibility is to design the data architecture that supports efficient data processing and analysis on the Google Cloud Platform.

- This involves understanding the data requirements of the organization and working closely with data scientists, business analysts, and other stakeholders to design effective data models and structures.

- You will need to choose the appropriate GCP services and technologies to build a scalable and robust data architecture that aligns with the organization's goals.

Data Pipeline Development :

- Developing data pipelines is a key responsibility of a Lead GCP Data Engineer. These pipelines enable the smooth flow of data from various sources to the desired destinations, ensuring data quality, reliability, and governance.

- You will work with GCP services like Google Cloud Storage, Bigquery, Dataflow, and Pub/Sub to build data ingestion, transformation, and processing pipelines.

- This involves coding, scripting, and configuring these services to ensure data is processed and transformed efficiently.

Data Transformation and Integration :

- Lead GCP Data Engineers are proficient in data transformation techniques and tools.

- You will leverage technologies like Apache Beam, Apache Spark, and Cloud Dataproc to clean, transform, and integrate data from diverse sources.

- This involves performing data cleansing, aggregation, enrichment, and normalization to ensure data consistency, accuracy, and usability for downstream applications and analytics.

Performance Optimization :

- Lead GCP Data Engineers are responsible for optimizing the performance of data processing workflows.

- You will monitor data pipelines, identify bottlenecks, and fine-tune the pipelines for optimal performance.

- This may involve optimizing data transformations, improving data partitioning and sharding, and leveraging GCP's autoscaling and load-balancing capabilities.

- Your goal is to ensure efficient resource utilization, reduce processing time, and achieve optimal performance for data processing and analysis tasks.

- Improving Skills Continuously

- To excel as a Lead GCP Data Engineer, continuous learning and staying updated with the latest advancements in data engineering and cloud technologies are crucial.

- You will actively explore new features and services offered by GCP and identify innovative solutions to improve data engineering processes.

- Continuous learning involves attending training sessions, pursuing relevant certifications, participating in industry events and forums, and staying connected with the data engineering community.

- By staying up to date with the latest trends, you can leverage new technologies and techniques to enhance data processing, analysis, and insights.

Conduct Research :

- Lead GCP Data Engineers often need to stay informed about the latest industry trends, emerging technologies, and best practices in data engineering.

- Researching and evaluating new tools, frameworks, and methodologies can help you identify opportunities for innovation and improvement within your organization.

- By conducting research, attending conferences, and staying connected with the data engineering community, you can bring fresh ideas and insights to enhance data engineering processes and drive continuous improvement.

Automate Tasks :

- As a Lead GCP Data Engineer, you will be responsible for automating data engineering tasks to improve efficiency and productivity.

- This involves developing scripts, and workflows, or using tools like Cloud Composer or Cloud Functions to automate repetitive or time-consuming data processes.

- By automating tasks such as data ingestion, transformation, or monitoring, you can reduce manual effort, minimize errors, and streamline data workflows.

Attributes & Competencies :

- Education : BE / BTECH | MTech | MCA

- Minimum 7+ years of work experience in development / Migration project. 3+ years of GCP experience.

- Experience working in GCP based Big Data deployments (Batch/Real-Time) leveraging Big Query, Big Table, Google Cloud Storage, PubSub, Data Fusion, Dataflow, Dataproc, Airflow.

Soft Skills :

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

- Ability to work independently and as part of a global team.

- Passionate for data solutions

- Self-motivated and able to work in a fast-paced environment.

- Detail-oriented and committed to delivering high-quality work.

Technical Skills :

- Strong coding skills in languages such as Python, PySpark, or Scala

- Hands-on experience with GCP data services such as Bigquery, Dataflow, Pub/Sub, DataProc, and Cloud Storage.

- Experience using Databricks (Data Engineering and Delta Lake components)

- Experience with source control tools such as GitHub and related dev process.

- Experience with workflow scheduling tools such as Airflow.

- Strong understanding of data structures and algorithms

- Experience building data lake solutions leveraging Google Data Products (e.g., DataProc, AI Building Blocks, Looker, Cloud Data Fusion, Dataprep, etc.), Hive, Spark

- Experience with relational SQL/No SQL