PySpark & ETL Data Engineer

We are CirrusLabs. Our vision is to become the world's most sought-after niche digital transformation company that helps customers realize value through innovation. Our mission is to co-create success with our customers, partners and community. Our goal is to enable employees to dream, grow and make things happen. We are committed to excellence. We are a dependable partner organization that delivers on commitments. We strive to maintain integrity with our employees and customers. Every action we take is driven by value. The core of who we are is through our well-knit teams and employees. You are the core of a values driven organization.You have an entrepreneurial spirit. You enjoy working as a part of well-knit teams. You value the team over the individual. You welcome diversity at work and within the greater community. You aren't afraid to take risks. You appreciate a growth path with your leadership team that journeys how you can grow inside and outside of the organization. You thrive upon continuing education programs that your company sponsors to strengthen your skills and for you to become a thought leader ahead of the industry curve.You are excited about creating change because your skills can help the greater good of every customer, industry and community. We are hiring a talented Pyspark to join our team. If you're excited to be part of a winning team, CirrusLabs (http://www.cirruslabs.io) is a great place to grow your career.Experience - 4-8 yearsLocation - Hyderabad/ BengaluruAbout the RoleCirrusLabs is seeking a skilled and experienced PySpark Data Engineer (ETL Lead) to join our growing data engineering team. As an ETL Lead, you will play a pivotal role in designing, developing, and maintaining robust data integration pipelines using PySpark and related technologies. You’ll work closely with data architects, analysts, and stakeholders to transform raw data into high-quality, actionable insights, enabling data-driven decision-making across the organization.This is an exciting opportunity for someone who is not only technically strong in PySpark and Python but also capable of leading data integration efforts for complex projects.Key Responsibilities- Lead Data Integration Projects: - Manage the data integration and ETL activities for enterprise-level data projects. - Gather requirements from stakeholders and translate them into technical solutions. - Develop PySpark Pipelines: - Design and develop scalable and efficient PySpark scripts, both generic frameworks and custom solutions tailored to specific project requirements. - Implement end-to-end ETL processes to ingest, clean, transform, and load data. - Schedule and Automate ETL Processes: - Create scheduling processes to manage and run PySpark jobs reliably and efficiently. - Integrate ETL workflows into automation tools and CI/CD pipelines. - Optimize Data Processing: - Optimize PySpark jobs for performance and resource efficiency. - Monitor, troubleshoot, and resolve issues related to data processing and pipeline execution. - Data Transformation and Curation: - Transform raw data into consumable, curated data models suitable for reporting and analytics. - Ensure data quality, consistency, and reliability throughout all stages of the ETL process. - Collaboration and Best Practices: - Collaborate with data architects, analysts, and business stakeholders to define requirements and deliver solutions. - Contribute to the evolution of data engineering practices, frameworks, and standards. - Provide guidance and mentorship to junior engineers on PySpark and ETL best practices. - Documentation: - Develop and maintain technical documentation related to ETL processes, data flows, and solutions.Required Skills and Qualifications- Experience: - 5–8 years of professional experience in data engineering, ETL development, or related fields. - Proven experience leading data integration projects from design to deployment. - Technical Skills: - Strong hands-on experience with PySpark for building large-scale data pipelines. - Proficiency in Python, including writing efficient, reusable, and modular code. - Solid knowledge of SQL for data extraction, transformation, and analysis. - Strong understanding of Spark architecture, including execution plans, partitions, memory management, and optimization techniques. - Data Engineering Expertise: - Experience working on data integration projects, such as data warehousing, data lakes, or analytics solutions. - Familiarity with processing structured and semi-structured data formats (e.g., Parquet, Avro, JSON, CSV). - Ability to transform and harmonize data from raw to curated layers.Additional Skills:- Familiarity with data pipeline orchestration tools (e.g., Airflow, Azkaban) is a plus. - Experience with cloud platforms (e.g., AWS, Azure, GCP) is desirable. - Strong analytical and problem-solving skills. - Excellent communication and collaboration skills.