Data Engineer

Title: Senior Data Engineer / ETL EngineerLocation: Pune / Hyderabad / MohaliJob Summary:As a Senior Data Engineer / ETL Engineer, you will be instrumental in designing, developing, and optimizing data processing systems that support our organization's data initiatives. Your expertise in Bigdata and Google Cloud Platform (GCP) will be essential in building robust ETL pipelines that transform raw data into actionable insights. You will collaborate with cross-functional teams to ensure that data is accurate, accessible, and valuable for decision-making.Responsibilities:Design and Develop ETL Processes:Lead the design and implementation of ETL processes using all kinds of batch/streaming tools to extract, transform, and load data from various sources into GCP.Collaborate with stakeholders to gather requirements and ensure that ETL solutions meet business needs.Data Pipeline Optimization:Optimize data pipelines for performance, scalability, and reliability, ensuring efficient data processing workflows.Monitor and troubleshoot ETL processes, proactively addressing issues and bottlenecks.Data Integration and Management:Integrate data from diverse sources, including databases, APIs, and flat files, ensuring data quality and consistency.Manage and maintain data storage solutions in GCP (e.g., BigQuery, Cloud Storage) to support analytics and reporting.GCP Dataflow Development:Write Apache Beam based Dataflow Job for data extraction, transformation, and analysis, ensuring optimal performance and accuracy.Collaborate with data analysts and data scientists to prepare data for analysis and reporting.Automation and Monitoring:Implement automation for ETL workflows using tools like Apache Airflow or Cloud Composer, enhancing efficiency and reducing manual intervention.Set up monitoring and alerting mechanisms to ensure the health of data pipelines and compliance with SLAs.Data Governance and Security:Apply best practices for data governance, ensuring compliance with industry regulations (e.g., GDPR, HIPAA) and internal policies.Collaborate with security teams to implement data protection measures and address vulnerabilities.Documentation and Knowledge Sharing:Document ETL processes, data models, and architecture to facilitate knowledge sharing and onboarding of new team members.Conduct training sessions and workshops to share expertise and promote best practices within the team.Qualifications:Education:Bachelor’s degree in Computer Science, Information Systems, or a related field.Experience:Minimum of 5 years of industry experience in data engineering or ETL development, with a strong focus on Data Stage and GCP.Proven experience in designing and managing ETL solutions, including data modeling, data warehousing, and SQL development.Technical Skills:Strong knowledge of GCP services (e.g., BigQuery, Dataflow, Cloud Storage, Pub/Sub) and their application in data engineering.Experience of cloud-based solutions, especially in GCP, cloud certified candidate is preferred.Experience and knowledge of Bigdata data processing in batch mode and streaming mode, proficient in Bigdata eco systems, e.g. Hadoop, HBase, Hive, MapReduce, Kafka, Flink, Spark, etc.Familiarity with Java & Python for data manipulation on Cloud/Bigdata platform.Analytical Skills:Strong problem-solving skills with a keen attention to detail.Ability to analyze complex data sets and derive meaningful insights.Company Description: Bits in Glass - IndiaIndustry Leader:Bits in Glass(BIG) has been in business for more than 20 years. In 2021 Bits in Glass joined hands with Crochet Technologies, forming a larger organization under the Bits In Glass brand to better serve customers across the globe.Offices across three locations in India: Pune, Hyderabad & Chandigarh.Specialized Pega partner since 2017, delivering Pega solutions with deep industry expertise and experience.Proudly ranked among the top 30 Pega partners, Bits In Glass has been one of the very few sponsors of the annual PegaWorld event.Elite Appian partner since 2008, delivering Appian solutions with deep industry expertise and experience.Operating in the United States, Canada, United Kingdom, and India.Dedicated global Pega CoE to support our customers and internal dev teams.Employee Benefits:Career Growth: Opportunities for career advancement and professional development.Challenging Projects: Work on innovative, cutting-edge projects that make a global impact. Global Exposure: Collaborate with international teams and clients to broaden your professional network.Flexible Work Arrangements: Support for work-life balance through flexible working conditions.Comprehensive Benefits: Competitive compensation packages and comprehensive benefits including health insurance, and paid time off.Learning Opportunities- Great opportunity to upskill yourself and work on new technologies like AI-enabled Pega solutions, Data engineering, Integration, cloud migration etc.Company Culture:Collaborative Environment: Emphasizes teamwork, innovation, and knowledge sharing.Inclusive Workplace: Values diversity and fosters an inclusive environment where all ideas are respected.Continuous Learning: Encourages professional development through ongoing learning opportunities and certifications.Core Values:Integrity: Commitment to ethical practices and transparency in all business dealings.Excellence: Strive for the highest standards in everything we do.Client-Centric Approach: Focus on delivering the best solutions tailored to client needs.