Big Data Engineering Lead
4 days ago
Job Title: Senior Data Engineer
About the RoleWe are seeking a highly skilled and experienced Senior Data Engineer to lead our data engineering team. The successful candidate will be responsible for designing, developing, and maintaining large-scale data processing systems.
The ideal candidate will have a strong technical background in big data engineering, excellent collaboration skills, and experience working with multiple data sources. They will be able to debug, fine-tune, and optimize large-scale data processing jobs and provide technical leadership to the team.
Key Responsibilities- Built pipelines to bring in a wide variety of data from multiple sources within the organization and from social media and public data sources.
- Collaborated with cross-functional teams to source data and make it available for downstream consumption.
- Worked with the team to provide an effective solution design to meet business needs.
- Ensured regular communication with key stakeholders, understanding any key concerns in how the initiative is being delivered or any risks/issues that have either not yet been identified or are not being progressed.
- Escalated dependencies and challenges (risks) and managed them. Escalated critical issues to the Sponsor and/or Head of Data Engineering.
- Managed timelines (milestones, decisions, and delivery) and ensured the value of the initiative was achieved without compromising quality and within budget.
- Developed an appropriate and coordinated communications plan for initiative execution and delivery, both internal and external.
- Work as a team player.
- Excellent problem analysis skills.
- Good experience with the Azure Databricks platform.
- Experience with at least one Cloud Infra provider (Azure/AWS).
- Experience in building data pipelines using batch processing with Apache Spark (Spark SQL, Dataframe API) or Hive query language (HQL).
- Experience in building streaming data pipelines using Apache Spark Structured Streaming or Apache Flink on Kafka & Delta Lake.
- Knowledge of NoSQL databases. Good experience in Cosmos DB, Restful APIs, and GraphQL.
- Knowledge of Big Data ETL processing tools, Data modelling, and Data mapping.
- Experience with Hive and Hadoop file formats (Avro / Parquet / ORC).
- Basic knowledge of scripting (shell / bash).
- Experience working with multiple data sources including relational databases (SQL Server / Oracle / DB2 / Netezza), NoSQL/document databases, flat files.
- Basic understanding of CI/CD tools such as Jenkins, JIRA, Bitbucket, Artifactory, Bamboo, and Azure Dev-ops.
- Basic understanding of DevOps practices using Git version control.
- Ability to debug, fine-tune, and optimize large-scale data processing jobs.
This role offers a unique opportunity to work with cutting-edge technology and be part of a dynamic team. The successful candidate will have the chance to develop their skills and expertise in big data engineering and contribute to the success of our organization.
About UsWe are a leading company in the field of big data engineering, and we are committed to delivering high-quality solutions to our clients. We offer a collaborative and supportive work environment, and we are looking for talented individuals who share our passion for innovation and excellence.
-
Big data engineer
1 day ago
Anantapur, Andhra Pradesh, India Idyllic Services Full timeJob Title: Big Data Engineer – ScalaLocation: Bangalore, Chennai, Gurgaon, Pune, Mumbai.Experience: 7–10 Years (Minimum 3+ years in Scala)Notice Period: Immediate to 30 DaysMode of Work: Hybrid ???? Click the link below to learn more about the role and take the AI Interview to begin your application journey: Pe NRole OverviewWe are looking for a highly...
-
Big Data Developer
2 weeks ago
Anantapur, Andhra Pradesh, India DigiHelic Solutions Pvt. Ltd. Full timeJob description Key Responsibilities: Big Data Architecture: Design, develop, and maintain scalable and distributed data architectures capable of processing large volumes of data. Data Storage Solutions: Implement and optimize data storage solutions using technologies such as Hadoop , Spark , and PySpark . PySpark Development: Develop and implement...
-
Big Data Specialist in BFSI
3 days ago
Anantapur, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000Job DescriptionWe seek a seasoned Data Engineer to lead our project in the BFSI domain.Key ResponsibilitiesCollaborate with data science teams to implement complex algorithms and analytical solutions.Develop scalable data pipelines using agile development practices.Model front-end and back-end data flows to empower business decision-making through advanced...
-
Big Data Developer Position
6 days ago
Anantapur, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 48,00,000Job SummaryWe are seeking a highly skilled Big Data Developer to join our team. This is an exciting opportunity for a talented individual to work on challenging projects and contribute to the growth of our organization.About the RoleThe ideal candidate will have experience in developing big data solutions using Azure Databricks, PySpark, and other relevant...
-
Cloud Data Engineering Team Lead
3 days ago
Anantapur, Andhra Pradesh, India beBeeDataLeader Full time ₹ 15,00,000 - ₹ 25,00,000**Data Engineering Leadership Opportunity**We are seeking a highly skilled professional to lead our data engineering team. The successful candidate will have around 6+ years of experience in data engineering or cloud computing data development.Data Pipeline Development: Design, develop, and optimize ETL/ELT data pipelines using Apache Spark (PySpark or...
-
Senior Big Data Expert
4 days ago
Anantapur, Andhra Pradesh, India beBeeDataEngineer Full time US$ 90,000 - US$ 1,80,000Job Opportunity:We are seeking a highly skilled Data Engineer with strong experience in PySpark and hands-on expertise in building scalable data pipelines. Main Responsibilities:Design, develop, and maintain large-scale data processing systems using PySparkCollaborate with cross-functional teams to identify business needs and develop data-driven...
-
Data Engineering Specialist
6 days ago
Anantapur, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000Job Title: Data Engineering SpecialistWe are seeking a skilled Data Engineer to help implement an application for large retailers. The ideal candidate will have hands-on expertise in Python and PySpark, as well as experience with Telemetry, Big Data, and backend engineering.Key Responsibilities:Design, develop, and deploy data processing pipelines using...
-
Data Architect Lead
4 days ago
Anantapur, Andhra Pradesh, India beBeeTechnical Full time US$ 1,80,000 - US$ 2,50,000Senior Engineering ManagerSigmoid empowers businesses to make informed decisions by combining cutting-edge data engineering with AI consulting. Our global team specializes in cloud data modernization, predictive analytics, generative AI, and DataOps, backed by 10+ delivery centers and innovation hubs.Job Responsibilities:Align Sigmoid's goals with key client...
-
Strategic Data Engineer
2 days ago
Anantapur, Andhra Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000Job Title: Strategic Data EngineerWe are seeking an experienced data engineer to lead our team's data engineering efforts. As a senior data engineer, you will play a critical role in designing, developing, and maintaining scalable data pipelines and data warehouse solutions on Google Cloud Platform (GCP).About the Position:This is an exceptional opportunity...
-
High-Performance Data Engineer Expert Wanted
6 days ago
Anantapur, Andhra Pradesh, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000Expert Big Data Engineer Wanted.Our team is looking for a highly skilled data engineer to join our professional services. As a big data engineer at our company, you will be responsible for designing and building high-performance data pipelines that process large volumes of data.The ideal candidate will have experience with Python, big data technologies, and...
