Senior Data Engineer

1 day ago

Delhi, Delhi, India FOSS United Full time
  • All Jobs
  • Senior Data Engineer at CivicDataLab Pvt Ltd

Senior Data Engineer

Apply

Posted on July 22, 2024 | Closed on November 19, 2024

CivicDataLab Pvt Ltd

New Delhi

Full TIme

Job Description
At CivicDataLab, we are building robust and automated tools focusing on data analytics, encompassing data curation, data cleansing, data standardization, and sophisticated data wrangling. These products would support our projects across different sectors – public finance, digital public goods, climate change etc. We are also standardizing open datasets to adhere to the Data Catalog Vocabulary (DCAT) metadata standards, making them searchable and more user-friendly. To assist us with this endeavor, we are seeking an experienced data engineer (with 2+ years of experience as a Data Engineer). This role requires the candidate to be based in Delhi.

Requirements: What You'll Be Doing - Design and develop scalable data orchestration pipelines using Prefect and Apache Airflow - Create and oversee data APIs responsible for collecting, managing, and analysing data from diverse public data sources. - Standardize metadata of open datasets by ensuring compliance with the DCAT metadata standard. - Collaborate with our partners to perform in-depth Exploratory Data Analysis (EDA) of various datasets. - Engage in the development of database models in accordance with the specific project requirements. - Maintain and monitor our existing open data platforms like Open Budgets India, Justice Hub, Open Contracting India. - Engage regularly with our diverse stakeholders and open-source communities to discuss and create reusable resources around use-cases of public data, data engineering best practices, and guidebooks. - Thoroughly document code, processes, and all activities performed by the data team, ensuring clarity and comprehensiveness. This includes documenting algorithms, methodologies, data transformations, and the overall workflow.

Skills You Should Bring - 2+ years of thorough experience working with Python and SQL. - Understanding of message brokers such as RabbitMQ. - Knowledge of open-source data scraping frameworks and tools such as Selenium and Scrapy. - Experience with building an end to end ETL pipeline. - Familiarity with building database systems. - Knowledge of API or Stream-based data extraction processes. - Comprehensive knowledge of a Git-based workflow - Comprehensive knowledge of metadata standards such as DCAT.

Good to have - An understanding of data privacy principles and practices, as well as experience in implementing data privacy algorithms, would be valuable in maintaining the confidentiality and integrity of sensitive datasets. - Prior experience collaborating with government or social sector research-based organizations. - Prior experience in analysing and presenting data using tools such as Apache Superset, Metabase etc. - Proficiency in working with spreadsheets - Knowledge of reading and writing code in R and other statistical programming environments. - Knowledge of working with geospatial data processing tools such as QGIS. - Prior experience in actively contributing to FOSS (Free and Open-Source Software) projects. - Familiarity in working with Agile methodologies and Scrum processes.

  • Senior Data Engineer

    1 day ago

    Delhi, Delhi, India FloData Analytics Full time ₹ 6,00,000 - ₹ 7,00,000 per year

    Job Title: Senior Data EngineerJob Type: Full-time (Remote)Shift Timing pm to 2.30-3am ISTExperience Required: 2–3 YearsAt Flodata Analytics, we are seeking a Senior Data Engineer to join our team. The ideal candidate will be a technical leader with a passion for transforming data into strategic business insights. This role requires a high degree of...

  • Senior Data Engineer with MLOps

    5 days ago

    Delhi, Delhi, India Strive4X Infotech Private Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    As a Senior Data Engineer at our company, you will be responsible for designing and implementing scalable data pipelines, productionizing ML workflows, and ensuring system observability. This is a hands-on role that requires expertise in Databricks, Spark, AWS, and MLOps. Your coding skills and execution focus will be crucial in contributing from day one....

  • Senior Data Engineer

    7 days ago

    Delhi, Delhi, India AagatiServe Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Experience:5–8 YearsLocation:Delhi(Onsite)Employment Type:Full-timeAbout the Role:We are looking for a Senior Data Engineer to join our data engineering team and help design, build, and maintain scalable data pipelines and infrastructure on AWS. The ideal candidate will have strong hands-on experience in PySpark, SQL, and AWS services, with a passion for...

  • Data Engineer

    2 weeks ago

    Delhi, Delhi, India Weekday Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    This role is for one of our clientsCompany Name: Industry: Technology, Information and MediaSeniority level: Mid-Senior levelMin Experience: 5 yearsLocation: NCRJobType: full-timeWe are seeking an experienced Data Engineer with deep expertise in the AWS data ecosystem to design, build, and optimize scalable data pipelines and platforms. The ideal candidate...

  • Azure Data Engineer

    2 weeks ago

    Delhi, Delhi, India Kryo Tech Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Role DescriptionThis is a contract remote role for anAzure Data Engineer/SeniorData Engineer job opening at aRetail companyoperates in Middle East. The Azure Data Engineer will be responsible for designing, implementing, and optimizing data pipelines using Azure Data Platform Services like Data Factory, Databricks, SQL Database, Synapse, Power BI. This...

  • Senior Data Engineer-

    2 weeks ago

    Delhi, Delhi, India Findem Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    What is Findem:Findem is the only talent data platform that combines 3D data with AI. It automates and consolidates top-of-funnel activities across your entire talent ecosystem, bringing together sourcing, CRM, and analytics into one place. Only 3D data connects people and company data over time - making an individual's entire career instantly accessible in...

  • Senior Software Application Development Engineer

    2 weeks ago

    Delhi, Delhi, India NTT DATA Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Continue to make an impact with a company that is pushing the boundaries of what is possible. At NTT DATA, we are renowned for our technical excellence, leading innovations, and making a difference for our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can continue to grow, belong, and thrive.Your career here...

  • Data Engineer

    6 days ago

    Delhi, Delhi, India AgileEngine Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards. WHY JOIN US If you're looking for a place to grow, make an...

  • senior data specialist

    2 weeks ago

    Delhi, Delhi, India RMgX Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are seeking a meticulous and highly responsible Senior Data Specialist who can work independently to design, manage, and optimize data workflows and pipelines. The ideal candidate should bring strong hands-on expertise in databases (PostgreSQL, MySQL), ETL processes, and practical experience with modern ETL tools. Familiarity with Apache NiFi, Talend, or...

  • Senior Data Warehouse Engineer

    5 days ago

    Delhi, Delhi, India TGS The Global Skills Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    Job Description:The Data Warehouse Engineer will develop and maintain pipelines that transform raw ingested data into clean, analytics-ready schemas consumed by executives, managers, analysts, and clients.This is a hands-on role ideal for someone who loves working at the intersection of engineering and analytics. The candidate will join a talented and...