Senior Data Engineer

14 hours ago

Delhi, Delhi, India FOSS United Full time
  • All Jobs
  • Senior Data Engineer at CivicDataLab Pvt Ltd

Senior Data Engineer

Apply

Posted on July 22, 2024 | Closed on November 19, 2024

CivicDataLab Pvt Ltd

New Delhi

Full TIme

Job Description
At CivicDataLab, we are building robust and automated tools focusing on data analytics, encompassing data curation, data cleansing, data standardization, and sophisticated data wrangling. These products would support our projects across different sectors – public finance, digital public goods, climate change etc. We are also standardizing open datasets to adhere to the Data Catalog Vocabulary (DCAT) metadata standards, making them searchable and more user-friendly. To assist us with this endeavor, we are seeking an experienced data engineer (with 2+ years of experience as a Data Engineer). This role requires the candidate to be based in Delhi.

Requirements: What You'll Be Doing - Design and develop scalable data orchestration pipelines using Prefect and Apache Airflow - Create and oversee data APIs responsible for collecting, managing, and analysing data from diverse public data sources. - Standardize metadata of open datasets by ensuring compliance with the DCAT metadata standard. - Collaborate with our partners to perform in-depth Exploratory Data Analysis (EDA) of various datasets. - Engage in the development of database models in accordance with the specific project requirements. - Maintain and monitor our existing open data platforms like Open Budgets India, Justice Hub, Open Contracting India. - Engage regularly with our diverse stakeholders and open-source communities to discuss and create reusable resources around use-cases of public data, data engineering best practices, and guidebooks. - Thoroughly document code, processes, and all activities performed by the data team, ensuring clarity and comprehensiveness. This includes documenting algorithms, methodologies, data transformations, and the overall workflow.

Skills You Should Bring - 2+ years of thorough experience working with Python and SQL. - Understanding of message brokers such as RabbitMQ. - Knowledge of open-source data scraping frameworks and tools such as Selenium and Scrapy. - Experience with building an end to end ETL pipeline. - Familiarity with building database systems. - Knowledge of API or Stream-based data extraction processes. - Comprehensive knowledge of a Git-based workflow - Comprehensive knowledge of metadata standards such as DCAT.

Good to have - An understanding of data privacy principles and practices, as well as experience in implementing data privacy algorithms, would be valuable in maintaining the confidentiality and integrity of sensitive datasets. - Prior experience collaborating with government or social sector research-based organizations. - Prior experience in analysing and presenting data using tools such as Apache Superset, Metabase etc. - Proficiency in working with spreadsheets - Knowledge of reading and writing code in R and other statistical programming environments. - Knowledge of working with geospatial data processing tools such as QGIS. - Prior experience in actively contributing to FOSS (Free and Open-Source Software) projects. - Familiarity in working with Agile methodologies and Scrum processes.

  • Senior Data Engineer with MLOps

    4 days ago

    Delhi, Delhi, India Strive4X Infotech Private Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    As a Senior Data Engineer at our company, you will be responsible for designing and implementing scalable data pipelines, productionizing ML workflows, and ensuring system observability. This is a hands-on role that requires expertise in Databricks, Spark, AWS, and MLOps. Your coding skills and execution focus will be crucial in contributing from day one....

  • Senior Data Engineer

    6 days ago

    Delhi, Delhi, India AagatiServe Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Experience:5–8 YearsLocation:Delhi(Onsite)Employment Type:Full-timeAbout the Role:We are looking for a Senior Data Engineer to join our data engineering team and help design, build, and maintain scalable data pipelines and infrastructure on AWS. The ideal candidate will have strong hands-on experience in PySpark, SQL, and AWS services, with a passion for...

  • Data Engineer

    2 weeks ago

    Delhi, Delhi, India Weekday Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    This role is for one of our clientsCompany Name: Industry: Technology, Information and MediaSeniority level: Mid-Senior levelMin Experience: 5 yearsLocation: NCRJobType: full-timeWe are seeking an experienced Data Engineer with deep expertise in the AWS data ecosystem to design, build, and optimize scalable data pipelines and platforms. The ideal candidate...

  • Azure Data Engineer

    2 weeks ago

    Delhi, Delhi, India Kryo Tech Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Role DescriptionThis is a contract remote role for anAzure Data Engineer/SeniorData Engineer job opening at aRetail companyoperates in Middle East. The Azure Data Engineer will be responsible for designing, implementing, and optimizing data pipelines using Azure Data Platform Services like Data Factory, Databricks, SQL Database, Synapse, Power BI. This...

  • Senior Data Engineer

    2 weeks ago

    Delhi, Delhi, India WIN Home Inspection Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    ABOUT THE PYTHON DATA ENGINEER ROLE:We are looking for a skilled Python Data Engineer to join our team and work on building high-performance applications and scalable data solutions. In this role, you will be responsible for designing, developing, and maintaining robust Python-based applications, optimizing data pipelines, and integrating various APIs and...

  • Senior Data Engineer-

    2 weeks ago

    Delhi, Delhi, India Findem Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    What is Findem:Findem is the only talent data platform that combines 3D data with AI. It automates and consolidates top-of-funnel activities across your entire talent ecosystem, bringing together sourcing, CRM, and analytics into one place. Only 3D data connects people and company data over time - making an individual's entire career instantly accessible in...

  • Data Engineer

    2 weeks ago

    Delhi, Delhi, India crescendo global Full time ₹ 8,00,000 - ₹ 12,00,000 per year

    Job Description : Data Engineer - Python - 5 Years - Delhi NCR (Hybrid) Discipline: AnalyticsJob Type: PermanentContact Name: Rohit KumarContact Email: Job Ref: 84244Published: about 12 hours ago Overview Are you a passionate Data Engineer with expertise in Python, eager to work in a dynamic and innovative environment? Heres your chance to be part of a...

  • senior data specialist

    2 weeks ago

    Delhi, Delhi, India RMgX Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are seeking a meticulous and highly responsible Senior Data Specialist who can work independently to design, manage, and optimize data workflows and pipelines. The ideal candidate should bring strong hands-on expertise in databases (PostgreSQL, MySQL), ETL processes, and practical experience with modern ETL tools. Familiarity with Apache NiFi, Talend, or...

  • Data Engineer

    6 days ago

    Delhi, Delhi, India AgileEngine Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards. WHY JOIN US If you're looking for a place to grow, make an...

  • Senior Engineer

    6 hours ago

    Delhi, Delhi, India Arting Digital Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Company DescriptionArting Digital is a Digital Transformation organization specializing in marketing, analytics, and innovation-driven solutions. We help enterprises make data-driven decisions to enhance the performance of their digital platforms, including websites, social media, mobile applications, and more. By combining progressive analytics, technology,...