Data Engineer

3 weeks ago

delhi, India Dhira Full time

Role: Data Engineer for AI/ML
Experience Required: 6+ years

We’re looking for a true data craftsman who can dexterously navigate the worlds of raw data engineering and refined machine learning pipelines.
You’re not the average “ETL engineer” who relies on pre-packaged tools like Informatica or Talend. You are a solid programmer, fluent in Python, and an artist with Py Spark, weaving together data flows that don’t just function—they perform.

Responsibilities:
• Develop and maintain APIs using Flask and FastAPI frameworks to support AI/ML services, including data access, transformation, and integration with Large Language Models (LLMs).
• Design and implement ETL processes to manage data pipelines and build Data Warehouses (DW) for efficient data storage and retrieval.
• Work on integrating LLMs with API services, ensuring secure and efficient data flow and model interaction.
• Develop APIs abstracting and concealing LLM functionality, providing a seamless interface for applications to interact with AI/ML models.
• Collaborate with database systems to handle data manipulation, storage, and retrieval, supporting API-driven machine learning workflows.
• Optimize Python code for performance and security, ensuring robust and scalable API deployment.
• Participate in cross-functional team discussions to align technical solutions with business objectives.
• Stay abreast of advancements in AI, machine learning, and software development practices to suggest and implement improvements.
• Research and develop new algorithms to improve AI system performance.
• Collaborate with cross-functional teams to integrate AI models and technologies into scalable products.
• Engineer solutions that are functional and optimized for performance, scalability, and maintainability.
• Work on batch processing and real-time data streams using tools like Kafka or Flink.
• Utilize cloud platforms (AWS, GCP, Azure) to manage data lakes and warehouses, optimize data engineering workflows, and support the full lifecycle of machine learning pipelines.

Key Qualifications:
• Strong proficiency in Python with 6 years of experience in developing APIs.
• Expertise in Flask and FastAPI for API development.
• Solid understanding of ETL processes, Data Warehousing, and working with relational databases such as PostgreSQL or MySQL.
• Experience with integrating and managing Large Language Models (LLMs) and concealing their APIs behind custom-built services.
• Knowledge of data transformation and access techniques to effectively feed AI/ML models.
• Familiarity with machine learning development is advantageous but not essential.
• Strong problem-solving skills with a focus on optimizing API performance and ensuring security.
• Ability to work both independently and collaboratively within a team.
• Effective communication skills to explain technical concepts clearly and concisely.
• Experience with distributed data processing using PySpark.
• Familiarity with cloud data services and orchestrators such as Airflow to automate data pipelines and workflows.
• Hands-on experience with big data tools like Spark or Flink to manage large-scale data pipelines.
• Cloud computing: Familiarity with cloud platforms such as AWS, Azure, or GCP.
• Big data technologies: Experience with big data tools and technologies like Spark or similar.
• Natural language processing (NLP): Knowledge of NLP techniques and applications.
• Computer vision: Understanding of computer vision algorithms and applications.
Desired Traits:
• Strong problem-solving skills focusing on optimizing API performance and ensuring security.
• Ability to work both independently and collaboratively within a team.
• Effective communication skills to explain technical concepts clearly and concisely.
• A deep curiosity to explore new data engineering challenges and continuously evolve your skills.
• A commitment to building clean, modular, and maintainable code that delivers high-quality solutions.

Educational Background:
Bachelor‘s degree in computer science, Information Technology, or a related field.

  • Data Warehouse Engineer | Data Warehouse Engineer | delhi

    2 weeks ago

    delhi, India Data Warehouse Engineer Full time

    Experience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.

  • Data Warehouse Engineer

    2 weeks ago

    Delhi, India Data Warehouse Engineer Full time

    Experience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.

  • Senior Data Integration Engineer | Senior Data Integration Engineer | delhi

    2 weeks ago

    delhi, India Senior Data Integration Engineer Full time

    Must Have Skills/Skill Requirement:Design and architect integration solutions to connect various enterprise applications, systems, and databases.Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.Utilize Azure Integration Services such as Azure Logic Apps,...

  • Senior Data Integration Engineer

    2 weeks ago

    Delhi, India Senior Data Integration Engineer Full time

    Must Have Skills/Skill Requirement:Design and architect integration solutions to connect various enterprise applications, systems, and databases.Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.Utilize Azure Integration Services such as Azure Logic Apps,...

  • Senior Data Integration Engineer

    2 weeks ago

    Delhi, India Senior Data Integration Engineer Full time

    Must Have Skills/Skill Requirement:- Design and architect integration solutions to connect various enterprise applications, systems, and databases.- Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.- Utilize Azure Integration Services such as Azure Logic...

  • Senior data integration engineer

    2 weeks ago

    Delhi, India Senior Data Integration Engineer Full time

    Must Have Skills/Skill Requirement:Design and architect integration solutions to connect various enterprise applications, systems, and databases.Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.Utilize Azure Integration Services such as Azure Logic Apps,...

  • Pyspark data engineer

    11 hours ago

    Delhi, India ITI Data Full time

    Location : IndiaType : Full-timeExperience : 10 – 13 yearsFunctions : Consulting, Finance, Information Technology, Big Data EngineeringIndustries : Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a Py Spark...

  • PySpark Data Engineer

    15 hours ago

    Delhi, India ITI Data Full time

    Location : IndiaType : Full-timeExperience : 10 – 13 yearsFunctions : Consulting, Finance, Information Technology, Big Data EngineeringIndustries : Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a PySpark...

  • Lead data engineer

    4 weeks ago

    Delhi, India Wavicle Data Solutions Full time

    Job Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, Py Spark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...

  • Data Platform Engineer

    4 days ago

    Delhi, India OSD Data Services Full time

    Data Platform EngineerLocation: RemoteType: InternshipAbout UsAt OSD Data, we’re redefining how businesses leverage their data. Our mission is to empower organizations with a seamless, unified, and scalable data infrastructure that enables smarter decision-making and faster innovation. With a focus on data lakehouse technologies, we’ve built a...

  • Data platform engineer

    4 days ago

    Delhi, India OSD Data Services Full time

    Data Platform Engineer Location : RemoteType : InternshipAbout Us At OSD Data, we’re redefining how businesses leverage their data. Our mission is to empower organizations with a seamless, unified, and scalable data infrastructure that enables smarter decision-making and faster innovation. With a focus on data lakehouse technologies, we’ve...

  • Data Platform Engineer

    4 days ago

    Delhi, India OSD Data Services Full time

    Data Platform EngineerLocation : RemoteType : InternshipAbout UsAt OSD Data, we’re redefining how businesses leverage their data. Our mission is to empower organizations with a seamless, unified, and scalable data infrastructure that enables smarter decision-making and faster innovation. With a focus on data lakehouse technologies, we’ve built a...

  • AWS Data Engineer

    4 weeks ago

    delhi, India ITI Data Full time

    Job Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....

  • AWS Data Engineer

    4 months ago

    Delhi, India ITI Data Full time

    Job DescriptionWe are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There...

  • AWS Data Engineer

    1 month ago

    delhi, India ITI Data Full time

    Job DescriptionWe are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There...

  • Aws data engineer

    4 weeks ago

    Delhi, India ITI Data Full time

    Job DescriptionWe are looking for an AWS Data with primary skills on Py Spark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....

  • ITI Data | PySpark Data Engineer

    20 hours ago

    Delhi, India ITI Data Full time

    Location:IndiaType: Full-timeExperience: 10 – 13 yearsFunctions: Consulting, Finance, Information Technology, Big Data EngineeringIndustries: Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a PySpark solutions developer...

  • PySpark Data Engineer

    17 hours ago

    Delhi, India ITI Data Full time

    Location:IndiaType: Full-timeExperience: 10 – 13 yearsFunctions: Consulting, Finance, Information Technology, Big Data EngineeringIndustries: Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a PySpark solutions developer...

  • OSD Data Services | Data Platform Engineer | india

    4 days ago

    Delhi, India OSD Data Services Full time

    Data Platform Engineer Location : RemoteType : InternshipAbout Us At OSD Data, we’re redefining how businesses leverage their data. Our mission is to empower organizations with a seamless, unified, and scalable data infrastructure that enables smarter decision-making and faster innovation. With a focus on data lakehouse technologies, we’ve built a...

  • Data Unveil | Quality Assurance Engineer

    2 days ago

    delhi, India Data Unveil Full time

    About us: At Data Unveil, we believe in delivering the best for our clients (Pharma Companies). We use the latest technology and tools to aggregate and analyze specialty healthcare data received from various data partners. We provide clear and hassle-free business insights to enhance the client’s vision and drive business success.Position Title: QA...