AmpsTek - Data Engineer - Python/Azure Data Factory

Title : Data Engineer.

Location : Hyderabad, India (Onsite).

Job Description :

We are seeking an experienced Data Engineer with 5-8 years of professional experience to design, build, and optimize robust and scalable data pipelines for our SmartFM platform.

The ideal candidate will be instrumental in ingesting, transforming, and managing vast amounts of operational data from various building devices, ensuring high data quality and availability for analytics and AI/ML applications.

This role is critical in enabling our platform to generate actionable insights, alerts, and recommendations for optimizing facility operations.

ROLES AND RESPONSIBILITIES :

- Design, develop, and maintain scalable and efficient data ingestion pipelines from diverse sources (e.g., IoT devices, sensors, existing systems) using technologies like IBM StreamSets, Azure Data Factory, Apache Spark, Talend Apache Flink and Kafka.

- Implement robust data transformation and processing logic to clean, enrich, and structure raw data into formats suitable for analysis and machine learning models.

- Manage and optimize data storage solutions, primarily within MongoDB, ensuring efficient schema design, data indexing, and query performance for large datasets.

- Collaborate closely with Data Scientists to understand their data needs, provide high-quality, reliable datasets, and assist in deploying data-driven solutions.

- Ensure data quality, consistency, and integrity across all data pipelines and storage systems, implementing monitoring and alerting mechanisms for data anomalies.

- Work with cross-functional teams (Software Engineers, Data Scientists, Product Managers) to integrate data solutions with the React frontend and Node.js backend applications.

- Contribute to the continuous improvement of data architecture, tooling, and best practices, advocating for scalable and maintainable data solutions.

- Troubleshoot and resolve complex data-related issues, optimizing pipeline performance and ensuring data availability.

- Stay updated with emerging data engineering technologies and trends, evaluating and recommending new tools and approaches to enhance our data capabilities.

REQUIRED TECHNICAL SKILLS AND EXPERIENCE :

- 5-8 years of professional experience in Data Engineering or a related field.

- Proven hands-on experience with data pipeline tools such as IBM StreamSets, Azure Data Factory, Apache Spark, Talend Apache Flink and Apache Kafka.

- Strong expertise in database management, particularly with MongoDB, including schema design, data ingestion pipelines, and data aggregation.

- Proficiency in at least one programming language commonly used in data engineering, such as Python or Java/Scala.

- Experience with big data technologies and distributed processing frameworks (e.g., Apache Spark, Hadoop) is highly desirable.

- Familiarity with cloud platforms (Azure, AWS, or GCP) and their data services.

- Solid understanding of data warehousing concepts, ETL/ELT processes, and data modeling.

- Experience with DevOps practices for data pipelines (CI/CD, monitoring, logging).

- Knowledge of Node.js and React environments to facilitate seamless integration with existing applications.

ADDITIONAL QUALIFICATIONS :

- Demonstrated expertise in written and verbal communication, adept at simplifying complex technical concepts for both technical and non-technical audiences.

- Strong problem-solving and analytical skills with a meticulous approach to data quality.

- Experienced in collaborating and communicating seamlessly with diverse technology roles, including development, support, and product management.

- Highly motivated to acquire new skills, explore emerging technologies, and stay updated on the latest trends in data engineering and business needs.

- Experience in the facility management domain or IoT data is a plus.

EDUCATION REQUIREMENTS / EXPERIENCE :

- Bachelors (BE / BTech) / Masters degree (MS/MTech) in Computer Science, Information Systems, Mathematics, Statistics, or a related quantitative field.