DataZymes - Databricks Engineer
7 days ago
We are seeking a skilled and motivated Databricks Engineer to join our dynamic data team. The ideal candidate will have 3-5 years of robust experience in data engineering, with a strong focus on the Databricks ecosystem. You will be responsible for designing, developing, and maintaining scalable and reliable data pipelines using PySpark. A key part of this role involves leveraging the Databricks Lakehouse Platform and its advanced AI/ML features to unlock data-driven insights and power our business intelligence and machine learning initiatives.
Key Responsibilities :
- Pipeline Development : Design, build, and maintain efficient and scalable ETL/ELT pipelines on the Databricks platform using PySpark, SQL, and Delta Live Tables (DLT).
- Lakehouse Management : Implement and manage data solutions within the Databricks Lakehouse Platform, ensuring best practices for data storage, governance, and management using Delta Lake and Unity Catalog.
- Code Optimization : Write high-quality, maintainable, and optimized PySpark code for large-scale data processing and transformation tasks.
- AI & ML Integration : Collaborate with data scientists to productionize machine learning models. Utilize Databricks AI features such as the Feature Store, MLflow for model lifecycle management, and AutoML for accelerating model development.
- Data Quality & Governance : Implement robust data quality checks and validation frameworks to ensure data accuracy, completeness, and reliability within the delta tables.
- Performance Tuning : Monitor, troubleshoot, and optimize the performance of Databricks jobs, clusters, and SQL warehouses to ensure efficiency and cost-effectiveness.
- Collaboration : Work closely with data analysts, data scientists, and business stakeholders to understand their data requirements and deliver effective solutions.
- Documentation : Create and maintain comprehensive technical documentation for data pipelines, architectures, and processes.
Required Qualifications & Skills :
- Experience : 3-5 years of hands-on experience in a data engineering role.
- Databricks Expertise : Proven, in-depth experience with the Databricks platform, including Databricks Workflows, Notebooks, Clusters, and Delta Live Tables.
- Programming Skills : Strong proficiency in Python and extensive hands-on experience with PySpark for data manipulation and processing.
- Data Architecture : Solid understanding of modern data architectures, including the Lakehouse paradigm, Data Lakes, and Data Warehousing.
- Delta Lake : Hands-on experience with Delta Lake, including schema evolution, ACID transactions, and time travel features.
- SQL Proficiency : Excellent SQL skills and the ability to write complex queries for data analysis and transformation.
- Databricks AI : Practical experience with Databricks AI/ML capabilities, particularly MLflow and the Feature Store.
- Cloud Experience : Experience working with at least one major cloud provider (AWS, Azure, or GCP).
- Problem-Solving : Strong analytical and problem-solving skills with the ability to debug complex data issues.
- Communication : Excellent verbal and written communication skills.
Preferred Qualifications :
- Databricks Certified Data Engineer Associate/Professional certification.
- Experience with CI/CD tools (e.g., Jenkins, Azure DevOps, GitHub Actions) for data pipelines.
- Familiarity with streaming technologies like Structured Streaming.
- Knowledge of data governance tools and practices within Unity Catalog.
-
Databricks Engineer
3 weeks ago
Bengaluru, Karnataka, India DataZymes Full timeABOUT US: Founded in 2016, DataZymes is a next-generation analytics and data science company driving technology and digital-led innovation for our clients, thus helping them get more value from their data and analytics investments. Our platforms are built on best-of-breed technologies, thus protecting current investments while providing clients more bang...
-
Databricks Engineer
3 weeks ago
Bengaluru, Karnataka, India DataZymes Full timeABOUT US: Founded in 2016, DataZymes is a next-generation analytics and data science company driving technology and digital-led innovation for our clients, thus helping them get more value from their data and analytics investments. Our platforms are built on best-of-breed technologies, thus protecting current investments while providing clients more bang...
-
Databricks Engineer
6 days ago
Bengaluru, Karnataka, India DataZymes Full time US$ 1,20,000 - US$ 2,00,000 per yearABOUT US:Founded in 2016, DataZymes is a next-generation analytics and data science company driving technology and digital-led innovation for our clients, thus helping them get more value from their data and analytics investments. Our platforms are built on best-of-breed technologies, thus protecting current investments while providing clients more bang for...
-
Bengaluru, Karnataka, India DATAZYMES ANALYTICS PRIVATE LIMITED Full timeRole Overview : Seasoned Databricks Senior Platform Engineer who can implement scalable, end-to-end data solutions on Databricks over AWS. This role is pivotal in building the foundational infrastructure for data products, ensuring data quality, and enabling machine learning capabilities Key Responsibilities : - Design and implement end-to-end ETL pipelines...
-
AWS Engineer
3 weeks ago
Bengaluru, Karnataka, India DataZymes Full timeABOUT US: Founded in 2016, DataZymes is a next-generation analytics and data science company driving technology and digital-led innovation for our clients, thus helping them get more value from their data and analytics investments. Our platforms are built on best-of-breed technologies, thus protecting current investments while providing clients more bang...
-
AWS Engineer
3 weeks ago
Bengaluru, Karnataka, India DataZymes Full timeABOUT US: Founded in 2016, DataZymes is a next-generation analytics and data science company driving technology and digital-led innovation for our clients, thus helping them get more value from their data and analytics investments. Our platforms are built on best-of-breed technologies, thus protecting current investments while providing clients more bang...
-
AWS Engineer
6 days ago
Bengaluru, Karnataka, India DataZymes Full time US$ 90,000 - US$ 1,20,000 per yearABOUT US:Founded in 2016, DataZymes is a next-generation analytics and data science company driving technology and digital-led innovation for our clients, thus helping them get more value from their data and analytics investments. Our platforms are built on best-of-breed technologies, thus protecting current investments while providing clients more bang for...
-
Senior Engineering Manager
6 days ago
Bengaluru, Karnataka, India Databricks Full time US$ 1,50,000 - US$ 2,00,000 per yearP-995At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...
-
Senior Software Engineer
6 days ago
Bengaluru, Karnataka, India Databricks Full time US$ 1,50,000 - US$ 2,00,000 per yearP-1346At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...
-
Staff Software Engineer
2 days ago
Bengaluru, Karnataka, India Databricks Full time US$ 1,50,000 - US$ 2,00,000 per yearP-1346At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...
