Cognitio Analytics Inc | Data Engineer/Senior Data Engineer | india
4 days ago
Experience:
1 to 8 years - DE/ Sr. DE/ AM /Manager
Must Have:
- SQL
- Python/ Pyspark
- Azure Databricks, ADLS Gen 2 Blob Storage, Key Vaults, BitBucket, Version control, Azure DevOps
- ETL, Building Data Pipelines, Data Warehousing, Data Modelling and Governance, Data Security
- Agile Practices, SDLC
- Data Exploration, Data Quality metrics, Historical data Management, Good problem-solving skill
- Excellent Comm Skills
- Exp in Data Engineering domain: 2+ Years.
- Multi years’ experience with Azure-Databricks ecosystem and Py spark.
- Ability to write clean, concise and organized Py spark code.
- Ability to break down the project into executable steps, prepare a DFD and execute the same
- Propose innovative DE solutions to achieve business objectives.
- Quick on his feet, good at tech and logically complex communication
- Good Knowledge of ADF, Docker /containerization.
Good to have:
- Event Hubs
- Any kind of health care background eg, - Pharma
- Azure Data Factory, Cosmos DB, Power BI, Azure Synapse
- Competitive coding and knows most Py spark syntax by heart.
Job Description:
We are seeking a highly skilled Azure Cloud Data Engineer to join our dynamic team. As an Azure Cloud Data Engineer, you will be responsible for designing, implementing, and managing cloud-based data solutions using Microsoft Azure. The ideal candidate should have a strong background in data engineering, with hands-on experience in Azure services and tools.
Essential Job Duties & Responsibilities:
- Set up workflows and orchestration processes to streamline data pipelines and ensure efficient data movement within the Azure ecosystem.
- Create and configure compute resources within Databricks, including All-Purpose and SQL Compute and Job Clusters to support data processing and analysis.
- Set up and manage Azure Data Lake (ADLS) Gen 2 storage accounts and establish a seamless integration with Databricks Workspace for data ingestion and processing.
- Create and manage Service Principals, key vaults to securely authenticate and authorize access to Azure resources.
- Utilize ETL (Extract, Transform, Load) techniques to design and implement data warehousing solutions and ensure compliance with data governance policies.
- Develop highly automated ETL scripts for data processing.
- Scale infrastructure resources based on workload requirements, optimizing performance and cost-efficiency.
- Profile new data sources in a different format including CSVs, JSONs etc.
- Apply problem-solving skills to address complex business and technical challenges, such as data quality issues, performance bottlenecks, and system failures.
- Demonstrate excellent soft skills and the ability to effectively communicate and collaborate with clients, stakeholders, and cross-functional teams.
- Implement Continuous Integration/Continuous Deployment (CI/CD) practices to automate the deployment and testing of data pipelines and infrastructure changes.
- Delivering tangible value very rapidly, collaborating with diverse teams of varying backgrounds and disciplines.
- Codifying best practices for future reuse in the form of accessible, reusable patterns, templates, and code bases.
- Manage timely appropriate communication and relationship with clients, partners and other stakeholders.
- Create and manage periodic reporting of project execution status and other trackers in standard accepted formats.
Required Competencies:
- Strong proficiency in SQL for data manipulation, querying, and optimization.
- Proficiency in Python and PySpark for data engineering and data processing tasks.
- Solid understanding of the Microsoft Azure tooling for large-scale data engineering efforts and deployments is highly preferred.
- Hands-on experience with Azure Databricks, including data ingestion, transformation, and analysis.
- Ability to set up and manage Azure Data Lake Storage (ADLS) Gen 2 accounts, and familiarity with data lake architecture and best practices.
- Knowledge of Azure Key Vaults for securely storing and managing cryptographic keys, secrets, and certificates.
- An ‘engineering’ mindset, willing to make rapid, pragmatic decisions to improve performance, accelerate progress or magnify impact.
- Comfort with working with distributed teams on code-based deliverables, using version control systems and code reviews.
- Use of agile and DevOps practices for project and software management including continuous integration and continuous delivery.
- Ability to conduct data analysis, investigation, and lineage studies to document and enhance data quality and access.
- Proven track record of scaling infrastructure to meet performance and scalability requirements.
- Strong analytical and problem-solving skills, with the ability to troubleshoot complex technical issues.
- Direct experience having built and deployed robust, complex production systems.
- Proficiency in Event Hubs and Azure Data Factory, particularly in the context of data streaming and transfer.
Qualifications:
- Bachelor’s degree in computer science, Engineering, or a quantitative discipline.
- 2+ years of professional experience as an Azure Cloud Data Engineer or a similar role.
- Excellent communication skills, both verbal and written.
- Strong attention to detail and the ability to work in a fast-paced environment.
-
india Cognitio Analytics Inc Full timeExperience:1 to 8 years - DE/ Sr. DE/ AM /ManagerMust Have:SQLPython/ Pyspark Azure Databricks, ADLS Gen 2 Blob Storage, Key Vaults, BitBucket, Version control, Azure DevOpsETL, Building Data Pipelines, Data Warehousing, Data Modelling and Governance, Data SecurityAgile Practices, SDLCData Exploration, Data Quality metrics, Historical data Management, Good...
-
india Cognitio Analytics Inc Full timeJob Description for Senior Clinical Tool Specialist You will assist and lead in the research, creation and development of Clinical Tools and Classification Systems such as groupers, classifiers, stagers, and patient segmentation. Work in a collaborative manner with support from data analyst to data scientist to ensure our clinical tools are world class...
-
india Cognitio Analytics Inc Full timeJob Description for Senior Clinical Tool Specialist You will assist and lead in the research, creation and development of Clinical Tools and Classification Systems such as groupers, classifiers, stagers, and patient segmentation. Work in a collaborative manner with support from data analyst to data scientist to ensure our clinical tools are world class and...
-
india Cognitio Analytics Inc Full timeJob Description for Senior Clinical Tool Specialist You will assist and lead in the research, creation and development of Clinical Tools and Classification Systems such as groupers, classifiers, stagers, and patient segmentation. Work in a collaborative manner with support from data analyst to data scientist to ensure our clinical tools are world class...
-
Data Engineer/Senior Data Engineer
4 days ago
India Cognitio Analytics Inc Full timeExperience: 1 to 8 years - DE/ Sr. DE/ AM /Manager Must Have: SQL Python/ Pyspark Azure Databricks, ADLS Gen 2 Blob Storage, Key Vaults, BitBucket, Version control, Azure DevOps ETL, Building Data Pipelines, Data Warehousing, Data Modelling and Governance, Data Security Agile Practices, SDLC Data Exploration, Data Quality metrics,...
-
india Strategic Systems Inc Full timeNeed 2 resources immediately Would prefer as much overlap with US ET and CT time zones. 2:00 pm IST to 11:00 pm IST shift is acceptable for a candidate that can work independently If the candidate passes the preliminary interview, there will be a live coding exercise as part of the technical screen. Role: Senior Data Analytics Engineer (Azure) Duration:...
-
Data Analyst to Manager
4 days ago
India Cognitio Analytics Inc Full timeExperience: 1 to 8 years - DA/ Sr. DA/ AM /Manager Must Haves: Python/PySpark SQL Advanced Excel Comm Skills Data Analysis and interpretation Project Management Story Telling Interest and ability to get to grips with the underlying business problem and value drivers of a project Good to have: Flask Team Management Any kind of...
-
Technical Lead
2 months ago
india Data Axle India Full timeData Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by leveraging...
-
india Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...
-
india Wavicle Data Solutions Full timeJob Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...
-
india Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...
-
Senior Data Engineer
1 month ago
Anywhere in India/Multiple Locations/Bangalore/Hyderabad/Chennai/Pune/Noida TETRAHED INC Full timeSenior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at TETRAHED INC. The ideal candidate will have a strong background in data engineering, with expertise in Oracle to BigQuery data warehouse migration and modernization.Key Responsibilities:Lead the migration and modernization of data warehouses from Oracle to BigQuery,...
-
india Data Minds Analytics Full timeCompany Description Data Minds is an Analytics and Technology company based in Hyderabad. We specialize in data science and AI services, focusing on product development, consulting, and corporate trainings. Role Description This is a full-time on-site role for a Python Fullstack Trainer at Data Minds Analytics in Hyderabad. The Trainer will be responsible...
-
india Data Minds Analytics Full timeCompany Description Data Minds is an Analytics and Technology company based in Hyderabad. We specialize in data science and AI services, focusing on product development, consulting, and corporate trainings. Role Description This is a full-time on-site role for a Python Fullstack Trainer at Data Minds Analytics in Hyderabad. The Trainer will be responsible...
-
Mumba Technologies, Inc. | Data Engineer | india
2 weeks ago
india Mumba Technologies, Inc. Full timeJob Title: Data Engineer Location: Remote Type: Contract to Hire Time Shift: EST Hrs About The Role & Team We are seeking a Sr. Data Engineer to join our Data Engineering team within our Enterprise Data Insights organization to build data solutions, design and implement ETL/ELT processes and manage our data platform to enable our cross functional...
-
Data Analytics Engineer
2 weeks ago
India Celito Tech, Inc. Full timeData Analytics Engineer Job Title: Data Analytics Engineer Reports To: Technical Architect (Data & Analytics), India Employment Type: FTE Location : Remote (India) Job Summary This role requires a skilled Data Analytics Engineer with a strong background in data analysis, data engineering, analytics development and...
-
Data Analytics Engineer
2 months ago
India Celito Tech, Inc. Full timeData Analytics Engineer Job Title: Data Analytics Engineer Reports To: Technical Architect (Data & Analytics), IndiaEmployment Type: FTELocation : Remote (India) Job SummaryThis role requires a skilled Data Analytics Engineer with a strong background in data analysis, data engineering, analytics development and administration of data analytics platforms....
-
Data Analytics Engineer
1 month ago
india Celito Tech, Inc. Full timeData Analytics Engineer Job Title: Data Analytics Engineer Reports To: Technical Architect (Data & Analytics), India Employment Type: FTE Location : Remote (India) Job Summary This role requires a skilled Data Analytics Engineer with a strong background in data analysis, data engineering, analytics development and administration of data...
-
Data Analytics Engineer
2 months ago
india Celito Tech, Inc. Full timeData Analytics Engineer Job Title: Data Analytics Engineer Reports To: Technical Architect (Data & Analytics), IndiaEmployment Type: FTELocation : Remote (India) Job SummaryThis role requires a skilled Data Analytics Engineer with a strong background in data analysis, data engineering, analytics development and administration of data analytics platforms....
-
Data Analytics Engineer
2 months ago
India Celito Tech, Inc. Full timeData Analytics Engineer Job Title: Data Analytics Engineer Reports To: Technical Architect (Data & Analytics), India Employment Type: FTE Location : Remote (India) Job Summary This role requires a skilled Data Analytics Engineer with a strong background in data analysis, data engineering, analytics development and administration of data...
