**DESCRIPTION** Supports, develops, and maintains a data and analytics platform. Effectively and efficiently processes, stores, and makes data available to analysts and other consumers. Works with Business and IT teams to understand requirements and best leverage technologies to enable agile data delivery at scale. **Note:- Although the role category in the GPP is listed as Remote, the requirement is for a Hybrid work model.** **Key Responsibilities:** + Oversee the development and deployment of end-to-end data ingestion pipelines using Azure Databricks, Apache Spark, and related technologies. + Design high-performance, resilient, and scalable data architectures for data ingestion and processing. + Provide technical guidance and mentorship to a team of data engineers. + Collaborate with data scientists, business analysts, and stakeholders to integrate various data sources into the data lake/warehouse. + Optimize data pipelines for speed, reliability, and cost efficiency in an Azure environment. + Enforce and advocate for best practices in coding standards, version control, testing, and documentation. + Work with Azure services such as Azure Data Lake Storage, Azure SQL Data Warehouse, Azure Synapse Analytics, and Azure Blob Storage. + Implement data validation and data quality checks to ensure consistency, accuracy, and integrity. + Identify and resolve complex technical issues proactively. + Develop reliable, efficient, and scalable data pipelines with monitoring and alert mechanisms. + Use agile development methodologies, including DevOps, Scrum, and Kanban. **RESPONSIBILITIES** **Technical Skills:** + Expertise in Spark, including optimization, debugging, and troubleshooting. + Proficiency in Azure Databricks for distributed data processing. + Strong coding skills in Python and Scala for data processing. + Experience with SQL for handling large datasets. + Knowledge of data formats such as Iceberg, Parquet, ORC, and Delta Lake. + Understanding of cloud infrastructure and architecture principles, especially within Azure. **Leadership&Soft Skills:** + Proven ability to lead and mentor a team of data engineers. + Excellent communication and interpersonal skills. + Strong organizational skills with the ability to manage multiple tasks and priorities. + Ability to work in a fast-paced, constantly evolving environment. + Strong problem-solving, analytical, and troubleshooting abilities. + Ability to collaborate effectively with cross-functional teams. **Competencies:** + **System Requirements Engineering:** Uses appropriate methods to translate stakeholder needs into verifiable requirements. + **Collaborates:** Builds partnerships and works collaboratively to meet shared objectives. + **Communicates Effectively:** Delivers clear, multi-mode communications tailored to different audiences. + **Customer Focus:** Builds strong customer relationships and delivers customer-centric solutions. + **Decision Quality:** Makes good and timely decisions to keep the organization moving forward. + **Data Extraction:** Performs ETL activities and transforms data for consumption by downstream applications. + **Programming:** Writes and tests computer code, version control, and build automation. + **Quality Assurance Metrics:** Uses measurement science to assess solution effectiveness. + **Solution Documentation:** Documents information for improved productivity and knowledge transfer. + **Solution Validation Testing:** Ensures solutions meet design and customer requirements. + **Data Quality:** Identifies, understands, and corrects data flaws. + **Problem Solving:** Uses systematic analysis to address and resolve issues. + **Values Differences:** Recognizes the value that diverse perspectives bring to an organization. **Preferred Knowledge&Experience:** + Exposure to Big Data open-source technologies (Spark, Scala/Java, Map-Reduce, Hive, HBase, Kafka, etc.). + Experience with SQL and working with large datasets. + Clustered compute cloud-based implementation experience. + Familiarity with developing applications requiring large file movement in a cloud-based environment. + Exposure to Agile software development and analytical solutions. + Exposure to IoT technology. **QUALIFICATIONS** **Qualifications:** + **Education:** Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field. + **Experience:** + 3 to 5 years of experience in data engineering or a related field. + Strong hands-on experience with Azure Databricks, Apache Spark, Python/Scala, CI/CD, Snowflake, and Qlik for data processing. + Experience working with multiple file formats like Parquet, Delta, and Iceberg. + Knowledge of Kafka or similar streaming technologies. + Experience with data governance and data security in Azure. + Proven track record of building large-scale data ingestion and ETL pipelines in cloud environments. + Deep understanding of Azure Data Services. + Experience with CI/CD pipelines, version control (Git), Jenkins, and agile methodologies. + Familiarity with data lakes, data warehouses, and modern data architectures. + Experience with Qlik Replicate (optional). **Job** Systems/Information Technology **Organization** Cummins Inc. **Role Category** Remote **Job Type** Exempt - Experienced **ReqID** 2411635 **Relocation Package** No