Databricks AWS Engineer

Profile

The Senior Databricks Engineer will help us build a solid, scalable data infrastructure and analytical environment. The ideal candidate enjoys data wrangling, optimizing data pipelines, and building solutions for data collection and analysis. Work assignments may cover requirements gathering, source-to-target mapping, data validation scripting and review, developing and monitoring ETL/ELT data pipelines, designing and implementing database schema/tables/views, and producing datasets as input to data science models and visualizations. The Senior data engineer will work directly with the Director of BI and Analytics to ensure consistency and compliance of deliverables to frameworks and governance processes. Duties and Responsibilities Design efficient and scalable data processing systems and pipelines on Databricks, APIs, and AWS Services. Design and implement tools to detect data anomalies. Ensure that data is accurate, complete, and across all platforms. Develop data models and mappings and build new data assets required by stakeholders. Perform exploratory analysis on existing products and datasets. Develop scalable and reusable frameworks for the ingestion and transformation of large datasets. Understand trends and the latest technologies. Evaluate the performance and applicability of potential tools for our requirements. Build the infrastructure required for optimal extraction, transformation, and loading of data from various data sources. Work with stakeholders, including the Executive, Product, Data, and Design teams, to assist with data-related technical issues and support their data infrastructure needs. Partner to deliver a modern data engineering model that follows Dev/Ops principles and standards for continuous integration/ continuous delivery (CI/CD) processes. Establish the documentation of reports and develops and maintain technical specification documentation for all reports and processes within Databricks. Collaborate with IT and business partners to define, manage and deliver innovative Data solutions to drive growth and adoption of capabilities Mentor team members on best practices, CI/CD, and pipeline orchestration. Assist with the administrative tasks of the Databricks platform, such as job schedule monitoring, users access control, and performance management. Other duties as required. Required Qualifications Education/ Years of Experience Bachelor's degree in computer science, information technology, or related field. Master's degree preferred. 5 years of experience in a data engineer role, including 2 years designing and building Databricks data pipelines, is REQUIRED. 3 years of SQL experience, specifically to write complex, highly optimized queries across disparate data sets, is REQUIRED. 2 years of hands-on Python/PySpark/SparkSQL experience is REQUIRED; experience with Scala is a plus. 2 years of Spark experience (especially Databricks Spark and Delta Lake) is REQUIRED. 2 years of experience deploying code using a source control product like GitLab/GitHub is REQUIRED. 2 years implementing data management life cycle, using data quality functions like standardization, transformation, rationalization, linking, and matching is REQUIRED. 2 years performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement is REQUIRED. 2 years of experience building and optimizing big data pipelines, architectures, and data sets. In-depth knowledge of Model and Design of DB schemas for read-and-write performance. Professional Experience with cloud data warehouse solutions (e.g., Redshift, Snowflake, Databricks) is HIGHLY DESIRABLE. Experience with Databricks Administration is PREFERRED. Working knowledge of API or stream-based data extraction processes is PREFERRED. Knowledge, Skills, Abilities, and Other Characteristics (KSAOs) Strong project management and organizational skills. Experience supporting and working with cross-functional teams and enterprise-wide data programs. Exceptional written, verbal, and listening communication skills Proven analytical, problem-solving, decision-making, and prioritization skills Able to contribute to the big picture, conceptualize outcomes, and identify trends
**MEMBERS ONLY**SIGN UP NOW***.

Company info

Sign Up Now - WashingtondcCrossing.com