Senior Data Engineer(Databricks) - New York

Data Engineer with Databricks & Spark Experience

We are looking for a data engineer who has strong experience in building scalable and reliable data pipelines using Databricks and Spark. You will be working with various data sources and formats, and transforming them into valuable insights for our business.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks and Spark, and other cloud technologies as needed
• Optimize data pipelines for performance, scalability, and reliability
• Ensure data quality and integrity throughout the data lifecycle
• Collaborate with data scientists, analysts, and other stakeholders to understand and meet their data needs
• Troubleshoot and resolve data-related issues, and provide root cause analysis and recommendations
• Document data pipeline specifications, requirements, and enhancements, and communicate them effectively to the team and management
• Create new data validation methods and data analysis tools, and share best practices and learnings with the data engineering community
• Implement ETL processes and data warehouse solutions, and ensure compliance with data governance and security policies

Please Note: This is a hybrid position and will require at least 2 days in the office per week. Successful candidates will need to complete a background check.

Requirements

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent work experience
• 5+ years of experience in data engineering with Databricks and Spark
• Proficient in SQL and Python, and familiar with Java or Scala
• Experience with cloud platforms, such as Azure or AWS
• Experience with big data technologies, such as Kafka, Hadoop, Hive, etc.
• Experience with data warehouse and data lake concepts and architectures
• Experience with data integration and ETL tools, such as Azure Data Factory or Talend
• Experience with data visualization and reporting tools, such as Power BI or Tableau
• Strong analytical and problem-solving skills
• Excellent communication and teamwork skills

Benefits

Benefits

• We are a collective group of people and collaboration is key to our process
• We don’t work for our clients, we work with them
• A Flexible Hybrid Working Environment
• Easily accessible downtown location
• Competitive compensation commensurate with experience
• Everyone brings something valuable to the table in our supportive, challenging, and collaborative, diverse work environment
  
  

All Employees Can Participate in:

• Company paid health benefits: 100% medical, dental and vision coverage
• Corporate-discounted Gym Membership through GoodLife
• Company discount program including Travel, Shopping, Attractions, Wellness, & Sporting events, just to name a few
• Access to an Employee & Family Assistance program (EAP)
• Employee Referral Program
• Employee Opportunity Program
• Professional Development Program
• Philanthropic Events
• Social Events
  
  

Inclusion & Diversity

We foster an inclusive and diverse workforce, believing our strength stems from our individual differences. Our employees, partners, and clients continuously benefit from the innovation and creativity grounded in these values. We strive to be a company that attracts a diverse group of highly skilled people who know that their contributions will be valued and that they will be heard. We are committed to building a corporate culture with people who are excited to join our team, do their best work, and grow with us!