Senior Data Engineer

Overview:

SOSi is seeking a Senior Data Engineer to assist the MLOps Team in creating, maintaining, and automating workflows leveraging cutting-edge technologies and supporting a government customer in Huntsville, Alabama . This role will be responsible for delivering automation to key national security missions interacting with petabyte-scale data using High-Performance Computing (HPC) environment.

In this role, you will lead a group of fellow Data Engineers and working with Data Scientists across the organization to construct digital infrastructure necessary to produce and deploy Machine Learning models. You will leverage scientific techniques to iteratively transform petabyte-scale data, construct efficient and effective Extract, Transform, and Load (ETL) pipelines, and refactor manual workflows into containerized-services. You will work with multiple database types including SQL and NoSQL databases, leveraging different storage techniques to maximize efficiency and capability of analytics products. Working on the client-site, you will be embedded with the end-user working to understand their processes and improve upon current standards.

The team will work with Open-Source technologies including:

• Scripting languages such as Python and Go.
• Databases such as Postgres and Arango DB.
• Apache products including Apache Airflow, Apache Spark, Apache AGE.
• Containerization software with Docker, Podman, Kubernetes, and Kubeflow.
• Configuration management tools like yaml, Github, Gitlab, and FastAPI.
• Agile tools such as Confluence, Jira, Scrum Methodologies.

SOSi is seeking Senior Data Engineering talent to unlock insights from massive datasets supporting strategic national security objectives. If you have a passion for data and want to apply your skills to impactful projects at scale, we want to hear from you!

Responsibilities:

• Lead the design, build, and maintenance of ETL processes.
• Configure storage systems for efficiency and effectiveness.
• Architect and develop services and extend infrastructure to enable MLOps.
• Oversee the integration of software components into functional data systems.
• Write clean, testable, maintainable code in Python and other languages.
• Create and maintain technical documentation for software systems.
• Collaborate across teams to share knowledge and leading practices.

Qualifications:

• Top Secret Security Clearance with SCI eligibility.
• 4-6 years of experience as a Data Engineer, advanced analytical role or very strong work samples.
• Extensive experience in data engineering and software development.
• Expert-level proficiency with Python, GIT, YAML, Docker, and SQL.
• Deep understanding of CI/CD, DevSecOps, and Agile methodologies.
• Proven track record of developing back-end systems and services at scale.
• Strong understanding of software design and system integration.
• Experience leading and mentoring data engineering teams.

Preferred Qualifications:

• Experience as a Site Reliability Engineer (SRE).
• Experience with Cloud-Native Services.
• Experience with MLOps and/or DevOps.
• Experience with large-scale data processing.
• Experience with multi-INT analytics.
• MS or PhD in Computer Science, Computer Engineering, Statistics, Mathematics, Physics or a quantitative field.

Working Conditions:

• On-site.
• Working conditions are normal for an office environment.
• Fast paced, deadline-oriented environment.
• May require periods of non-traditional working hours including consecutive nights or weekends (if applicable).