This position will be filled in the Data Analytics team within Technology Services. Data Engineer will be responsible for designing, developing, and maintaining our data infrastructure. This role involves working closely with data scientists, analysts, and other stakeholders to ensure that data pipelines are efficient, reliable, and scalable. The ideal candidate will have a strong background in data engineering, a passion for technology, and a commitment to delivering high-quality solutions.
Education/Experience:
- Bachelor’s degree in computer science/engineering or related degree preferred
- Experience with implementing analytics solutions using the Microsoft analytics toolset and Microsoft Azure
- Experience working in a fast-paced, competitive information technology organization
- Exempt 05 requires minimum 3 years of experience
- Exempt 06 requires minimum 5 years of experience
Knowledge/Skills:
1. Strong Programming Skills
- Languages: Proficient in languages like SQL, Python, or Java. These are essential for building data pipelines, scripting automation, and manipulating data.
- Efficiency: Writing efficient, scalable, and clean code to handle large datasets and optimize performance.
2. Deep Knowledge of Databases & Data Warehousing
- SQL and NoSQL: Mastery in relational databases (like MS SQL Server, MySQL, PostgreSQL, Oracle) and NoSQL databases (like MongoDB, Cassandra, DynamoDB), knowing when and how to use each type.
- Data Warehousing: Expertise in using platforms like MS SQL Server, Oracle, Amazon Redshift, Google BigQuery, Snowflake, or traditional data warehouses to store, query, and manage large volumes of data.
- Database Optimization: Skills in optimizing queries, indexing, partitioning, and designing efficient database schemas for high performance.
3. Expertise in Data Pipeline Construction
- ETL/ELT Processes: Proven ability to design and build robust ETL pipelines (Extract, Transform, Load) for collecting, cleaning, and moving data.
- Real-time and Batch Processing: Experience working with both batch and real-time data processing frameworks (e.g., Apache Kafka, Apache Flink, Apache Spark, Databricks).
- Data Orchestration: Familiarity with orchestration tools like Apache Airflow.
- Strong knowledge of Medallion Architecture
4. Cloud Platform Expertise
- Cloud Services: Proficiency in cloud platforms like Azure* (SQL Database, Data Lake, Lake House), AWS (S3, Lambda, Redshift), or Google Cloud (BigQuery, Dataflow). *Azure is preferred.
- Scalability: Ability to design systems that scale efficiently in the cloud, handling big data and increasing demand without sacrificing performance.
5. Data Transformation and Cleaning Skills
- Data Quality Management: Experience in data cleansing, validation, and transformation, ensuring that data is accurate, complete, and in the right format for analysis.
- Data Integration: Expertise in integrating data from diverse sources (internal and external) while resolving issues like inconsistency or format mismatches.
6. Performance Optimization and Troubleshooting
- Query Optimization: Ability to fine-tune queries, databases, and pipelines to reduce latency, optimize resource usage, and speed up data processing.
- System Monitoring: Familiarity with monitoring systems and logging tools to detect, diagnose, and resolve performance or data issues.
7. Strong Analytical Skills
- Data Interpretation: Ability to translate business requirements into technical solutions, ensuring the correct data is collected and processed for reporting, analytics, and decision-making.
- Problem Solving: A strong ability to troubleshoot and resolve complex data challenges or inconsistencies that can affect the integrity and availability of data.
8. Knowledge of Big Data Tools
- Big Data Frameworks: Familiarity with big data technologies such as Spark or Flink for processing large datasets across distributed systems.
- Data Lakes and Data Pipelines: Experience with data lakes (e.g., Azure Lakehouse, AWS S3, HDFS) for storing raw and unstructured data and building pipelines to process it efficiently.
9. Collaboration and Communication Skills
- Cross-functional Collaboration: Proven ability to work closely with data scientists, analysts, and other stakeholders to understand data needs and deliver optimal solutions.
- Clear Communication: Ability to explain technical concepts to non-technical stakeholders, ensuring that data infrastructure decisions align with business goals.
10. Data Security & Governance Awareness
- Data Privacy: Knowledge of data privacy regulations (e.g., GDPR, CCPA) and ensuring that systems comply with these laws while managing sensitive data.
- Access Control: Implementing strong data access controls, encryption, and monitoring to secure data both at rest and in transit.
11. Adaptability to New Tools and Technologies
- Continuous Learning: A strong commitment to keeping up with the rapidly evolving tech landscape, experimenting with and implementing new tools, frameworks, and approaches to data engineering.
- Flexibility: Adaptability to changing data architectures or business needs, ensuring data systems remain resilient and future-proof.
12. Automation & DevOps Skills
- Automation Tools: Familiarity with CI/CD pipelines and automation tools (like Jenkins, Docker, Kubernetes) to streamline development and deployment of data engineering solutions.
- Infrastructure as Code (IaC): Experience with tools like Terraform or Azure Resource Manager (ARM) to manage data infrastructure efficiently.
General Job Responsibilities:
- Data Pipeline Development:
- Designing, building, and maintaining robust data pipelines to collect, clean, and process data from various sources. Using Microsoft Analytics Tool stack on-prem and Azure Cloud. For example, SSIS, SSAS, SQL Server, Azure Lake House, MS Fabric, Databricks etc.
- Ensuring data is stored efficiently for easy access and retrieval (e.g., in data lakes, warehouses).
- Data Integration:
- Integrating data from multiple systems, applications, and external sources.
- Ensuring data is harmonized and available in a format suitable for analysis.
- Database Management:
- Creating and managing databases, ensuring that they are optimized for performance, security, and scalability.
- Handling structured and unstructured data.
- Data Quality Assurance:
- Monitoring and ensuring the accuracy, consistency, and reliability of data.
- Implementing processes for data validation, cleansing, and enrichment.
- Performance Optimization:
- Optimizing queries and databases for faster performance and lower latency.
- Tuning and troubleshooting database performance issues.
- Collaboration with Data Scientists & Analysts:
- Working closely with data scientists, analysts, and other stakeholders to understand data needs.
- Ensuring data is ready and available for analysis, machine learning models, and reporting.
- ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) Processes:
- Implementing ETL/ELT processes to extract data from different sources, transform it into usable formats, and load it into data warehouses or lakes.
- Understanding the different and when to apply each approach
- Automation and Monitoring:
- Automating repetitive tasks related to data processing and integration.
- Setting up monitoring to track data quality, pipeline performance, and system health.
- Data Security & Compliance:
- Ensuring data security measures are in place to protect sensitive information.
- Adhering to data privacy regulations and compliance standards, such as GDPR or CCPA.
- Tool and Technology Evaluation:
- Evaluating and implementing new tools, frameworks, and technologies to improve data infrastructure.
- Keeping up to date with the latest trends in big data and cloud technologies.
Core Behavioral Attributes:
Additional required behavioral attributes:
- Results-oriented
- Innovative
- Strategic thinker with an enterprise view for sustainable solutions
- Committed to continuous learning and improvement
- Committed to the development of others
- Committed to building and maintaining constructive partnerships with business partners
- Works well both independently and with others
- Acts with speed and decisiveness
- Committed to ethical conduct
- Lives and works safely
Note: The position is Hybrid, Atlanta, GA or Birmingham, AL. Four days in the office; and one day remote a week.
About Southern Company
Southern Company (NYSE: SO ) is a leading energy provider serving 9 million customers across the Southeast and beyond through its family of companies. Providing clean, safe, reliable and affordable energy with excellent service is our mission. The company has electric operating companies in three states, natural gas distribution companies in four states, a competitive generation company, a leading distributed energy solutions provider with national capabilities, a fiber optics network and telecommunications services. Through an industry-leading commitment to innovation, resilience and sustainability, we are taking action to meet customers' and communities' needs while advancing our goal of net-zero greenhouse gas emissions by 2050. Our uncompromising values ensure we put the needs of those we serve at the center of everything we do and are the key to our sustained success. We are transforming energy into economic, environmental and social progress for tomorrow. Our corporate culture has been recognized by a variety of organizations, earning the company awards and recognitions that reflect Our Values and dedication to service. To learn more, visit www.southerncompany.com .
Southern Company invests in the well-being of its employees and their families through a comprehensive total rewards strategy that includes competitive base salary, annual incentive awards for eligible employees and health, welfare and retirement benefits designed to support physical, financial, and emotional/social well-being. This position may also be eligible for additional compensation, such as an incentive program, with the amount of any bonus/awards subject to the terms and conditions of the applicable incentive plan(s). A summary of the benefits offered for this position can be found here https://seo.nlx.org/southernco/pdf/SOCO-Benefits.pdf . Additional and specific details about total compensation and benefits will also be provided during the hiring process.
Southern Company is an equal opportunity employer where an applicant's qualifications are considered without regard to race, color, religion, sex, national origin, age, disability, veteran status, genetic information, sexual orientation, gender identity or expression, or any other basis prohibited by law.
Job Identification: 18886
Job Category: Information Technology
Job Schedule: Full time
Company: Southern Company Services
