Data Engineer
ATI Business Group · Central Jakarta
Sekilas tentang pekerjaan ini
ATI Business Group mencari Data Engineer berpengalaman minimal 3 tahun untuk mengembangkan dan memelihara pipeline serta platform data. Posisi ini berlokasi di Central Jakarta.
Tugas utama
- Merancang dan memelihara pipeline data menggunakan Databricks dan Medallion Architecture.
- Mengembangkan proses ETL produksi dengan PySpark, Python, dan SQL.
- Mengoptimalkan kinerja pipeline, Spark, dan penyimpanan data.
- Membangun workflow ingest, penyimpanan, dan orkestrasi data cloud.
- Menerapkan CI/CD dan deployment otomatis menggunakan Azure DevOps dan Git.
Kualifikasi utama
- Sarjana bidang Computer Science atau bidang teknis terkait.
- Minimal 3 tahun pengalaman relevan di Data Engineering.
- Pengalaman Databricks, Apache Spark, PySpark, SQL, Delta Lake, dan Unity Catalog.
- Pengalaman platform cloud; Microsoft Azure sangat diutamakan.
- Kemampuan Python dan SQL serta pengalaman Git dan praktik CI/CD.
Ringkasan ini dibuat dengan AI dari deskripsi di bawah. Yang berlaku adalah deskripsi lengkapnya di situs ATI Business Group.
Deskripsi pekerjaan
Key Responsibilities
- Design, develop, and maintain scalable data pipelines to ingest, transform, and process data from internal and external sources using Databricks and Medallion Architecture (Bronze, Silver, and Gold).
- Build and manage data ingestion, storage, and orchestration workflows across cloud environments, preferably using Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS), and Delta Lake.
- Develop and maintain production-ready ETL processes using PySpark, Python, and SQL, ensuring data quality, reliability, and efficient processing.
- Optimise data pipelines, Spark workloads, and storage solutions to improve performance, scalability, and cost efficiency.
- Design and develop reusable, metadata-driven engineering frameworks to standardise data ingestion, transformation, error handling, logging, and data quality processes.
- Implement and maintain CI/CD pipelines, version control, and automated deployment processes using Azure DevOps and Git.
- Collaborate with technical teams and stakeholders to understand data requirements, address engineering challenges, and continuously improve data platform capabilities.
Is this you?
- Bachelor's degree in Computer Science, Information Technology, Information Systems, Software Engineering, or a related technical field.
- At least 3 years of relevant experience in Data Engineering, particularly in developing and maintaining production data pipelines.
- Strong, hands-on experience with Databricks, including Apache Spark, PySpark, SQL, Delta Lake, and Unity Catalog.
- Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP, with Azure experience being highly preferred.
- Familiarity with cloud-based data services and ETL tools, particularly Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 (ADLS).
- Strong programming skills in Python and SQL, with the ability to develop, troubleshoot, and optimise data processing solutions.
- Experience with Git and CI/CD practices, including version control, branching strategies, and automated deployment of data solutions.
- Good understanding of data modelling, ETL/ELT processes, and Lakehouse architecture, including Medallion Architecture.
- Experience designing reusable, metadata-driven data engineering frameworks, with an understanding of modular development, testing, logging, and error handling, is a plus.
- Familiarity with Databricks Lakeflow, including Declarative Pipelines, Jobs, or Connect, is an advantage.
- Exposure to Lakebase, Databricks Apps, or Genie (AI/BI), or an interest in exploring emerging Databricks technologies, is a plus.
- Strong analytical and problem-solving skills, attention to detail, and the ability to collaborate effectively with technical teams and stakeholders.