Data Engineer

ATI Business Group · Central Jakarta

Lamar

Sekilas tentang pekerjaan ini

ATI Business Group mencari Data Engineer berpengalaman minimal 3 tahun untuk mengembangkan dan memelihara pipeline serta platform data. Posisi ini berlokasi di Central Jakarta.

Tugas utama

  • Merancang dan memelihara pipeline data menggunakan Databricks dan Medallion Architecture.
  • Mengembangkan proses ETL produksi dengan PySpark, Python, dan SQL.
  • Mengoptimalkan kinerja pipeline, Spark, dan penyimpanan data.
  • Membangun workflow ingest, penyimpanan, dan orkestrasi data cloud.
  • Menerapkan CI/CD dan deployment otomatis menggunakan Azure DevOps dan Git.

Kualifikasi utama

  • Sarjana bidang Computer Science atau bidang teknis terkait.
  • Minimal 3 tahun pengalaman relevan di Data Engineering.
  • Pengalaman Databricks, Apache Spark, PySpark, SQL, Delta Lake, dan Unity Catalog.
  • Pengalaman platform cloud; Microsoft Azure sangat diutamakan.
  • Kemampuan Python dan SQL serta pengalaman Git dan praktik CI/CD.

Ringkasan ini dibuat dengan AI dari deskripsi di bawah. Yang berlaku adalah deskripsi lengkapnya di situs ATI Business Group.

Deskripsi pekerjaan

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines to ingest, transform, and process data from internal and external sources using Databricks and Medallion Architecture (Bronze, Silver, and Gold).
  • Build and manage data ingestion, storage, and orchestration workflows across cloud environments, preferably using Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS), and Delta Lake.
  • Develop and maintain production-ready ETL processes using PySpark, Python, and SQL, ensuring data quality, reliability, and efficient processing.
  • Optimise data pipelines, Spark workloads, and storage solutions to improve performance, scalability, and cost efficiency.
  • Design and develop reusable, metadata-driven engineering frameworks to standardise data ingestion, transformation, error handling, logging, and data quality processes.
  • Implement and maintain CI/CD pipelines, version control, and automated deployment processes using Azure DevOps and Git.
  • Collaborate with technical teams and stakeholders to understand data requirements, address engineering challenges, and continuously improve data platform capabilities.

Is this you?

  • Bachelor's degree in Computer Science, Information Technology, Information Systems, Software Engineering, or a related technical field.
  • At least 3 years of relevant experience in Data Engineering, particularly in developing and maintaining production data pipelines.
  • Strong, hands-on experience with Databricks, including Apache Spark, PySpark, SQL, Delta Lake, and Unity Catalog.
  • Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP, with Azure experience being highly preferred.
  • Familiarity with cloud-based data services and ETL tools, particularly Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 (ADLS).
  • Strong programming skills in Python and SQL, with the ability to develop, troubleshoot, and optimise data processing solutions.
  • Experience with Git and CI/CD practices, including version control, branching strategies, and automated deployment of data solutions.
  • Good understanding of data modelling, ETL/ELT processes, and Lakehouse architecture, including Medallion Architecture.
  • Experience designing reusable, metadata-driven data engineering frameworks, with an understanding of modular development, testing, logging, and error handling, is a plus.
  • Familiarity with Databricks Lakeflow, including Declarative Pipelines, Jobs, or Connect, is an advantage.
  • Exposure to Lakebase, Databricks Apps, or Genie (AI/BI), or an interest in exploring emerging Databricks technologies, is a plus.
  • Strong analytical and problem-solving skills, attention to detail, and the ability to collaborate effectively with technical teams and stakeholders.
Lowongan ini diambil dari situs karier resmi ATI Business Group. Lokr tidak memproses lamaran: kamu melamar langsung di situs perusahaan, dan info terbaru tentang lowongan ini ada di sana.

Lowongan lain di ATI Business Group

Lowongan Komputer & Perangkat Lunak di perusahaan lain