รับสมัคร Data Engineer

Tradesquare Co.,LTD

หมวดหมู่ งานวิเคราะห์ธุรกิจ งานวิเคราะห์ข้อมูล

จำนวนที่รับ 1 อัตรา

ลักษณะงาน

Key Responsibilities Customer-Centric Approach: - Engage with clients to understand their specific data needs, business objectives, and challenges - Design data solutions that align with their goals and encourage a data-driven culture within their organization Data Lake Design: - Architect and implement a multi-tiered data zone such as raw, curated, refined, and production-ready layers Data Pipeline Development: - Design, develop, and maintain ETL pipelines to ingest data from various service into the AWS Data Lake solution, Lake House Architecture using various services such as AWS Glue services, open table format e.g. Apache Hudi Data Transformation: - Perform data cleansing, transformation, and enrichment processes using Glue Spark ETL, Python and open table format to ensure data quality and consistency, all while addressing client requirements SQL optimization: - Perform SQL script optimization from customer existing report or dataset utilizing SparkSQL, PySpark or native-python - Optimization should be done according to multi-tiered data zones on Data Lake architecture design Performance Optimization: - Monitor and optimize data pipelines, especially those using Apache Spark, for performance, scalability, and cost-effectiveness for batch and schedule jobs while ensuring alignment with client goals Collaboration: - Work closely with data scientists, analysts, and other stakeholders to understand their data requirements and provide the necessary data engineering support Documentation: - Maintain clear and comprehensive documentation of data pipelines, processes, and workflows, focusing on client-specific configurations and requirements Apache Hudi Integration: Leverage Apache Hudi for incremental data processing, data partitioning, data deduplication, and simplifying data ingestion processes as per client needs Troubleshooting: - Identify and resolve data-related issues and provide support for data-related incidents, with a focus on meeting client expectations Continuous Learning: - Stay up-to-date with the latest AWS data service, Apache Spark, Apache Hudi, and data engineering trends, best practices, and tools to better serve our clients

คุณสมบัติผู้สมัคร

Requirements - Bachelor's degree in Computer Science, Information Technology, or a related field - Proven experience as a Data Engineer, with a focus on building data pipelines using Apache Spark, Hive, open table format, and AWS Data Lake while working closely with clients - Proficiency in AWS services such as AWS Glue, Lake Formation, S3, Athena and Lambda or equivalent tools on another public cloud providers - Proficiency in programming languages such as SQL, Python, Scala, or Java - Proficiency in SQL script optimization and conversion to SparkSQL, PySpark or native-Python - Experience with big data processing frameworks like Apache Spark - Solid understanding of data modeling, database design, and data warehousing concepts - Knowledge of data governance, data quality, and data security best practices and the ability to tailor them to client needs - Experience with Apache Hudi for managing data changes and improving data reliability - Excellent problem-solving, communication, and client-facing skills - AWS certifications (e.g., AWS Certified Data Analytics, AWS Certified Data Analytics) a plus. - Able to communicate in English mainly documentation

เงินเดือน/ค่าตอบแทน

- ตามความสามารถผู้สมัคร

สวัสดิการ

จำนวนวันและเวลาทำงาน

5 วัน/สัปดาห์