We are looking for a Lead Full Stack Engineer to join the Big Data Engineering function at Zameen.
You will be responsible for the architecture, project management, prioritization of new workload over maintenance and evolution of the existing full-stack data gatehering solution while ensuring ongoing high standards of data quality.
You will be extracting data from publicly available internet websites and storing it in a structured format for analysis and market intelligence leveraging the existing Big Data infrastructure.
You will have the opportunity to work in a modern cloud-based data warehousing environment alongside learning from the best team of diverse and talented engineers in the region gaining invaluable experience.
● Maintaining the existing web crawlers
● Developing new web crawlers and related functionality (pipelines, database)
● Wrangle, cleanse and transform huge amount of semi-structured and structured data
● Championing the data quality of the web crawlers daily output, plan and execute corrective actions when needed
● (Assisting in) analysis related to the competitive data from the crawlers
● (Assisting in) building dashboards with the collected competitive data
● Create and Maintain web interfaces for data mapping
● Be able to work autonomously without supervision
● Ensuring data quality and system stability
● Coding standards in SQL, Python
● Working with other groups on cross-functional projects
● Requirements elicitation, documentation, and verification that the implementation matches what was requested
● Maintain documentation in an organized matter of existing capabilities of the data gathering solution
● Living the team values: Simpler. Better. Faster.
● Continually researching current and emerging technologies and proposing changes where needed
● Responsible for the technical excellence of other team member's work
● Responsible for data quality and timely delivery of KPI and reports that makes common business sense
● Responsible cost and efficiency of the technical solution
● Be able to work autonomously and supervise and review other team member's technical work
● Mentors others from a technical point of view
● Proactively engaging with the manager and other team members about progression of current sprint and roadmap
● Manages relationships with external vendors with technical support and technical architects taking ownership of successful root cause analysis and troubleshooting
● Project manage small size cross-functional project, estimate tasks effort and prioritize against value, be able to set the right level of expectation with Stakeholders and deliver within agreed timeline
● General working knowledge of e-commerce frontend, backend, native apps and API
● 4+ year demonstrated ability to create and maintain a full-stack python application framework for both web and native apps
● 1+ year experience in creating and maintaining end-to-end data pipelines (from data collection to analysis)
● Scrapy framework
● Kubernetes, Docker containers deployment (AWS EKR, EKS)
● AWS data ecosystem: especially S3, EC2, RDS, Athena
● JSON and external tables in S3 with Athena and Spectrum along with compressed columnar file format such as Parquet and AVRO
● Version Control system such as GIT with Ci/Cd
● Working experience with SQL, relational and non-relational data structures
● Manipulate and visualize data using Excel and a web-based professional data visualization tool
● REST API
● Terminal operations and Linux workflows
● Good knowledge of networking, security aspects, load balancing, and anonymization of traffic requests using proxy servers such as ProxyMesh, Luminati
● Modern data visualization platforms such as Sisense for cloud data teams (formerly Periscope Data), Google Data Studio, Tableau, etc.
● Serverless framework
● Ability to create and maintain a full-stack python application framework
● Able to manipulate and visualize data using SQL, Excel, or any other data visualization tool
● Good communication & interpersonal skills. This is an absolute must. We have heated debates around complicated subjects and you will be expected to defend your points of view.
● Ability to communicate insights and findings to a non-technical audience
● Ability to prioritize tasks and quickly adjust in a rapidly changing environment;
● Written and verbal proficiency in English
● Driven and self-motivated.
● Strong curiosity and strive for continuous learning
● Analytical thinking; Conceptual thinking; Detail-oriented; Business Acument.
● Entrepreneurial spirit and ability to think creatively; highly-driven and self-motivated; strong curiosity and strive for continuous learning
● Being part and contributing towards a strong team culture and ambition