We are looking for an experienced Full Stack Engineer to join the Big Data Engineering function at Zameen.
You will be extracting data from publicly available internet websites and storing it in a structured format for analysis and market intelligence leveraging the existing Big Data infrastructure.
You will work in a modern cloud-based data warehousing environment alongside a team of diverse, intense, and interesting co-workers.
- Maintaining the existing web crawlers
- Translate business requirements into technical solutions
- Developing new web crawlers and related functionality (pipelines, database)
- Wrangle, cleanse and transform huge amount of semi-structured and structured data
- Championing the data quality of the web crawlers daily output, plan and execute corrective actions when needed
- (Assisting in) analysis related to the competitive data from the crawlers
- (Assisting in) building dashboards with the competitive data
- Research new internet technologies to innovate existing solutions
- Create and Maintain web interfaces for data mapping
- Business acumen and attention to detail
Required minimum experience (must)
- General working knowledge of e-commerce frontend, backend, native apps and API
- 2+ year demonstrated ability to create and maintain a full stack python application framework
- 1+ year experience on creating and maintaining end-to-end data pipelines (from data collection to analysis)
- Confident with SQL, relational and non-relational data structures
- Able to manipulate and visualize data using Excel and a web-based professional data visualization tool
- Good knowledge of networking, security aspects, load balancing, and anonymization of traffic requests using proxy servers such as ProxyMesh, Luminati
- Confident with Version Control system such as GIT with Ci/Cd
- Ability to communicate insights and findings to a non-technical audience
- Written and verbal proficiency in English
- Entrepreneurial spirit and ability to think creatively; highly-driven and self-motivated; strong curiosity and strive for continuous learning
Additional experience (strong plus)
- Scrapy framework
- Experience with AWS data ecosystem (or other cloud providers)
- Knowledge of JSON and external tables in S3 with Athena and Spectrum along with compressed columnar file format such as Parquet and AVRO
- Kubernetes, Docker containers deployment (AWS EKR, EKS)
- Experience with modern big data ETL tools (e.g. Matillion)
- Experience with modern data visualization platforms such as Sisense for cloud data teams (formerly Periscope Data), Google Data Studio, Tableau, etc.
- Serverless framework