Lead Data Engineer
London – Hybrid
We are looking for a Lead Data Engineer to lead our Data Ingest team, responsible for scraping the data that fuels our products. You’ll play a pivotal role in shaping how we capture, process and scale the data pipelines that underpin our insights.
What you’ll be working on:
Line management of the Data Ingest team engineers
Translating the strategic roadmap into actionable small tasks and providing visibility on delivery progress to stakeholders
Doing or overseeing research and technical investigations, representing and prioritising the resulting improvements against other stakeholder priorities
Improving and automating team processes – release of new features, cross-team collaboration and inspiring the culture of innovation and growth in the team
Implementation of new and maintenance of existing scrapers
Migration of scrapers to AWS and from old Jenkins-based orchestration to a new microservice-based system
Improvements to the scraping platform in the areas of data quality, performance and observability
About the team:
You will join a technical team of c. 40 technologists with a key purpose of continuously and reliably collecting large amounts of financial data describing cross-border payments, then processing the data, checking the quality and serving it to our users so that they can make the best business decisions
The Data Ingest team is the oldest team at FXC Intelligence, existing in some form for over six years now. We provide the platform for collecting unstructured data from open third-party websites and mobile apps, and make sure that the data collected corresponds to the company's data quality standards and SLAs. As the business grows and requirements change, so do we – the platform has survived several iterations and is currently in the process of being migrated to its fourth generation. The team also maintains over 100 scrapers for different providers of data, which power most of the company’s critical datasets
You should apply if you have:
Advanced skills in Python and SQL
Involvement in web scraping and/or backend development
Experience utilising AI to increase team productivity, efficiency and data quality
A proactive approach, with the ability to set and exemplify high standards
The ability to work in a fast-paced environment and adapt to change quickly
Excellent communication skills, with the ability to communicate clearly and effectively
Proven experience and a desire to manage a team
A desire to understand and engage with the business domain and product
A love for learning and improving your skills
These skills will help, but aren’t essential:
Prior experience working for a data product company or being involved with Big Data
Experience with advanced database management and data modelling
Experience building or maintaining third-party or in-house data quality and cataloguing solutions
DBT, Airflow (or other task orchestration tools), Kafka/RabbitMQ
Experience with documentation of system architecture
Experience collaborating with a BI, DA and/or ML team
Tech Stack:
Python, PostgreSQL, RabbitMQ, Redis, Puppeteer, Playwright, Selenium, Pandas, Numpy and SKlearn
AI Usage Disclaimer:
At FXC Intelligence, we are enthusiastic about the use of AI tools and value candidates with experience or interest in AI technologies. However, we kindly ask that candidates refrain from using AI tools during interviews or for technical tasks unless explicitly instructed by us. This is because we want to understand your thought process and approach to tasks. If AI assistance is used in tasks or interviews where it was not permitted and is detected, the submission may be discounted.
About us:
FXC Intelligence is a leading provider of cross-border payments data and intelligence, providing some of the world's biggest companies, central banks and non-governmental organisations with the strategic insights, expertise and awareness to effectively compete in their chosen markets. By joining us, you will be diving into a world of data-driven exploration and innovation, revolutionising financial insights through cutting-edge technologies, machine learning and predictive analytics.
Your contributions will shape the future of cross-border finance, helping clients to uncover better paths to growth and profitability, as well as being a trusted reference and source for many of the leading international publications.
We are proud to produce industry-changing data and intelligence, aided by our company values of being customer-focused, taking ownership, knowledge, communication and leadership.
What’s in it for you?
25 days holiday per annum (plus bank holidays), increasing to 26 days after 2 years of service
Healthcare for you and your family with AXA
Annual bonus/option scheme
Pension scheme
Enhanced maternity and paternity policy
After four years of employment, you are entitled to a 6-week paid sabbatical
After one year of employment, you can work abroad for up to three months of the year, with approval from your manager
Quarterly external one-to-one coaching sessions
£1,000 annual learning budget for courses, books and conferences
Annual salary reviews with your manager
A chance to work with a vibrant, friendly team in a relaxed but committed culture
Monthly London-based socials paid for by the company, from mini golf to cooking challenges to escape rooms!
We operate in a hybrid working environment, trusting employees to work where they work best, with the expectation that you come into our London Liverpool Street office each month for company update meetings and for ad-hoc team sessions
The opportunity to join our LGBTQIA+ and Women in Fintech employee networks
In our last engagement survey, 95% of employees felt that "People from all backgrounds are being treated fairly”
We’re an innovative company that strives to look after its team, and we take pride in providing a positive company culture. Feel free to check out our blog, Instagram, and Glassdoor pages to learn more about our culture and hear directly from our team about what it's really like to work with us!
At FXC Intelligence, we believe in embracing diversity in all forms and fostering an inclusive environment. All applicants will be considered for employment without attention to ethnicity, religion, sexual orientation, gender identity, family or parental status, national origin, veteran status, neurodiversity status or disability status.