Client & context
A client needed to run a news aggregation and ranking platform that:
- Collected and ranked content based on social network activity and other signals.
- Supported experimentation with ranking algorithms.
- Powered a Next.js newsletter front-end and automated newsletter creation workflows.
I joined to sustain and enhance an existing Rust application and its surrounding tooling and integrations.
Challenges
- The Rust codebase for news aggregation was hard to modify and lacked a clean development environment.
- Existing ranking algorithms needed both performance improvements and better readability.
- The system had to integrate with:
- Multiple data providers.
- A Next.js front-end.
- Google Sheets and LLM-based processing for newsletters.
- The business needed better monitoring, logs, and reports to trust the system.
What I did
1. Algorithm and codebase improvements in Rust
- Researched the existing ranking algorithms, including their mathematical foundations.
- Fixed bugs and improved performance of the core logic.
- Refactored the codebase to make the structure more readable and maintainable.
- Added a Docker-based development environment to make it easy to run and test the system locally.
2. Experimentation and data sourcing
- Helped build code paths for testing hypotheses and tuning algorithms.
- Implemented text matching using embeddings to better relate posts and articles.
- Built a custom web scraper using a headless browser for sources not covered by standard APIs.
- Implemented fetching from two primary data providers, selecting from test integrations with other providers for coverage and quality.
3. Newsletter front-end & AI-assisted content workflows
- Made integrations, enhancements, and design improvements for a Next.js newsletter front-end application.
- Implemented an integration between Google Spreadsheets and various LLMs to:
- Process and summarize texts.
- Format data for newsletters.
- Export data into parameterized Canva templates.
- Automated the generation of newsletters and sending via Mailchimp, reducing manual editorial work.
4. Observability and control
- Added authentication, log fetching, monitoring, reports, and statistics so the team could:
- Inspect system behavior.
- Understand content flows.
- Monitor ranking outcomes and newsletter performance.
Results
- The Rust-based aggregation engine became faster, clearer, and easier to extend.
- The team had a reliable local dev environment and tooling for experimenting with algorithms.
- Newsletter creation became significantly more automated, with LLMs and templates handling much of the repetitive work.
- Better logging and monitoring gave stakeholders more trust and insight into the system.
Tech & responsibilities
- Role: Senior engineer for news aggregation and newsletter tooling
- Technologies: Rust, Docker, headless browser scraping, embeddings, Next.js, Google Sheets API, LLM APIs, Canva templates, Mailchimp, authentication, monitoring
- Scope: Algorithm and performance improvements, dev environment, scraping and data providers, front-end enhancements, AI-assisted newsletter workflows, observability
If you need to combine high-performance data aggregation with LLM-powered editorial workflows, I can help design and implement the backend, integrations, and tooling.
Back to all case studies
