Competitor · national chain
Drogaraia across 14 states
Full catalog with real regional prices by ZIP code, EAN, brand, active ingredient and dosage — via network traffic interception and reverse GraphQL.
Every competitor’s price, region by region, updated on its own.
I designed and built the Discovery Stack — the platform that tracks, on its own and every two weeks, how much pharmacy retail charges for each product in Brazil. It brings together the prices of competitor chains and public government databases, organizes everything in one place and delivers it ready for Masterfarma to compare price by product, region and competitor.
Role · Infrastructure architecture, data engineering, orchestration, API reverse engineering and resilient scraping
Scope · End-to-end project · in production
In short, for decision-makers
The problem
Masterfarma needed to know how much competitors charge for each medicine, in each region — but those prices were scattered across several sites and government databases, each in its own format, impossible to track by hand.
What LNData did
We built a platform that collects those prices on its own, every two weeks, brings everything into one place and delivers it ready to compare by product, region and competitor.
The result
The chain now sees the real price charged across 14 states in a single dashboard and sets pricing and purchasing based on up-to-date numbers — with no manual spreadsheets and no one having to type anything.
4
data sources monitored
14
states with real regional prices
~80k
EANs per 30-day window
biweekly
automated cadence
5
pipelines orchestrated in cascade
~869
categories mapped across sources
64
scraping tasks in parallel per run
millions
of price observations in the data warehouse
Architecture
The platform collects from multiple sources with distinct techniques, normalizes and conforms everything into an OLAP-oriented columnar data warehouse and delivers the result as price comparison across competitors, government sources and regions.
Data sources
- Drogaraia · 14 states · reverse GraphQL
- Preço Popular · VTEX Intelligent Search
- Nota Paraná · gov. API
- Menor Preço Brasil · authenticated SVRS-SC
Data platform
Airflow orchestration · CeleryExecutor
ClickHouse data warehouse (OLAP)
Resilient scraping · containerized · nginx TLS
Consumption
- BI dashboards · Superset
- Comparison by EAN, region and competitor
- Unified model · chains + gov. sources
Monitored sources
Two competitor chains and two government lowest-price databases — each requiring its own collection technique, all converging into the same data model.
Competitor · VTEX platform
Preço Popular via Intelligent Search
Reverse engineering of VTEX's search API, with sort fallback and dynamic category discovery, stored in the same model as Drogaraia.
Government database · PR
Lowest price via e-invoice (Paraná)
Public Menor Preço / Nota Paraná API queried by GTIN and region, seeded from the already-collected catalog — the pipeline feeds itself.
Government database · SC
Authenticated lowest price (Santa Catarina)
SVRS / gov.br API with a Bearer token, search by latitude/longitude and radius, and an automated alert when the credential nears expiration.
Applied engineering
The technical decisions that make collection accurate and resilient — not a proof of concept, but a platform that runs on its own in production.
Network interception, not DOM scraping
Instead of scraping fragile HTML, collection listens to network responses and extracts data straight from the application’s own JSON (__NEXT_DATA__, GraphQL) — robust to layout changes.
ZIP-code geolocated sessions
Automation opens and confirms each state’s ZIP code and persists the browser session, capturing the correct regional price without repeating the handshake for every category.
Deterministic parallelism
Dynamic Task Mapping and hash-based chunking split ~80k EANs into stable slices, processed in parallel, with Pool rate limiting so the source is not overloaded.
Production · Resilience & operation
Built to run without a babysitter
Scraping in production fails all the time — sites change, APIs throttle, sessions drop. The platform treats failure as the expected case: it isolates what broke, recovers on its own and leaves a trail of what happened.
Three-layer retry
Task retry, per-page retry with progressive strategies and a session gate that gracefully skips an entire region when it fails — without bringing down the run.
Real rate limiting
Airflow Pools cap concurrency against each target site independently of cluster parallelism, keeping collection within what the source tolerates.
Actionable dry-run mode
An Airflow Variable switches on a test mode (1 category / 1 region / 1 page) to validate changes directly in production, safely and without triggering downstream.
Failure observability
Every failure is written to a structured error table — company, location, path, error — instead of just breaking the task, with throttling diagnostics by type.
Airflow PoolsTaskGroupsDynamic Task MappingVariablescron
Impact
Competitor and government prices, once scattered and invisible, now live in a single analytical source — comparable by EAN.
Real regional pricing: the platform sees the price charged in each of the 14 states, not a generic national price.
Unified data model — competitor chains and gov. databases in the same table — enables direct comparison with fast queries over millions of rows.
Autonomous, auditable collection: chained pipelines run on their own twice a month, with an error trail and credential alerts, requiring minimal intervention.
Platform stack
PythonAirflow (Celery)ClickHousePostgreSQLRedisPlaywrightDockernginx
Need to monitor prices or collect data on the web?
From API reverse engineering to resilient scraping in production, with a data warehouse and BI at the end — tell us what you need to track and we’ll come back with scope, timeline and deliverables.