From competitor ad monitoring idea to paying customers

Testing demand for an ad monitoring product

A department head at an advertising agency I worked with proposed an ad intelligence service: a tool for browsing competitors' ads and analyzing them. He wanted to find out whether the agency could sell access to its own product.

There were already many similar tools on the market. We needed to establish which data we could collect ourselves and whether customers would want the service. The department head handled sales through his professional network. I was responsible for building a product he could take to customers.

Data sources and development responsibilities

I started by mapping the available data sources. Together, we decided which ads to track and which specialists we would need to collect them.

Mobile developers built software that collected ads on devices and sent them to my backend. I wrote web ad scrapers and platform API integrations, collecting social ads both through APIs and by scraping actual pages. I also built the backend that received data from the mobile device farm.

All sources fed into a shared processing pipeline. The backend removed duplicates, classified ads, and compressed the data. Users received processed material organized by category.

Physical devices and the initial budget

The first constraint was coverage. Collecting ads at scale required mobile device farms and substantial operating costs. The client was not prepared to make that investment just to test the idea.

We experimented with emulators, but major platforms' SDKs recognized them. Ads either did not appear or differed from those shown on real devices. Physical devices were therefore necessary, and the budget allowed us to start with a small farm of just a few devices.

I designed collection around those limits. Rather than trying to cover every available ad, we focused on a few niches and sources. That was the scope of the data available when the product reached its first buyers.

The web service and paid access

I built the web service with an admin panel, several pricing plans, and multiple payment methods. Users could register, try the service for free, or pay for a plan immediately. They could browse ads and analytics for the sources, categories, and market segments covered by our collection.

Agency staff used the service first. The department head then began selling access to external customers. This answered the question of willingness to pay: the product had real paying users. It did not yet establish whether further scaling would make economic sense.

Adapting the architecture as data grew

The next challenge emerged as ads accumulated. Data volume grew nonlinearly, and every few months the system reached a new level of load. Architecture designed for the previous scale began producing excessive delays.

These were not simple list-filtering queries. Users applied custom selection criteria, grouped results across multiple levels, and requested daily impression statistics for selected groups. The system had to combine these operations over a continually growing dataset.

I substantially reworked storage and result retrieval several times. I used several database engines for different tasks and implemented automatically prewarmed caching for core and frequently used queries.

Each revision significantly increased the system's capacity. Further architectural changes became necessary at the next level of data volume, rather than because the same problem remained unresolved. The goal was to maintain acceptable response times as queries became more complex and the database grew.

The decision to keep the service internal

After evaluating the economics of scaling, the company decided to keep the service for internal work. I handed the system over to the employees who would maintain it.

By then, agency staff were using the product and external customers had paid for access. The company could base its decision on a working service and actual sales experience, having started with a small device farm rather than a major investment in collection infrastructure.

I worked with the same department head on another product validation project: a news app with its own advertising platform. For other areas of my work, see development for advertising and performance marketing.