Building a Government Procurement Aggregator: 15 Data Sources, Two Languages, One Search Box

Every week, somewhere in Uzbekistan, a supplier loses a contract they never saw. The tender was public — announced on an official platform, open to any bidder — but it appeared on one of fifteen different websites, written in one of two languages, with a submission window measured in days. By the time anyone on the supplier's team noticed it, the window had closed.

A client came to us with a deceptively simple brief: make that impossible. Collect every public tender in the country, translate it, classify it, and let a supplier find it with one search — then send an alert the moment the next one like it appears. This case study covers what that actually took: the anti-bot walls, the platform that publishes no product codes at all, a classification system with no common ancestor to our own, and the one HTTP response that quietly wrote 10 GB to disk and took the site down.

The Task: Every Public Tender in One Search Box

Public procurement in Uzbekistan is not one system. Purchases flow through national e-procurement portals, sectoral marketplaces, regional platforms, and the tender boards of international financial institutions — fifteen sources in total, once we counted everything our users needed. Each has its own site, its own data model, and its own publication habits. Some publish structured lots with commodity codes, budgets, and buyer details. Others post a title, a deadline, and an attached spreadsheet.

None of them offers a usable public API.

For a supplier, the daily routine looks like this: open fifteen tabs, scan each list, guess which vague titles hide relevant work, and repeat tomorrow. Submission windows are short — many tenders close within a week or two of publication — so skipping a few days can mean missing the quarter's most important contract. Larger companies assign an employee to do nothing else. Smaller ones simply miss tenders.

A detail that shaped the pipeline early: much of the substance lives in attachments. Budgets, quantities, and technical requirements often sit in spreadsheets and PDF files rather than on the page itself, so "collect" had to mean parsing documents, not just saving links.

The product had to do four things reliably:

• Collect every open tender from all fifteen sources, refreshed throughout the day
• Understand each one — normalize the format, summarize the essence, classify it against a product taxonomy
• Translate between the country's two working languages, Uzbek and Russian
• Deliver — semantic search, saved filters, and a notification the moment a matching tender appears

Two constraints shaped every technical decision that follows. Freshness: a tender surfaced a day late is worthless. Correctness: a misclassified tender is a contract nobody bids on.

Why This Is Harder Than It Looks

On paper, this is a scraping-and-search project. In practice, nearly every convenient assumption failed on contact with the real sources. Five problems consumed most of the engineering budget.

1. Some sources only talk to real browsers

Several platforms sit behind anti-bot protection that inspects far more than a User-Agent string: TLS fingerprints, JavaScript execution, browser APIs, request timing. A plain HTTP client doesn't get a clean 403 — it gets an empty shell of a page, which is worse, because nothing visibly fails.

We ended up with a three-tier access ladder, and each source runs on the cheapest tier that works for it:

The ladder matters because cost grows by an order of magnitude at every step. A browser session burns far more compute per page than an HTTP request, and an unlocker costs real money per call. Running every source through the top tier would make a two-hour refresh cycle economically absurd; running everything through the bottom tier would silently return empty pages from half the catalog.

2. The platform that publishes no product codes

One major source publishes no commodity code anywhere. Not in the listing. Not in the lot detail. Not in the attached XLSX files. There is a title written by a procurement officer, and that is all.

Category is the backbone of the whole product — search filters, alerts, and market analytics all hang off it — so "no code" was not an option. We derive the category from the title text itself: an AI classification step reads the title and whatever description can be extracted, maps the tender into the taxonomy, and records a confidence score. We validated the approach against the sources that do publish codes — classify their tenders from text alone, compare against the declared code — and tuned until the text-only path agreed often enough to trust. Low-confidence items land in a review queue instead of silently landing in the wrong category.

3. A taxonomy with no common ancestor

Our product taxonomy descends from OKPD2/KPVED — the product classifiers used across much of the CIS. Uzbekistan's official classifier, IKPU, is a 17-digit system with its own numbering logic. There is no official crosswalk between the two, and it is not a rounding problem: entire branches of one tree correspond to no single branch of the other.

We built the mapping layer ourselves: semantic similarity between category descriptions proposes candidates, curated rules override the branches where semantics mislead, and the same review queue catches the tail. The lesson worth passing on: when two classification systems have different ancestries, plan for a mapping process — with confidence scores, exceptions, and a human loop — not a one-time mapping table.

4. Two scripts, one search

Uzbek is written in Latin script; Russian in Cyrillic. The same tender may be published in either language or both, and organization names transliterate inconsistently between scripts. A search for «школьная мебель» has to find a tender titled "maktab mebeli" — and the other way around.

Keyword search alone cannot do this. Our answer was to normalize every tender into a bilingual summary and run semantic search on vector embeddings, which live in a space where the two languages overlap. Full-text search stays in the stack for exact matches — lot numbers, organization names — but meaning-level queries are answered by vectors, whichever script they arrive in.

5. The 10 GB response

And then there is the war story. One source returned a response that decompressed at roughly 1000:1. Our harvester dutifully decompressed it — about 10 GB onto the application disk — the disk filled, and the site went down. Nobody was attacking us; by every appearance it was a misconfigured export on the source's side. The effect was identical to a gzip-bomb attack anyway.

The fixes were unglamorous and permanent: streaming decompression with a hard cap on output size, per-source disk budgets, and alerts on anomalous response profiles. The principle we took away: treat every external source as hostile input — not because anyone is out to get you, but because at fifteen sources times hundreds of requests per cycle, the improbable becomes routine.

The Architecture That Handles It

The stack is deliberately boring: Node.js with NestJS, PostgreSQL as the system of record, Redis-backed job queues, and pgvector for embeddings — one database engine for both relational data and vector search, which keeps operations simple and joins cheap.

Every two hours, a pipeline runs end to end:

• Harvest — each source runs through its access tier, with per-source budgets and retries
• Normalize and deduplicate — one tender, one record, even when it appears on several platforms
• Summarize — an AI pass turns raw listings and attachments into a clean, human-readable brief
• Translate — every tender gets both an Uzbek and a Russian version
• Classify — taxonomy assignment with confidence scores
• Embed — vectors written to pgvector for semantic search
• Notify — saved searches are matched and alerts go out

Deduplication deserves a special mention. The same tender frequently appears on two or three platforms at once, sometimes with slightly different titles and attachments. Records are merged on stable identifiers where the sources provide them, and on content fingerprints where they do not, so a supplier sees one tender with every source listed — not three near-copies competing in the search results. Aggregators that skip this step do worse than annoy users: they silently double-count entire market segments in any analytics they publish.

Two decisions did the most for the economics of the AI stages.

One call, two languages. Summaries are generated in a single model call that returns structured output with both the Uzbek and the Russian version. That is half the cost of calling the model twice — and, just as valuable, the two versions cannot drift apart, because they are produced together from the same context.

Right-size the model. We benchmarked frontier-class models against Claude Haiku on our actual summarization and classification tasks. On this bounded, well-prompted workload, Haiku matched the quality we needed at roughly one-fifteenth the price. At tens of thousands of documents per month, that is the difference between an AI feature and an AI line item. The expensive models still earn their keep — on the low-confidence classification cases the cheap model flags, a fraction of a percent of the volume.

One more pipeline rule, learned the hard way: an empty harvest is an error, not a success. If a source returns zero tenders, the pipeline raises an alert instead of quietly recording "done" — because "done, found nothing" and "blocked, saw nothing" look identical on a green dashboard.

What Shipped

• 15 sources, refreshed every 2 hours
• 3,600+ open tenders in the catalog as of this writing, with several hundred added on a busy day
• 22,000+ pages in the search index — one per tender, buyer, and category
• Natural-language search on the web and in a Telegram bot — ask for school furniture in Tashkent in either language and get the actual lots
• Saved searches and alerts, market analytics, and Excel export for teams that live in spreadsheets

The product is live at UzTender — the catalog and search are open without registration, so you can judge the result directly.

What This Means for Your Project

Strip away the procurement specifics, and this project is a pattern that transfers to most industries:

• Aggregation without APIs. Real-estate listings, logistics rates, healthcare registries, financial disclosures — valuable public data usually sits scattered across sites that "don't have an API". That is not a blocker; it is an engineering line item with a known playbook and a known cost curve. The playbook scales down, too: pulling even three or four competitor price lists into one dashboard uses the same tiered-access and monitoring machinery.
• AI enrichment as a pipeline, not a demo. Summaries, translation, and classification run unattended on every record, with confidence scores, review queues, and deliberate model-cost engineering — the same discipline behind our AI development services.
• Multilingual search that matches how people actually write. Embedding-based search made two scripts and two languages a non-issue; the same approach covers any multilingual market.

If your team checks the same five websites every morning and copies rows into a spreadsheet, that workflow is a product waiting to be built. Tell us what data you need in one place — we will map the sources, the risks (yes, including the gzip bombs), and the budget it takes to ship it.