Birddog

Lead lists rot the moment you buy them. I wanted the opposite: an agent that goes out to the open web on a schedule, finds real named people who match a brief, pulls the email that actually belongs to each one, and never contacts anyone. Just discover and record, so the rest of a pipeline has something honest to work from.

So I built Birddog. It is a single-file agent that runs one iteration per invocation, reaches search and scraping through MCP, and keeps everything it finds and everything it remembers in Supabase Postgres. Here is how it works.

One iteration, and everything else lives in Postgres

Birddog is one prompt. A run does exactly one unit of work: claim a request, generate one fresh query, search, scrape, save the profiles that pass, write a memory row, exit. No inner loop. An outer scheduler re-invokes it, and every run ends by printing a single machine-readable line the scheduler parses:

Saved: 14, Status: in_progress, Error: none

Because the agent holds no state between runs, all of it sits in three Postgres tables:

• lead_requests is the queue of briefs, each a plain-text query with a status and a running count.
• profiles is the output: one row per named person, with their email, location, headline, and a JSONB blob of everything extracted.
• pg_memory is the agent's memory, one append-only row per iteration.

A run can crash and lose at most that one iteration. The queue, the results, and the memory survive in the database.

Two MCP servers do the reaching

Birddog has no filesystem writes and no bespoke API clients. Everything it touches, it touches through MCP.

                    ┌───────────── Brightdata MCP
   Birddog ─────────┤   search_engine   → SERP pages
   (one prompt)     │   scrape_batch    → page markdown
                    │
                    └───────────── Supabase MCP
                        list_tables    → schema check
                        apply_migration→ first-run bootstrap
                        execute_sql    → claim, save, remember

The Brightdata server is the reach into the web: search_engine returns SERP pages, which the agent pages through, and scrape_batch pulls ten URLs at a time back as markdown. The Supabase server is the reach into state: execute_sql runs every claim, save, and memory write against Postgres. Swapping either capability is swapping an MCP server, not rewriting the agent.

The database provisions itself

An agent should not assume someone ran a migration first. So step one of every run is list_tables, and if lead_requests, profiles, or pg_memory is missing, the agent applies the DDL itself through apply_migration. Point a fresh Supabase project id at Birddog and the first run builds its own schema. Onboarding a new backend is one line of config.

Claiming work without two agents colliding

Because the scheduler can fire more than one Birddog at once, claiming a request has to be atomic. Postgres does the locking:

with c as (
  select id from lead_requests
  where status in ('pending','in_progress') and count < 200
  order by priority desc, status desc, updated_at asc
  limit 1 for update skip locked
)
update lead_requests set status='in_progress', updated_at=now()
from c where lead_requests.id = c.id returning lead_requests.*;

for update skip locked is the whole trick: two concurrent runs never grab the same request, and neither one blocks waiting on the other. Zero rows back means there is nothing to do, and the agent exits idle.

What counts as a lead, and what gets thrown out

A lead is a strict thing: one named human, first and last, with an email that provably belongs to that person and matches the brief. Most of the code is the filter that enforces it.

Before scraping, obvious dead ends are dropped: job boards that list postings instead of people, and paywalled aggregators like ZoomInfo or Apollo where the email is masked behind a login. After scraping, each page is read for a named person tied to a real email, and a mailbox is rejected if it is masked (j****@), generic (info@, sales@), or wrapped by Cloudflare's email protection. A generic mailbox is only accepted when the page names exactly one owner and lists no one else, which is how a solo founder's hello@ legitimately becomes theirs.

A single scraped page can yield several people, and each save is atomic, incrementing the request's count only when the insert actually lands:

with ins as (
  insert into profiles (source_url, request_id, search_query,
                        name, email, location, headline, structured_data)
  values (...)
  on conflict do nothing returning id
)
update lead_requests set count = count + 1
where id = '<rid>' and exists (select 1 from ins) returning count;

Unique constraints on both source_url and email mean on conflict do nothing silently absorbs duplicates, and the count only moves on a genuine new row.

Memory that makes the next run smarter

The interesting part of a stateless agent is how it remembers. At the start of a run Birddog reads its own last three pg_memory rows, newest first. The newest is the authoritative handoff from the previous iteration; the older two are context. At the end it appends one fresh snapshot built from a fixed rubric: what it ran this iteration, what it observed, any pattern worth noting, one focused handoff for next time, and a carry-forward list of every query it has already tried.

That last list is the load-bearing invariant. It is the global dedup ledger, and it is append-only across every request, so the agent never burns an iteration re-running a search that already came up dry. Rows are never updated, only inserted, so the full trail stays reproducible and the running list has to be merged forward into each new snapshot rather than patched in place.

The same memory drives two stop conditions. If the last three iterations on a request all saved zero, the angle space is exhausted in practice and the request is marked complete. If the running count hits its target of 200, same result. The agent knows when to quit because the memory tells it.

When this is the wrong tool

Birddog drives real search and scraping infrastructure, so it is shaped for one careful request at a time, not for hammering a single query in a loop. It also assumes a database is in the loop; the whole design leans on Postgres as the memory and the queue that outlive each stateless run. Take that away and there is nowhere to claim work from and nothing to hand off to.

But for the narrow job of turning a plain-text brief into a growing, deduplicated list of real people you could actually reach, the single-iteration, MCP-driven, Postgres-backed shape has held up well. The agent is one prompt; the durability is all in Supabase.