How to Scrape Bing Search Results Without Getting Blocked

I tried to scrape Bing search results the lazy way first: one requests.get against bing.com/search from my workstation, expecting either clean HTML or a clear 403. What I got was stranger and more useful. The same endpoint returned HTTP 200 every time, but the body changed depending on the request, and adding a realistic browser User-Agent made the results disappear.

This guide is the record of that test. I ran the code in June 2026, I show you exactly what Bing returned, and I walk through the three setups that actually get data back. Along the way you get the robots.txt position, the now-dead official API, and runnable Python and PHP.

Why does Bing block scrapers?

Bing blocks scrapers on IP reputation and the full request fingerprint, and it does this without sending an obvious error code. When my request looked automated, Bing answered with HTTP 200 and a page that quietly swapped the search results for a CAPTCHA challenge. There was no 403, no 429, no retry-after. The status line looked healthy and the data was simply gone.

Here is what I measured. I hit https://www.bing.com/search?q=web+scraping three ways from the same IP, back to back:

The plain request with no User-Agent slipped through and returned ten organic results. The two requests that looked like a real browser came back smaller and empty, and the HTML contained Bing’s challenge wiring:

"verifyEndpoint": "https://www.bing.com/challenge/verify?partner=7&token=",
"captchaSuccessPostMessage": "verificationComplete"

That /challenge/verify endpoint is Bing’s anti-bot gate: the page returns 200 so a casual check passes, then JavaScript decides whether to show results or demand verification. This is the core of Bing’s anti-bot measures for search results, and it is why a status-code check is not enough; you have to inspect the body. The pattern matches how modern anti-bot systems score requests in general, and Cloudflare’s bot detection documentation describes a score built from several signals together: IP reputation, the TLS fingerprint (it compares the observed TLS handshake against what the claimed browser should send), HTTP header consistency, and a machine-learning model trained on global request behavior. The User-Agent string is one weak input among them.

One more wrinkle worth flagging. The result I got back was localized to Lithuanian (the UI string Ieškoti appeared in the markup), because Bing geolocates by IP. If you scrape from a datacenter region that does not match your target market, the SERP you parse is the wrong one before blocking even enters the picture.

What does Bing’s robots.txt say about scraping?

Bing’s robots.txt disallows the search path for every crawler. I fetched https://www.bing.com/robots.txt live in June 2026, and the catch-all block contains these lines among others:

User-agent: *
Disallow: /search
Disallow: /Search
Disallow: /results
Disallow: /images/search?

So the Bing robots.txt scraping policy is explicit: /search and /Search (both capitalizations) are off limits under the catch-all User-agent: * rule, as are the results and image-search endpoints. robots.txt is a crawling convention, and it does not carry the force of law on its own. The US legal picture for public data leans toward scrapers after hiQ Labs v. LinkedIn, where the Ninth Circuit found that accessing publicly available data is unlikely to be unauthorized access under the CFAA. Microsoft’s Services Agreement still restricts automated use, so the practical position is that robots.txt records Bing’s stated wishes, and you weigh that against your jurisdiction and use case. I go deeper on the rules in is scraping Bing legal.

Is there still an official Bing Search API?

No. Microsoft retired the standalone Bing Search APIs on August 11, 2025. The official lifecycle announcement states that “any existing instances of Bing Search APIs will be decommissioned completely, and the product will no longer be available to be used or new customer signup.” The endpoints stopped serving after that date, which removed the easy default for the Bing SERP API official vs scraping 2026 decision.

The migration path Microsoft recommends is Grounding with Bing Search inside Azure AI Foundry. That product is built to feed live web data into an LLM agent, and it is priced at $14 per 1,000 transactions for both the standard and custom search plans per Microsoft’s Grounding with Bing pricing page, with a ceiling of 150 transactions per second and 1 million per day. One detail that rules it out for SERP work: Microsoft’s own Foundry documentation states that “developers and end users don’t have access to raw content returned from Grounding with Bing Search.” You get an LLM-generated answer with citations. You do not get ten ranked organic results as JSON.

Here is the landscape as it stands in 2026:

The Grounding price has moved since launch. Reporting from ppc.land put the August 2025 launch price at $35 per 1,000 transactions, a 40% to 483% jump over the old S1 through S3 tiers that sat at $25, $15, and $6. The figure Microsoft publishes today is $14, so I quote the current pricing page and treat the launch coverage as historical context.

How do you scrape Bing search results with Python?

The naive Python approach is one requests call to bing.com/search, and you saw above that it works only by luck. Let me show you the exact code so you can recognize both states, then the parsing trap that bites people even on a successful response.

This is the request that fails quietly. It returns 200, so a status check looks fine, while the organic results are gone:

import requests
from bs4 import BeautifulSoup

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")

r = requests.get(
    "https://www.bing.com/search?q=web+scraping",
    headers={"User-Agent": UA}, timeout=25,
)
print(r.status_code)                       # -> 200 (looks fine)
soup = BeautifulSoup(r.text, "html.parser")
print(len(soup.select("li.b_algo")))       # -> 0 (results are gone)
print("challenge/verify" in r.text)        # -> True (CAPTCHA served)

The status code lies here. The real signal is the empty li.b_algo list and the presence of the challenge endpoint in the body. Any robust Bing scraper Python script has to assert on the parsed result count, because an r.status_code == 200 check passes straight through a challenge page.

When a request does get through, Bing organic results live in li.b_algo, with the title in an h2 > a. Here is the parse that ran successfully for me:

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.bing.com/search?q=web+scraping", timeout=25)
soup = BeautifulSoup(r.text, "html.parser")

for li in soup.select("li.b_algo")[:5]:
    h2 = li.find("h2")
    link = h2.find("a") if h2 else None
    if link:
        title = h2.get_text(strip=True)
        href = link["href"]
        print(title)
        print(href)   # -> https://www.bing.com/ck/a?!&&p=... (a Bing redirect; the target is hidden)

The trap is in the href. In my successful run the links were not destination URLs. Every one came back as a Bing click-tracking redirect shaped like https://www.bing.com/ck/a?!&&p=..., so the real target is hidden until you follow the redirect. The titles themselves parsed cleanly with get_text(strip=True) for me (What is Web Scraping and How to Use It? - GeeksforGeeks), but the URL unwrapping is unavoidable work: to recover the actual link you issue a follow-up request per result and read the final URL, which multiplies your request count and gives Bing more chances to challenge you. That is the kind of code for scraping Bing that looks done in a tutorial and breaks the moment you need clean URLs at volume.

So the honest summary of how to scrape Bing with raw Python: it is a few lines to start, and then a real project to keep alive across CAPTCHA challenges, redirect-wrapped links, geolocated SERPs, and IP bans.

How do you avoid getting blocked when scraping Bing?

You avoid Bing’s blocks by changing the IP reputation and the request rate. These are the levers that actually moved the result in my testing, in rough order of impact.

• Scrape from residential or private proxy IPs. Datacenter ranges are the first thing Bing’s anti-scraping measures flag. To scrape Bing with a private proxy, route each request through a clean residential or ISP IP so it presents as an ordinary home connection.
• Slow the request rate down. Bing tolerates a steady, human cadence and punishes bursts with the CAPTCHA challenge. One request every several seconds per IP is a safer starting point than parallel floods.
• Rotate IPs across queries. A single IP firing many distinct searches is an obvious pattern. Spreading queries across a pool keeps any one address below the threshold.
• Set a plausible, stable header set. A clean browser header set will not rescue a flagged IP. A contradictory one, like a mobile User-Agent paired with desktop headers, gives Bing another reason to challenge you.
• Handle the challenge state in code. Detect challenge/verify in the body and an empty b_algo list, treat that as a block, and retry the query through a fresh IP. Trusting the bare 200 will quietly fill your dataset with empty pages.

A common shortcut people reach for is to scrape DuckDuckGo instead, on the theory that DuckDuckGo scraping tolerance is higher than Google or Bing. There is some truth to it for casual use: DuckDuckGo exposes a lightweight html.duckduckgo.com endpoint that is simpler to pull than a full Bing SERP. But DuckDuckGo’s own Help Pages on result sources state that it sources its traditional web links largely from Bing, so the detour hands you Bing-derived links through a different ranking, without Bing’s own SERP features. For Bing’s ranking and Bing’s SERP features you have to solve Bing’s blocking.

The honest tradeoff is the same one I hit with every search engine. Doing all of this yourself means buying a residential proxy pool, rotating it, detecting the challenge page, solving or routing around CAPTCHAs, and following redirect links to clean URLs. That becomes a maintenance project once you pass a few thousand queries, which is why most teams hand the blocking problem to a scraper API.

How do you scrape Bing at scale without managing proxies?

A scraper API removes the blocking work by taking your Bing query and returning parsed JSON, with the proxy rotation, the CAPTCHA handling, and the geolocation handled on the server side. You send one request and get structured organic results, with no challenge page to debug and no redirect URLs to unwrap.

I use ChocoData for this. The call is a single GET with your query and an API key:

curl "https://chocodata.com/api/v1/bing/search?q=web+scraping&api_key=$CHOCO_API_KEY"

The same call from Python, which is the Python Bing scraper version I run in pipelines:

import os, requests

resp = requests.get(
    "https://chocodata.com/api/v1/bing/search",
    params={"q": "web scraping", "api_key": os.environ["CHOCO_API_KEY"]},
    timeout=30,
)
data = resp.json()
for item in data["organic_results"]:
    print(item["position"], item["title"])
    print(item["url"])          # clean destination URL, redirect already resolved

Because the anti-bot work is server side, the language on your end does not matter. Here is the same request in PHP, for anyone who needs to scrape Bing with PHP from an existing app:

<?php
$key = getenv('CHOCO_API_KEY');
$url = "https://chocodata.com/api/v1/bing/search?q=web+scraping&api_key={$key}";
$json = file_get_contents($url);
$data = json_decode($json, true);

foreach ($data['organic_results'] as $item) {
    echo $item['position'] . ' ' . $item['title'] . "\n";
    echo $item['url'] . "\n";
}

That is the whole job. No proxy pool, no challenge/verify detection, no li.b_algo parsing, and no /ck/a redirect unwrapping, because the URLs come back resolved. The API also exposes the SERP features that are painful to parse by hand, including the organic results, the related searches and questions, the knowledge graph and answer box, Bing images, and Bing ads results.

For a one-off pull of a few hundred queries, the raw Python route is fine if you have a clean IP. For continuous collection across many queries and markets, offloading the blocking and parsing is the cheaper path once you price in your own time. If you want to see how the managed options compare on coverage and price, I rank them in best Bing scrapers and APIs in 2026.

What I would do

If I needed Bing data tomorrow, I would not rebuild the proxy-and-CAPTCHA stack from scratch, and I would not wait for the retired official API to come back, because it will not. For a quick experiment I would run the raw Python parse from a residential IP and accept the intermittent challenge page. For anything ongoing I would send the query to a scraper API so my time goes to the data while the blocking is handled for me. Start with the ChocoData free signup and the single-request example above, then scale the query list once the shape of the JSON matches what you need.