TL;DR Just give me the code

Here is all you need to scrape eBay search results with GridPanel:

import requests
from bs4 import BeautifulSoup

def parse_page(html):
    result = []
    soup = BeautifulSoup(html, 'html.parser')
    items = soup.find_all('li', attrs={'class': 's-item'})
    for item in items:
        link = item.find("a", href=True)
        href = link['href']
        result.append(href)
    
    return result

# Setup our api key
api_key = <YOUR_KEY>
url = f"https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=nike+air+force+1&_sacat=0&rt=nc&LH_BIN=1"

# Setup the params for the GridPanel API
params = {
    'api_key': api_key,
    'url': url
}

# Scrape!
response = requests.get("https://gridpanel.net/api/scrape", params=params)
body = response.json().get("result", {}).get("body")
listing_urls = parse_page(body)

How to scrape eBay search results with the GridPanel Scraping API

In this article, we'll demonstrate the process of extracting eBay search results with the user-friendly GridPanel Scraping API. Our comprehensive guide will encompass all aspects of scraping, including the scraping API, setup, proper usage, and result processing. While our examples today utilise Python, it's important to note that this tutorial is language-agnostic and can be effortlessly adapted to your preferred programming language.

What is scraping and why do we need an API?

Before diving into the details, let's clarify what web scraping is and why the GridPanel Scraping API is valuable for this purpose. Web scraping is the process of extracting data from websites and web pages by retrieving relevant information from HTML. This automation is used to gather substantial amounts of data from target websites, requiring stealth to avoid potential blocks that can render your scraper ineffective.

Numerous strategies exist for maintaining a low profile while conducting web scraping activities. However, the GridPanel Scraping API stands out by encapsulating an array of highly effective techniques within a convenient API framework, empowering you to employ these methods seamlessly when scraping your intended websites. What sets this API apart is its utilisation of authentic devices, each associated with genuine device IP addresses. Consequently, when you make requests to scrape websites, you are doing so with the utmost authenticity—no need for any spoofing or deceptive maneuvers. This authenticity ensures that your scraping endeavors maintain their credibility and are free from any suspicion or interference. You can read more about our scraping API to see how it helps us scrape eBay.

What are we scraping on eBay?

This post is focused on scraping search results from eBay, in particular extracting the links associated with each search result, however with the code we provide and our Scraping API you can really scrape anything you like from eBay with a few small changes. We will cover a few of those below.

To give an example, we are going to focus on gathering all the buy-it-now listings for Nike Air Force 1 trainers, which you can view at this URL: 

"https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=nike+air+force+1&_sacat=0&rt=nc&LH_BIN=1"

This will result in the following page:

eBay search results of Nike Air Force 1 Trainers

As you can see there are a number of listing returns, each listing links to another eBay page which contains information about the listing itself. Let's say we wanted to scrape all of the eBay listing URLs so we could then extract further information about each listing itself. This is what we will achieve in this post.

How to scrape eBay search results

Now you know about scraping, and what we are aiming to scrape, we can move on to how to use the GridPanel Scraping API to achieve this.

Get your API key

Firstly you need to make sure that you set setup with the GridPanel Scraping API, you can do that by signing up for a free account. Once you have an account, you will be given 1000 credits and can access your API credentials here: gridpanel.net/dashboard/scraping. You will also find a handy request builder and information about your API usage.

The API request

Scraping the search results from eBay really is as simple as making a single API request through the GridPanel API. The following request will load the correct eBay listings page, ensure that the page and listings have properly loaded, and grab the HTML itself.

import requests

# Setup our api key
api_key = <YOUR_KEY>
url = f"https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=nike+air+force+1&_sacat=0&rt=nc&LH_BIN=1"

# Setup the params for the GridPanel API
params = {
    'api_key': api_key,
    'url': url
}

# Scrape!
response = requests.get("https://gridpanel.net/api/scrape", params=params)

Processing the results

If you run the above code you will receive a result object which contains the HTML body of the eBay listings page for the trainers. To process each one we need to utlise Python's beautifulsoup library. This library will allow us to parse and search the HTML body itself. You can see in the code below how we parse the HTML and extract all of the listing URLs for further processing.

import requests
from bs4 import BeautifulSoup

def parse_page(html):
    result = []
    soup = BeautifulSoup(html, 'html.parser')
    items = soup.find_all('li', attrs={'class': 's-item'})
    for item in items:
        link = item.find("a", href=True)
        href = link['href']
        result.append(href)
    
    return result

# Setup our api key
api_key = <YOUR_KEY>
url = f"https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=nike+air+force+1&_sacat=0&rt=nc&LH_BIN=1"

# Setup the params for the GridPanel API
params = {
    'api_key': api_key,
    'url': url
}

# Scrape!
response = requests.get("https://gridpanel.net/api/scrape", params=params)
body = response.json().get("result", {}).get("body")
listing_urls = parse_page(body)

If you run the above code, you will have an array (listing_urls) which is populated with all the eBay listing links to the trainers from the search results page. This will look like the following:

['https://ebay.com/itm/123456?hash=item28caef0a3a:g:E3kAAOSwlGJiMikD&amdata=enc%3AAQAHAAAAsJoWXGf0hxNZspTmhb8%2FTJCCurAWCHuXJ2Xi3S9cwXL6BX04zSEiVaDMCvsUbApftgXEAHGJU1ZGugZO%2FnW1U7Gb6vgoL%2BmXlqCbLkwoZfF3AUAK8YvJ5B4%2BnhFA7ID4dxpYs4jjExEnN5SR2g1mQe7QtLkmGt%2FZ%2FbH2W62cXPuKbf550ExbnBPO2QJyZTXYCuw5KVkMdFMDuoB4p3FwJKcSPzez5kyQyVjyiIq6PB2q%7Ctkp%3ABlBMULq7kqyXYA',
 'https://www.ebay.co.uk/itm/134711644281?hash=item1f5d709079%3Ag%3AeNUAAOSw7J9k8hQd&amdata=enc%3AAQAIAAAAwK%2B%2B%2BAurbxQ2yNf36Xto0LNSqH9Pdeg6B4P%2FIajH%2BLPSt09xoc8%2BKIX%2FpGjnkQkTus0ZcCSrYFfI7I2XCfI%2BOGZxvjNLZFmJnVo2aMsdGt%2FCxwJc01aHuZPBLYZH9peIS3IO2CA7V%2BYVZTQFctlDh4L4UxXXB%2By75NXmadvadCFdiDxVyvZIcIi8uNUojNdE6eljDS60RTWfjeQSpJk42%2BzYK1FJoUuS%2Bqju0ZoFxckfbuceftTyhRWUJd86broTew%3D%3D%7Ctkp%3ABlBMUPSC9oLXYg&LH_BIN=1',
 'https://www.ebay.co.uk/itm/165807313923?hash=item269ae2d403%3Ag%3AxG8AAOSwqzdjh4vd&amdata=enc%3AAQAIAAAAwL0KoIYCZufGNnJtmdAKQxvqvqjJC3ObKXMskpLzEUr9E2Y73rppS8zXlcjI4OmhX5qq4Sef6VmrA9qA1%2B2NeUVhyo6FkRMS2UQ2dK1tY4siryoVx2VsuNEj718w9CvKD9IhwzaMriTOw0xlyppb%2FJd%2BtyeNzZiHHf4vlbgmfVxxpHCSvqj9LU1EeeG%2FqgSKCxCMNtsh7gk21FhbBQgyTwi5KEBJgTdW9l2Yx3rMWnnTDnIuZi6U7A0f2cgluy7UZg%3D%3D%7Ctkp%3ABlBMUPSC9oLXYg&LH_BIN=1',
 'https://www.ebay.co.uk/itm/355043294609?epid=11053225568&hash=item52aa3abd91%3Ag%3ANfYAAOSw9pRkz5yW&amdata=enc%3AAQAIAAAA4P1RkQ8HmpVcuajNiFHwCwmN%2BLKd7vebQ07yFGKaIKc6Dd4rcQeOzQ7CBc7kpfixrbVUc5nMWz19aS3Ofv6rrjQiWX87BfO6aX5128f7vufzDOFyPYV6D4YwbXrE3TTEHDYoxi6XpTuyrDASalJ7Jcnysi6%2FRfoBZ9Xg7o5Z0M7GJ%2FnNIkP%2FQgNqanRvNPh88b132z8t3srQgK4kr6Vn5iPywr%2BikoUtJ4Zxh%2FAD0cP%2FBNlbJsdVKVSU%2BCpTa0jiusdoV5xeB772tuug1XpAdAo%2B5dOG8bOGYdxa6XChLtUw%7Ctkp%3ABFBM9IL2gtdi&LH_BIN=1', ...]

From here, you can really do what you like, you could then crawl each listing itself, again using the GridPanel Scraping API, to extract further information regarding the actual listing.

Extensions

You can now see how simple it is to not only scrape eBay search but to utilise the GridPanel Scraping API to do so. It would be trivial to extend this scraper to extract much more information about the listing themselves, for example extracting:

  • Imagery
  • Text
  • Pricing
  • Auction details
  • Seller information
  • Etc!

Conclusion

In this blog post, we've shown you how to utilse the GridPanel Scraping API to successfully scrape eBay listings and extract all the listing URLs for a search term. If you are looking for more information on our API or on scraping other sites stay tuned as we have plenty more examples and scraping posts to come!

Sign up for an account and join our grid today to keep you ahead of your competitors.