Skip to main content

Scrape Search Result Data using SerpApi with Python

Updated on: webdamn PHP, Resources & Reviews, Reviews

Search Engine Results Pages known as SERP data is the search result pages served by Search Engines. The search results returned by Search Engines relevant to search keywords.

The search Engines allows manual keyword searching but sometimes we need search result of multiple keywords from thousands of web pages. For this, we develop algorithm and systems to get SERP data. But it’s always challenge to get accurate SERP data because the search engines constantly keeps on changing their SERP structure and algorithm.

So if you’re looking for solution to get accurate SERP data, then you’re here at the right place. In this post, you will learn how to integrate SERP API to get the real time SERP data.

The SerpApi is a secure real time Google Search API that get Google results from anywhere in the world with their location parameter and returns response data into JSON data.

Also, read:

So let’s start integrating the SerpApi with Python:

Step1: Get API Secret Key

We need to create an account on SerpApi to get the Secret key to access the API. We will use the secret key when we will make API request.

https://serpapi.com/search?api_key = API_SECRET_KEY

Step2: Get Google Search Results Python Module

We will install the Google search result Python module to access the SerpAPi.

pip install google-search-results

After installation, we will use module in our script.

from serpapi.google_search_results import GoogleSearchResults

Step3: Build Search Query with Params

We will create search query with parameters to get the search result as per our need. We will pass location, engine and also api_key to access the API.

searchParams = {
    "engine": "google",
    "q": "Coffee",
    "location": "Austin, Texas, United States",
    "google_domain": "google.com",
    "gl": "us",
    "hl": "en",
    "api_key": "API_SECRET_KEY"
}

Step4: Get Result Data

We will pass the search param query to the GoogleSearchResults() and get the response result data.

client = GoogleSearchResults(searchParams)
results = client.get_dict()

Step5: Complete Code

Here is the complete code to get make SerpApi HTTP GET request with search parameters and get the result data.

from serpapi.google_search_results import GoogleSearchResults

searchParams = {
    "engine": "google",
    "q": "Coffee",
    "location": "Austin, Texas, United States",
    "google_domain": "google.com",
    "gl": "us",
    "hl": "en",
    "api_key": "API_SECRET_KEY"
}

client = GoogleSearchResults(searchParams)
results = client.get_dict()

Following is sample response JSON data after running above example code.

{
  "search_metadata": {
    "id": "5cc470041befcd91e24b126a",
    "status": "Success",
    "created_at": "2019-04-27 15:06:44 UTC",
    "processed_at": "2019-04-27 15:06:44 UTC",
    "google_url": "https://www.google.com/search?q=Coffee&oq=Coffee&uule=w+CAIQICIaQXVzdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM&hl=en&gl=us&sourceid=chrome&ie=UTF-8",
    "total_time_taken": 2.59
  },
  "search_parameters": {
    "q": "Coffee",
    "location_requested": "Austin, Texas, United States",
    "location_used": "Austin,Texas,United States",
    "google_domain": "google.com",
    "hl": "en",
    "gl": "us",
    "device": "desktop"
  },
  "search_information": {
    "organic_results_state": "Results for exact spelling",
    "total_results": 3240000000,
    "time_taken_displayed": 0.54,
    "query_displayed": "Coffee"
  },
  "local_map": {
    "link": "https://www.google.com/search?gl=us&hl=en&q=Coffee&npsic=0&rflfq=1&rldoc=1&rlha=0&rllag=30267374,-97742553,138&tbm=lcl&sa=X&ved=2ahUKEwijjeaarNrlAhUPKqwKHc8DAh8QtgN6BAgLEAQ",
    "image": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
    "gps_coordinates": {
      "latitude": 30.267374,
      "longitude": -97.742553,
      "altitude": 138
    }
  },
  "local_results": {
    "more_locations_link": "https://www.google.com/search?gl=us&hl=en&q=Coffee&npsic=0&rflfq=1&rldoc=1&rlha=0&rllag=30267374,-97742553,138&tbm=lcl&sa=X&ved=2ahUKEwijjeaarNrlAhUPKqwKHc8DAh8QjGp6BAgLEEc",
    "places": [
      {
        "position": 1,
        "title": "Houndstooth Coffee",
        "place_id": "11265938073076301333",
        "lsig": "AB86z5Vdw6C2pJpM0xQ6JUx2KONU",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location_requested=Austin%2C+Texas%2C+United+States&lsig=AB86z5Vdw6C2pJpM0xQ6JUx2KONU&ludocid=11265938073076301333&q=Coffee&tbm=lcl",
        "rating": 4.6,
        "reviews": 625,
        "price": "$$",
        "type": "Coffee shop",
        "hours": "Closed ⋅ Opens 6:30AM",
        "address": "401 Congress Ave #100c",
        "description": "Cozy hangout for carefully sourced brews",
        "thumbnail": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
        "gps_coordinates": {
          "latitude": 30.2661803,
          "longitude": -97.74290409999999
        }
      },
      {
        "position": 2,
        "title": "The Hideout Coffee House",
        "place_id": "15498522356495312950",
        "lsig": "AB86z5WSxdnDKVF_iLXNN6Lg0UQ5",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location_requested=Austin%2C+Texas%2C+United+States&lsig=AB86z5WSxdnDKVF_iLXNN6Lg0UQ5&ludocid=15498522356495312950&q=Coffee&tbm=lcl",
        "rating": 4.4,
        "reviews": 296,
        "price": "$",
        "type": "Coffee shop",
        "hours": "Closed ⋅ Opens 7AM",
        "address": "617 Congress Ave",
        "description": "Area institution with a theater upstairs",
        "thumbnail": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
        "gps_coordinates": {
          "latitude": 30.268568999999996,
          "longitude": -97.74215099999999
        }
      },
      {
        "position": 3,
        "title": "Starbucks",
        "place_id": "10605736027611436825",
        "lsig": "AB86z5XTJ_Io_anVBu2fU6Zaqu3b",
        "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location_requested=Austin%2C+Texas%2C+United+States&lsig=AB86z5XTJ_Io_anVBu2fU6Zaqu3b&ludocid=10605736027611436825&q=Coffee&tbm=lcl",
        "rating": 4.1,
        "reviews": 404,
        "price": "$$",
        "type": "Coffee shop",
        "hours": "Closed ⋅ Opens 6AM",
        "address": "600 Congress Ave Ste. G-270",
        "description": "Iconic Seattle-based coffeehouse chain",
        "thumbnail": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
        "gps_coordinates": {
          "latitude": 30.268259,
          "longitude": -97.74295599999999
        }
      }
    ]
  },
  "related_questions": [
    {
      "question": "Who first drank coffee?",
      "snippet": "The story of Kaldi, the 9th-century Ethiopian goatherd who discovered coffee when he noticed how excited his goats became after eating the beans from a coffee plant, did not appear in writing until 1671 and is probably apocryphal.",
      "title": "Coffee - Wikipedia",
      "link": "https://en.m.wikipedia.org/wiki/Coffee",
      "displayed_link": "https://en.m.wikipedia.org/wiki/Coffee"
    },
    {
      "question": "What is coffee used for?",
      "snippet": "Coffee is a drink made from coffee beans, which are the roasted fruit of the Coffea arabica bush. People most commonly drink coffee to relieve mental and physical fatigue and to increase mental alertness. Coffee is also used to prevent Parkinson's disease, Alzheimer's disease, dementia, and cognitive decline.",
      "title": "Coffee: Uses, Side Effects, Interactions, Dosage, and Warning",
      "link": "https://www.webmd.com/vitamins/ai/ingredientmono-980/coffee",
      "displayed_link": "https://www.webmd.com/vitamins/ai/ingredientmono-980/coffee"
    },
    {
      "question": "Which type of coffee is the best?",
      "snippet": "Types Of CoffeeMacchiato. More like an espresso, the only difference is presence of a bit of steamed milk and foam. ... Americano. A much loved breakfast drink, Americano are usually taken for breakfast. ... Latte. A bit towards the milder side, Lattes have more milk compared to cappuccinos. ... Piccolo Latte. ... Cappuccino. ... Flat white. ... Mocha. ... Affogato.Sep 19, 2017",
      "title": "What Am I Ordering – Different Types Of Coffee Drinks And Their Taste",
      "link": "https://homekitchenary.com/different-types-of-coffee-drinks-and-their-taste/",
      "displayed_link": "https://homekitchenary.com/different-types-of-coffee-drinks-and-their-taste/"
    },
    {
      "question": "Which country introduced coffee to Taipei?",
      "snippet": "While tea had a long and classical upbringing in China that was naturally transplanted to Taiwan, it was the Dutch who introduced coffee – not long after the brew had spread from its original homeland of Ethiopia in the 16th century.Jan 15, 2015",
      "title": "Taipei Coffee Culture: Rich, Robust, and Satisfying - Taiwan ...",
      "link": "https://topics.amcham.com.tw/2015/01/taipei-coffee-culture-rich-robust-and-satisfying/",
      "displayed_link": "https://topics.amcham.com.tw/2015/01/taipei-coffee-culture-rich-robust-and-satisfying/"
    }
  ] 
  }
}

Step6: Conclusion

In this tutorial you have learned how to integrate SerpApi with Python to scrape the Google search result data. You can enhance the search by using more advance options by going through documentation within your script.

You may also like: