Is there a way to better streamline the following PyTrends/Pandas web-scraper code into a python function?

  dataframe, excel, pandas, python

Is it possible to change the following code into a function so that the inputs can be changed automatically?

Where in the code the lines KEYWORDS=[...] and DATE_INTERVAL='...' and COUNTRY=[...] and the second parameter of the line: df_trends.columns=['date','Unemployment (Denmark)'] can be defined by the user? Where a new dataframe is generated at the end of the function?

import pandas as pd
import pytrends
from pytrends.request import TrendReq
pytrend = TrendReq()

KEYWORDS=['Arbejdsløshed'] 
KEYWORDS_CODES=[pytrend.suggestions(keyword=i)[0] for i in KEYWORDS] 
df_CODES= pd.DataFrame(KEYWORDS_CODES)


EXACT_KEYWORDS=df_CODES['mid'].to_list()
DATE_INTERVAL='2004-01-04 2009-01-04'
COUNTRY=["DK"]
CATEGORY=0
SEARCH_TYPE=''



Individual_EXACT_KEYWORD = list(zip(*[iter(EXACT_KEYWORDS)]*1))
Individual_EXACT_KEYWORD = [list(x) for x in Individual_EXACT_KEYWORD]
dicti = {}
i = 1

for Country in COUNTRY:
    for keyword in Individual_EXACT_KEYWORD:
        pytrend.build_payload(kw_list=keyword, 
                              timeframe = DATE_INTERVAL, 
                              geo = Country, 
                              cat=CATEGORY,
                              gprop=SEARCH_TYPE) 
        dicti[i] = pytrend.interest_over_time()
        i+=1
        
    
df_trends = pd.concat(dicti, axis=1)


df_trends.columns = df_trends.columns.droplevel(0) #drop outside header
df_trends = df_trends.drop('isPartial', axis = 1) #drop "isPartial"
df_trends.reset_index(level=0,inplace=True) #reset_index
df_trends.columns=['date','Unemployment (Denmark)'] #change column names



df_trends[""] = "XXXX" # Creates a blank column as blank-space between columns

Thank you

Source: Python Questions

LEAVE A COMMENT