I’m trying to scrape two fields product_title and item_code from this webpage using requests module. When I execute the script below, I always get AttributeError in place of the result as the data I’m after are not in page source. However, I’ve come across several solutions in here which are able to fetch data from ..
for my computing project I am attempting at making a financial forecasting website. one of the elements in the code is a web scraping api that scraps data from the income statement of a company on yahoo finance. however even though the url is correct I still keep on getting a 404 error and was ..
I am attempting to extract bid information from this site. I am a Scrapy newbie, and bit stuck as to why I don’t getting any output, instead, I get Crawled (200)…(referer: None) and no output. I am unable to figure out what I am missing or need to change. I really don’t know where the ..
I have been trying to scrape data from investing.com and get Volume and Net Volume data for some bonds from the technical analysis charts (Example: https://ng.investing.com/rates-bonds/us912810rk60-streaming-chart) Technical analysis chart showing volume and net volume I attempted this using python requests module but got stuck when retrieving data from the graph itself. Source: Python..
there. I am building a SCRAPY SPIDER where I send requests to an API. I need to check if a condition is met, I need to break the loop. I have the loop in the parse method and content and parse_api method. I tried to use the following logic but it didn’t work. if ‘date ..
This is my web scrap target site. https://www.aliexpress.com/wholesale?catId=0&SearchText=ipad&SortType=default&g=n&page=1 With this code, I can get 60 items. import time from selenium import webdriver from time import sleep options = webdriver.ChromeOptions() options.add_argument(‘headless’) options.add_experimental_option(‘excludeSwitches’, [‘enable-logging’]) options.add_argument(‘–lang=en’) driver = webdriver.Chrome(r’c:chromedriverchromedriver.exe’, options=options) options.add_argument( "user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36") url = ‘https://www.aliexpress.com/wholesale?catId=0&SearchText=ipad&SortType=default&g=n&page=1’ driver.get(url) ..
I am on this page on Tokyo Olympic Website I would like to get all the elements that has class name beginning with a specific string. For instance I want to get all elements that begin with ‘col-sm-‘. html.r.find(‘.col-sm-6’) gives me all elements that have class name col-sm-6. However I would like to get all ..
I am getting the output  after this from bs4 import BeautifulSoup import requests html_text = requests.get(‘https://www.naukri.com/python-jobs?k=python’).text soup = BeautifulSoup(html_text, ‘lxml’) jobs = soup.find_all(‘div’, class_ = ‘jobTupleHeader’) print(jobs) Source: Python-3x..
So I have this item whose url is this I was checking out the Network Header in Inspect mode and found that when I click on the second page of reviews there was an API address. I am trying to only scrape reviews "With Comments" import requests import pandas as pd import csv rows = ..
I’m trying to scrape all monetary policy reports on this ECB website here using python’s Selenium package. Below is my code: from selenium import webdriver CHROME_PATH = <INSERT_CHROME_PATH_HERE> url = "https://www.ecb.europa.eu/press/govcdec/mopo/html/index.en.html" xpath = """//*[@id=’snippet*’]/dd/div/span/a | # xpath of monetary policy report links //*[@id=’snippet1′]/dd/div/span/a | //*[@id=’snippet2′]/dd/div/span/a | //*[@id=’snippet3′]/dd/div/span/a | //*[@id=’snippet4′]/dd/div/span/a | //*[@id=’snippet5′]/dd/div/span/a | //*[@id=’snippet6′]/dd/div/span/a | //*[@id=’snippet7′]/dd/div/span/a ..