Python, Selenium and Chrome – How do can I detect the end of a page with dynamically generated content?

  python, selenium, web-scraping

I have gone through existing questions and google results of a similar nature, every solution posed has not worked for me within the particular website I am currently scraping.

https://dutchie.com/embedded-menu/revolutionary-clinics-somerville/menu

I am sending page down keys to the body element, which loads each item to be scraped. I have two issues with this, first I am unable to detect when the scrolling has stopped. Second, I have to manually click the browser window as it opens to allow the keys to be sent. I am not sure how to mimic this same focus giving behavior via code.

elem = driver.find_element_by_tag_name("body")
elem.send_keys(Keys.PAGE_DOWN)

I have tried the following, in many different iterations and the number printed never charged regardless of how far down the page I am or if I used innerHeight, or body instead of documentElement.

height = driver.execute_script("return document.documentElement.scrollHeight")

If I attempt to scroll down the page using a similar approach, this page does not move.

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

I am unsure if this has to do with iframes or if I am simply misunderstanding the best approach.

Thank you!

Source: Python Questions

LEAVE A COMMENT