Web scraping with beautiful soup, entering all links and getting information

I am trying to open each company from StackOverflow companies and get specific informations(such as the whole description).Is there an easy way to do that using Beautiful Soup?For now I am getting the links of the companies of the first page.

import requests
from bs4 import BeautifulSoup

r = requests.get('https://stackoverflow.com/jobs/companies')
src = r.content
soup = BeautifulSoup(src,'lxml')
urls=[]

for h2_tag in soup.find_all("h2"):
    a_tag = h2_tag.find('a')
    urls.append(a_tag.attrs['href'])

print(urls)

Source: Python Questions

LEAVE A COMMENT