🕸️ Web Scraping with Selenium and Beautiful Soup

Driver instantiation and retrieving HTML data

Making soup

Getting the title

Getting the author

Getting the article text


Inputting the search query and hitting "enter"

Making soup

Getting All Article Titles

Drop the two non-titles from the end of the list

Remove the 4 extra titles between header article and other articles

Getting head article

Checking the number of articles in the heading

Getting the rest of the articles

Combine article titles and urls as a dataframe

Scraping the first article in the dataframe

The title

What can be found in the meta data?

Title via meta data

URL via meta data

Image via meta data

Getting author data

First method of getting article text

Second method of getting article text

Compiling the data as a dataframe