Showing posts with label query selectors. Show all posts
Showing posts with label query selectors. Show all posts

Friday, June 12, 2020

Basic Web Scraping Using Python - A Beginner's Guide to using Requests and Selenium

Beginner Guide to Web Scraping Using Python For Requests and Selenium (Live Examples)

 
Web scraping is gathering data from websites for later processing. You can do it manually or automated using softwares like selenium and can be written in different languages.
 
We will try to do web scraping and these set of blogs are for step by step learning on how to scrape data. I divided the blogs into pieces so that you will not have information overload and can learn the different codes to gather data from a website. The website that we are going to scrape is the HTML sample that I uploaded for learning purposes. You can access the following html samples:
 

The codes can be accessed here:

 
 

Python Selenium - A Beginners Guide Series:

 
Hope you learn and enjoy this series!
 
 

Saturday, May 23, 2020

Web Scraping Using Selenium - Navigation

Selenium Navigation - driver.get

Navigations in selenium can be done in different ways. In this tutorial, we will use the driver.get functionality.
But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Navigation Code Using driver.get

  1. Create a file seleniumnav.py and paste the following codes
    from selenium import webdriver
    import time
    
    The codes above imports the required library that we will use.
  2. Add this line
    driver = webdriver.Firefox(executable_path="geckodriver.exe")
    
    The code above will create a webdriver instance for Firefox.
  3. Add this line
    driver.get("https://www.google.com")
    
    The line will got to the website (https://www.google.com).
  4. Add this line
    time.sleep(10)
    
    We will pause the program for 10 seconds.
  5. Add this line
    driver.get("https://slackingslacker.github.io/seleniumindex")
    
    The line will got to the website (https://slackingslacker.github.io/seleniumindex). This is way to navigate to different pages of the site or different websites.
  6. Add this line
    time.sleep(10)
    
    We will pause the program for 10 seconds.
  7. Add this line
    driver.close()
    
    The line will close the webdriver as well as the browser.
  8. Run the seleniumnav.py. It should do the following:
    • Open the firefox browser
    • Browser goes to www.google.com
    • Halts for 10 seconds
    • Browser goes to https://slackingslacker.github.io/seleniumindex
    • Halts for 10 seconds
    • Closes the browser
 

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://www.google.com")
time.sleep(10)

driver.get("https://slackingslacker.github.io/seleniumindex")
time.sleep(10)

driver.close()

 

Conclusion

Navigation from page to page or website to website can be done using driver.get functionality
 

Tuesday, May 5, 2020

Web Scraping Using Requests - Query Selectors/CSS Selectors

Requests and Beautifulsoup Searching Using CSS Selectors

We will use beautifulSoup to get different tags using different selectors.
 
Before we proceed, please make sure you have read the first and second blogs on this series to do the prerequisites.
 

Lets start to code.

  1. Create a file simpleselectors.py and paste the following codes. We will do it differently to print the tags that we will find using a method.
    from bs4 import BeautifulSoup
    import requests
    
    url = "https://slackingslacker.github.io/simpleselectors.html"
    html_doc = requests.get(url).text
    soup = BeautifulSoup(html_doc, "html.parser")
    
    
    def find_by_selector(selector: str):
        elements = soup.select(selector)
        for el in elements:
            print("============================")
            print("name : " + el.name)
            print("attributes : " + str(el.attrs))
            print(el)
    
  2. Add the code below.
    find_by_selector(".column")
    
    Run the code. It should print all the div tag elements that has the class attribute with value column
  3. Comment the recently added line by putting # at start of the line.
    find_by_selector(".table.is-narrow")
    
    It should print all table tags that has the class attribute with value table and is-narrow.
  4. Comment the recently added line and add the line below then run the code.
    find_by_selector(".columns .table")
    
    It should print all table tags that has the class attribute with value table and must be a child of an element (div) with class attribute and value columns.
  5. Comment the recently added line and add the line below then run the code.
    find_by_selector("#total")
    
    It should print a span tag that has the id attribute with value total.
  6. Comment the recently added line and add the line below then run the code.
    find_by_selector("div")
    
    It should print all div tags.
  7. Comment the recently added line and add the line below then run the code.
    find_by_selector("td.has-text-link")
    
    It should print all td tags that has the class attribute with value has-text-link.
  8. Comment the recently added line and add the line below then run the code.
    find_by_selector("b,i")
    
    It should print all b and i tags.
  9. Comment the recently added line and add the line below then run the code.
    find_by_selector("table th")
    
    It should print all th tags under the table tags.
  10. Comment the recently added line and add the line below then run the code.
    find_by_selector("div > span")
    
    It should print all span tags that is a direct child of div tags.
  11. Comment the recently added line and add the line below then run the code.
    find_by_selector("span~p")
    
    It should print all p tags that is preceeded by span tags.
  12. Comment the recently added lines and add the line below then run the code.
    find_by_selector("[colspan]")
    
    It should print all tags with colspan attribute.
  13. Comment the recently added lines and add the line below then run the code.
    find_by_selector("[colspan='2']")
    
    It should print all tags with colspan attribute that has a value of 2.
  14. Comment the recently added lines and add the line below then run the code.
    find_by_selector("[class^='has']")
    
    It should print all tags with class attribute that has a value that starts with has.
  15. Comment the recently added lines and add the line below then run the code.
    find_by_selector("[class$='link']"
    
    It should print all tags with class attribute that has a value that ends with link.
  16. Comment the recently added lines and add the line below then run the code.
    find_by_selector("[class*='text']")
    
    It should print all tags with class attribute that has a value that contains text.
  17. Comment the recently added lines and add the line below then run the code.
    find_by_selector("span:empty")
    
    It should print all tags that have no child.
  18. Comment the recently added lines and add the line below then run the code.
    find_by_selector("tr:first-child")
    
    It should print all tr tags that is the first child of a parent tag.
  19. Comment the recently added lines and add the line below then run the code.
    find_by_selector("tr:last-child")
    
    It should print all tr tags that is the last child of a parent tag.
  20. Comment the recently added lines and add the line below then run the code.
    find_by_selector("td:nth-child(3)")
    
    It should print all td tags that is the 3rd child of a parent tag.
  21. Comment the recently added lines and add the line below then run the code.
    find_by_selector("a:first-of-type")
    
    It should print all a tags that is the first child (first of type anchor) of a parent tag.
  22. Comment the recently added lines and add the line below then run the code.
    find_by_selector("a:last-of-type")
    
    It should print all a tags that is the last child (last of type anchor) of a parent tag.
  23. Comment the recently added lines and add the line below then run the code.
    find_by_selector("a:nth-of-type(2)")
    
    It should print all a tags that is the 2nd child (2nd of type anchor) of a parent tag.
  24. Comment the recently added lines and add the line below then run the code.
    find_by_selector("span:only-child")
    
    It should print all span tags that is the only child of a parent tag.
  25. Comment the recently added lines and add the line below then run the code.
    find_by_selector("table > tr:not(:first-child)")
    
    It should print all tr tags that is not the first child of the table. It will print all tr tags that does not contain the titles for the columns.
  26. Comment the recently added lines and add the line below then run the code.
    find_by_selector("div.column:nth-of-type(1) > table > tr:nth-child(4) > td:nth-child(2)")
    
    It should print all td tags that is the 2nd child of a parent tr which is the 4th child of a table tag which is the child of a div that is the first of type (first div) from a parent tag.
  27. Comment the recently added lines and add the line below then run the code.
    print(soup.select_one("div.column:nth-of-type(1) > table > tr:nth-child(4) > td:nth-child(2)").text)
    
    By using the query selector from previous example, we just get a single tag using select_one which will find the first tag matching the query and displaying the text Chicken Legs.
 

Conclusion

We used the select and select_one to get all or one of the Tags in different ways.
 

Programming

Basic Web Scraping Using Python - A Beginner's Guide to using Requests and Selenium

Beginner Guide to Web Scraping Using Python For Requests and Selenium (Live Examples)   Web scraping is gathering da...