Web Scraping Using Selenium - Explicit Wait for Presence of Any Elements

Selenium Wait - Explicit Wait For Presence of Any Elements

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for presence of any elements given a locator. This is a way to check if the elements already load.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Presence of Any Elements Given a Locator

1. Create a file seleniumwaitpresenceall.py and paste the following codes

   from selenium import webdriver
   from selenium.webdriver.common.by import By
   from selenium.webdriver.support.ui import WebDriverWait
   from selenium.webdriver.support import expected_conditions as EC
   from selenium.common.exceptions import TimeoutException
   from datetime import datetime
   

   The codes above imports the required library that we will use.
2. Add this line

   driver = webdriver.Firefox(executable_path="geckodriver.exe")
   

   The code above will create a webdriver instance for Firefox.
3. Add this line

   driver.get("https://slackingslacker.github.io/seleniumindex")
   

   The line will got to the website (https://slackingslacker.github.io/seleniumindex).
4. Add this function as is

   def wait_for_the_elements(wait_time: int, el_name: str):
       try:
           print("[{}] Finding element {}".format(str(datetime.now()), el_name))
           WebDriverWait(driver, wait_time).until(
               EC.presence_of_all_elements_located((By.TAG_NAME, el_name))
           )
           print("[{}] Element found".format(str(datetime.now())))
       except TimeoutException as e:
           print("[{}] Element did not load".format(str(datetime.now())))
   

   This method will wait for the any element given a locator to load at a given waiting time. It will print a message if any element loaded or not.
5. Add this line

   wait_for_the_elements(3, "nav")
   

   This line will call the method we created and will display Element found.
6. Add this line

   wait_for_the_elements(6, "table")
   

   This line will call the method we created and wait for 6 seconds until it gives an error.
7. Add this line

   wait_for_the_elements(9, "ul")
   

   Again will call the method with a non existing element this time it is 9 seconds.
8. Add this line

   driver.close()
   

   The line will close the webdriver as well as the browser.
9. Run the seleniumwaitpresenceall.py. It should do the following:
   • Open the firefox browser
   • Browser goes to https://slackingslacker.github.io/seleniumindex
   • Call the Method 3 times which prints messages in the console
   • Closes the browser

 

Program Sample Output

[2020-06-01 01:02:57.360303] Finding element nav
[2020-06-01 01:02:57.375942] Element found
[2020-06-01 01:02:57.375942] Finding element table
[2020-06-01 01:03:03.456108] Element did not load
[2020-06-01 01:03:03.456108] Finding element ul
[2020-06-01 01:03:12.497395] Element did not load

Output explanations

1. The code looks for any element given the tag nav
2. The code found the element
3. The code looks for any element given the tag table
4. The code does not find the element within 6 seconds
5. The code looks for any element given the tag ul
6. The code does not find the element within 9 seconds

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

 

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")

def wait_for_the_elements(wait_time: int, el_name: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), el_name))
        WebDriverWait(driver, wait_time).until(
            EC.presence_of_all_elements_located((By.TAG_NAME, el_name))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_elements(3, "nav")
wait_for_the_elements(6, "table")
wait_for_the_elements(9, "ul")
driver.close()

 

Conclusion

Waiting time in selenium can be set to wait for any element given a locator.

 

Web Scraping Using Selenium - Navigation Using Browser History

Selenium Navigation - History

Navigations in selenium can be done in different ways. In this tutorial, we will use the back and forward button of the browser as a form of navigation.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Navigation History - Back and Forward

1. Create a file seleniumnavhistory.py and paste the following codes

   from selenium import webdriver
   import time
   

   The codes above imports the required library that we will use.
2. Add this line

   driver = webdriver.Firefox(executable_path="geckodriver.exe")
   

   The code above will create a webdriver instance for Firefox.
3. Add this line

   driver.get("https://slackingslacker.github.io/seleniumindex")
   

   The line will execute a script to go to the webpage https://slackingslacker.github.io/seleniumindex.
4. Add this line

   time.sleep(3)
   

   We will pause the program for 3 seconds.
5. Add this line

   driver.get("https://slackingslacker.github.io/seleniumindex#/about")
   

   The line will execute a script to go to the webpage https://slackingslacker.github.io/seleniumindex#/about.
6. Add this line

   time.sleep(3)
   

   We will pause the program for 3 seconds. At this time we have history in the browser.
7. Add this line

   driver.back()
   

   The line will go back to the previous page which is https://slackingslacker.github.io/seleniumindex.
8. Add this line

   time.sleep(10)
   

   We will pause the program for 10 seconds. You may have noticed that the current page is the main page.
9. Add this line

   driver.forward()
   

   The line will go back to the next page which is https://slackingslacker.github.io/seleniumindex#/about.
10. Add this line

    time.sleep(10)
    

    We will pause the program for 10 seconds. You may have noticed that the current page is the about page.
11. Add this line

    driver.close()
    

    The line will close the webdriver as well as the browser.
12. Run the seleniumnavhistory.py. It should do the following:
    • Open the firefox browser
    • Browser goes to https://slackingslacker.github.io/seleniumindex
    • Halts for 3 seconds
    • Browser goes to https://slackingslacker.github.io/seleniumindex#/about
    • Halts for 3 seconds
    • Browser goes to https://slackingslacker.github.io/seleniumindex using the back of history
    • Halts for 10 seconds
    • Browser goes to https://slackingslacker.github.io/seleniumindex#/about using the forward of history
    • Halts for 10 seconds
    • Closes the browser

 

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")
time.sleep(3)
driver.get("https://slackingslacker.github.io/seleniumindex#/about")
time.sleep(3)

driver.back()
time.sleep(10)

driver.forward()
time.sleep(10)

driver.close()

 

Conclusion

Navigation in selenium can be done using browser history.

 

Web Scraping Using Selenium - Navigation Using Script Execution

Selenium Navigation - Using Javascript

Navigations in selenium can be done in different ways. In this tutorial, we will invoke a script to navigate to a different page.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Navigation Executing a Script

1. Create a file seleniumnavscript.py and paste the following codes

   from selenium import webdriver
   import time
   

   The codes above imports the required library that we will use.
2. Add this line

   driver = webdriver.Firefox(executable_path="geckodriver.exe")
   

   The code above will create a webdriver instance for Firefox.
3. Add this line

   driver.execute_script("window.location.href='https://slackingslacker.github.io/seleniumindex#/about';")
   

   The line will execute a script to go to the webpage https://slackingslacker.github.io/seleniumindex#/about.
4. Add this line

   time.sleep(10)
   

   We will pause the program for 10 seconds so that you can see that the page loaded.
5. Add this line

   driver.close()
   

   The line will close the webdriver as well as the browser.
6. Run the seleniumnavscript.py. It should do the following:
   • Open the firefox browser
   • Execute a script to navigate to a page
   • Halts for 10 seconds
   • Closes the browser

 

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.execute_script("window.location.href='https://slackingslacker.github.io/seleniumindex#/about';")
time.sleep(10)

driver.close()

 

Conclusion

Navigation in selenium can be done using script execution.

 

Web Scraping Using Selenium - Navigation Using Form Submit

Selenium Navigation - Form Submission

Navigations in selenium can be done in different ways. In this tutorial, we will use the form submission functionality.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Navigation Using Form Element Submit

1. Create a file seleniumnavform.py and paste the following codes

   from selenium import webdriver
   import time
   

   The codes above imports the required library that we will use.
2. Add this line

   driver = webdriver.Firefox(executable_path="geckodriver.exe")
   

   The code above will create a webdriver instance for Firefox.
3. Add this line

   driver.get("https://slackingslacker.github.io/seleniumindex#/forms")
   

   The line will go to the website (https://slackingslacker.github.io/seleniumindex#/forms).
4. Add this line

   time.sleep(5)
   

   We will pause the program for 5 seconds.
5. Add this line

   el = driver.find_element_by_css_selector("div > form")
   

   This code will find an element using a CSS selector. The code will find a form element that is a direct child of a div.
6. Add this line

   el.submit()
   

   This will submit form.
7. Add this line

   time.sleep(10)
   

   We will pause the program for 10 seconds to see that it submitted the form.
8. Add this line

   driver.close()
   

   The line will close the webdriver as well as the browser.
9. Run the seleniumnavform.py. It should do the following:
   • Open the firefox browser
   • Browser goes to https://slackingslacker.github.io/seleniumindex#/forms
   • Halts for 5 seconds
   • Finds the form tag
   • Submits the form without click a button
   • Halts for 10 seconds
   • Closes the browser

 

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/forms")
time.sleep(5)

el = driver.find_element_by_css_selector("div > form")
el.submit()
time.sleep(10)

driver.close()

 

Conclusion

Navigation in selenium can be done using submission of a form element.

 

Web Scraping Using Selenium - Navigation with Anchor Element

Selenium Navigation - Anchor Navigation

Navigations in selenium can be done in different ways. In this tutorial, we will use the click functionality of an anchor tag.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Navigation Using Anchor Element

1. Create a file seleniumnavachor.py and paste the following codes

   from selenium import webdriver
   import time
   

   The codes above imports the required library that we will use.
2. Add this line

   driver = webdriver.Firefox(executable_path="geckodriver.exe")
   

   The code above will create a webdriver instance for Firefox.
3. Add this line

   driver.get("https://slackingslacker.github.io/seleniumindex")
   

   The line will got to the website (https://slackingslacker.github.io/seleniumindex).
4. Add this line

   time.sleep(5)
   

   We will pause the program for 5 seconds.
5. Add this line

   el = driver.find_element_by_css_selector("div[class='navbar-start'] > a:last-of-type")
   

   This code will find an element using a CSS selector. We will tackle this more in the future. The element that we are looking for is the About link in the Menu Bar at the top
   
6. Add this line

   el.click()
   

   This will click the About in the menu.
7. Add this line

   time.sleep(10)
   

   We will pause the program for 10 seconds to see that it loaded the about page.
8. Add this line

   driver.close()
   

   The line will close the webdriver as well as the browser.
9. Run the seleniumnavachor.py. It should do the following:
   • Open the firefox browser
   • Browser goes to https://slackingslacker.github.io/seleniumindex
   • Halts for 5 seconds
   • Finds the anchor tag for the About in the top menu
   • Clicks the link
   • Halts for 10 seconds
   • Closes the browser

 

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")
time.sleep(5)

el = driver.find_element_by_css_selector("div[class='navbar-start'] > a:last-of-type")
el.click()
time.sleep(10)

driver.close()

 

Conclusion

Navigation in selenium can be done using an anchor element.