Web Scraping Using Selenium - Implicit Wait

Selenium Wait - Implicit Wait

Waiting in selenium can be done in different ways. In this tutorial, we will use the implicit wait functionality.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Implicit Wait

1. Create a file seleniumimplicitwait.py and paste the following codes

   from selenium import webdriver
   from datetime import datetime
   

   The codes above imports the required library that we will use.
2. Add this line

   driver = webdriver.Firefox(executable_path="geckodriver.exe")
   

   The code above will create a webdriver instance for Firefox.
3. Add this line

   driver.implicitly_wait(5)
   

   This line will set the waiting time for the element to load. It is set to 5 seconds.
4. Add this line

   driver.get("https://slackingslacker.github.io/seleniumindex")
   

   The line will got to the website (https://slackingslacker.github.io/seleniumindex).
5. Add this function as is

   def find_the_element(selector: str):
       try:
           print("[{}] Finding element {}".format(str(datetime.now()), selector))
           driver.find_element_by_css_selector(selector)
           print("[{}] Element found".format(str(datetime.now())))
       except Exception as e:
           print("[{}] Error is {}".format(str(datetime.now()), str(e)))
   

   This method will check for the element and will handle the error if the element did not load or do not exists in the given waiting time.
6. Add this line

   find_the_element("nav[role='navigation']")
   

   This line will call the method we created and checks if the CSS selector exists or wait for the given time.
7. Add this line

   find_the_element("#noneExistentId")
   

   Again will call the method with a non existing CSS selector.
8. Add this line

   find_the_element("#anotherNoneExistentId")
   

   Again will call the method with a non existing CSS selector.
9. Add this line

   driver.close()
   

   The line will close the webdriver as well as the browser.
10. Run the seleniumimplicitwait.py. It should do the following:
    • Open the firefox browser
    • Sets the implicit wait to 5 seconds
    • Browser goes to https://slackingslacker.github.io/seleniuminde
    • Call the Method 3 times which prints messages in the console
    • Closes the browser

 

Program Sample Output

[2020-05-31 22:20:55.018838] Finding element nav[role='navigation']
[2020-05-31 22:20:55.034466] Element found
[2020-05-31 22:20:55.034466] Finding element #noneExistentId
[2020-05-31 22:21:00.058297] Error is Message: Unable to locate element: #noneExistentId
[2020-05-31 22:21:00.058297] Finding element #anotherNoneExistentId
[2020-05-31 22:21:05.074049] Error is Message: Unable to locate element: #anotherNoneExistentId

Output explanations

1. The code looks for the selector nav[role='navigation']
2. The code found the CSS selector
3. The code looks for the selector #noneExistentId
4. The code does not find the CSS selector within 5 seconds
5. The code looks for the selector #anotherNoneExistentId
6. The code does not find the CSS selector within 5 seconds

As you may have noticed that the waiting time is constant to 5 seconds as we have set earlier.

 

Final Selenium Code

from selenium import webdriver
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.implicitly_wait(5)
driver.get("https://slackingslacker.github.io/seleniumindex")

def find_the_element(selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        driver.find_element_by_css_selector(selector)
        print("[{}] Element found".format(str(datetime.now())))
    except Exception as e:
        print("[{}] Error is {}".format(str(datetime.now()), str(e)))

find_the_element("nav[role='navigation']")
find_the_element("#noneExistentId")
find_the_element("#anotherNoneExistentId")
driver.close()

 

Conclusion

Waiting time in selenium can be set to a constant value throughout the selenium life cycle.