Web Scraping Using Selenium - Explicit Wait For Visibility of All Elements

Selenium Wait - Explicit Wait For Visibility of All Elements

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for visibility of elements. All elements for a locator must be visible (can be seen or has at least 1px width).

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Visibility of Any Element Given Its CSS Selector

1. Create a file seleniumwaitvisibilityall.py and paste the following codes

   from selenium import webdriver
   from selenium.webdriver.common.by import By
   from selenium.webdriver.support.ui import WebDriverWait
   from selenium.webdriver.support import expected_conditions as EC
   from selenium.common.exceptions import TimeoutException
   from datetime import datetime
   

   The codes above imports the required library that we will use.
2. Add this line

   driver = webdriver.Firefox(executable_path="geckodriver.exe")
   

   The code above will create a webdriver instance for Firefox.
3. Add this line

   driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
   

   The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).
4. Add this function as is

   def wait_for_the_elements(wait_time: int, selector: str):
       try:
           print("[{}] Finding element {}".format(str(datetime.now()), selector))
           WebDriverWait(driver, wait_time).until(
               EC.visibility_of_all_elements_located((By.CSS_SELECTOR, selector))
           )
           print("[{}] Element found".format(str(datetime.now())))
       except TimeoutException as e:
           print("[{}] Element did not load".format(str(datetime.now())))
   

   This method will wait for any elements given its selector to become visible given the waiting time. It will print a message if the element is visible or not.
5. Add this line

   wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
   

   This line will call the method we created and will display Element found.
6. Add this line

   wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
   

   This line will call the method we created and wait for 6 seconds until it gives an error.
7. Add this line

   wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
   

   Again will call the method with a non visible element this time it is 9 seconds.
8. Add this line

   wait_for_the_elements(12, "div#hiddenElements > div")
   

   Again will call the method with a non visible and visible elements this time it is 12 seconds.

   <div id="hiddenElements">
       <div style="display: none;">This is a hidden element</div>
       <div style="width: 0px; height: 0px;"></div>
       <div>Visible div</div>
   </div>
   

   The above HTML is taken from the website.
9. Add this line

   driver.close()
   

   The line will close the webdriver as well as the browser.
10. Run the seleniumwaitvisibilityall.py. It should do the following:
    • Open the firefox browser
    • Browser goes to https://slackingslacker.github.io/seleniumindex
    • Call the Method 4 times which prints messages in the console
    • Closes the browser

 

Program Sample Output

[2020-06-01 23:41:56.278252] Finding element div#hiddenElements > div:nth-of-type(3)
[2020-06-01 23:41:56.331652] Element found
[2020-06-01 23:41:56.331652] Finding element div#hiddenElements > div:nth-of-type(1)
[2020-06-01 23:42:02.804026] Element did not load
[2020-06-01 23:42:02.804026] Finding element div#hiddenElements > div:nth-of-type(2)
[2020-06-01 23:42:12.094494] Element did not load
[2020-06-01 23:42:12.094494] Finding element div#hiddenElements > div
[2020-06-01 23:42:24.440167] Element did not load

Output explanations

1. The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(3) and checks visibility
2. The code found the element
3. The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(1) and checks visibility
4. The code does not find the element within 6 seconds
5. The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(2) and checks visibility
6. The code does not find the element within 9 seconds
7. The code looks for all elements given the CSS selector div#hiddenElements > div and checks visibility
8. The code does not find the element within 12 seconds becuase only 1 div is visible

<div style="display: none;">This is a hidden element</div>
<div style="width: 0px; height: 0px;"></div>
<div>Visible div</div>

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

 

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_all_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
wait_for_the_elements(12, "div#hiddenElements > div")
driver.close()

 

Conclusion

Waiting time in selenium can be used for waiting all element's visibility via locators.