Web Scraping Using Selenium - Explicit Wait for Title Element

Selenium Wait - Explicit Wait For Title

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for title. Good example is loading a blog or a product page with a specific title.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Specific Title

1. Create a file seleniumwaittitle.py and paste the following codes

   from selenium import webdriver
   from selenium.webdriver.support.ui import WebDriverWait
   from selenium.webdriver.support import expected_conditions as EC
   from selenium.common.exceptions import TimeoutException
   from datetime import datetime
   

   The codes above imports the required library that we will use.
2. Add this line

   driver = webdriver.Firefox(executable_path="geckodriver.exe")
   

   The code above will create a webdriver instance for Firefox.
3. Add this line

   driver.get("https://slackingslacker.github.io/seleniumindex")
   

   The line will got to the website (https://slackingslacker.github.io/seleniumindex).
4. Add this function as is

   def wait_for_the_title(wait_time: int, doc_title: str):
       try:
           print("[{}] Waiting for title {}".format(str(datetime.now()), doc_title))
           WebDriverWait(driver, wait_time).until(EC.title_is(doc_title))
           print("[{}] Title found".format(str(datetime.now())))
       except TimeoutException as e:
           print("[{}] Error waiting for title".format(str(datetime.now())))
   

   This method will wait for the an specific document title to load at a given waiting time. It will print a message if the document title loaded loaded or not.
5. Add this line

   wait_for_the_title(3, "Do It Simpler - VUE - Bulma For Scraping")
   

   This line will call the method we created and will display Title found.
6. Add this line

   wait_for_the_title(6, "Not the title")
   

   This line will call the method we created and wait for 6 seconds until it gives an error.
7. Add this line

   wait_for_the_title(9, "Another wrong title")
   

   Again will call the method with a non existing element this time it is 9 seconds.
8. Add this line

   driver.close()
   

   The line will close the webdriver as well as the browser.
9. Run the seleniumwaittitle.py. It should do the following:
   • Open the firefox browser
   • Browser goes to https://slackingslacker.github.io/seleniumindex
   • Call the Method 3 times which prints messages in the console
   • Closes the browser

 

Program Sample Output

[2020-05-31 23:58:37.029019] Waiting for title Do It Simpler - VUE - Bulma For Scraping
[2020-05-31 23:58:37.044668] Title found
[2020-05-31 23:58:37.044668] Waiting for title Not the title
[2020-05-31 23:58:43.274813] Error waiting for title
[2020-05-31 23:58:43.274813] Waiting for title Another wrong title
[2020-05-31 23:58:52.510780] Error waiting for title

Output explanations

1. The code looks for the title that is equal to Do It Simpler - VUE - Bulma For Scraping
2. The code found the title
3. The code looks for the title that is equal to Not the title
4. The code does not find the title within 6 seconds
5. The code looks for the title that is equal to Another wrong title
6. The code does not find the title within 9 seconds

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

 

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")

def wait_for_the_title(wait_time: int, doc_title: str):
    try:
        print("[{}] Waiting for title {}".format(str(datetime.now()), doc_title))
        WebDriverWait(driver, wait_time).until(EC.title_is(doc_title))
        print("[{}] Title found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Error waiting for title".format(str(datetime.now())))

wait_for_the_title(3, "Do It Simpler - VUE - Bulma For Scraping")
wait_for_the_title(6, "Not the title")
wait_for_the_title(9, "Another wrong title")
driver.close()

 

Conclusion

Waiting time in selenium can be set to wait for a specific title.