Basic Programming - Do It Simpler: 2020

Friday, June 12, 2020

Basic Web Scraping Using Python - A Beginner's Guide to using Requests and Selenium

Beginner Guide to Web Scraping Using Python For Requests and Selenium (Live Examples)

Web scraping is gathering data from websites for later processing. You can do it manually or automated using softwares like selenium and can be written in different languages.

We will try to do web scraping and these set of blogs are for step by step learning on how to scrape data. I divided the blogs into pieces so that you will not have information overload and can learn the different codes to gather data from a website. The website that we are going to scrape is the HTML sample that I uploaded for learning purposes. You can access the following html samples:

The codes can be accessed here:

Python Scraper Codes

Python Request with Beautifulsoup - A Beginners Guide Series:

Python Selenium - A Beginners Guide Series:

Hope you learn and enjoy this series!

Web Scraping Using Selenium - Explicit Wait For Element Selection Using Locator

Selenium Wait - Explicit Wait For Element Selection Using Locator

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is to be selected using locator.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Selection Using Locator

Create a file seleniumwaitselection.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_selection(wait_time: int, selector: str, is_selected: bool):
    try:
        print("[{}] Waiting element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.element_located_selection_state_to_be((By.CSS_SELECTOR, selector), is_selected)
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not found".format(str(datetime.now())))

This method will wait for the element to be checked. It will print a message if the element did loaded or not.

Add this line
```
wait_for_element_selection(3, "div#elementSelection input[name='forSelected']", True)
```
This line will call the method we created and will wait within 3 seconds until it displays Element found. True parameter means that the checkbox must be checked.
Add this line
```
wait_for_element_selection(6, "div#elementSelection input[name='forSelected']", False)
```
This line will call the method we created and will wait within 6 seconds until it displays Element not found. False parameter means that the checkbox must NOT be checked.
Add this line
```
wait_for_element_selection(9, "div#elementSelection input[name='notSelected']", True)
```
This line will call the method we created and will wait within 9 seconds until it displays Element not found. True parameter means that the checkbox must NOT be checked.
Add this line
```
wait_for_element_selection(12, "div#elementSelection input[name='notSelected']", False)
```
This line will call the method we created and will wait within 12 seconds until it displays Element found. False parameter means that the checkbox must be checked.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitselection.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 4 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 23:19:12.181093] Waiting element div#elementSelection input[name='forSelected']
[2020-06-12 23:19:12.250153] Element found
[2020-06-12 23:19:12.250153] Waiting element div#elementSelection input[name='forSelected']
[2020-06-12 23:19:18.562138] Element not found
[2020-06-12 23:19:18.562138] Waiting element div#elementSelection input[name='notSelected']
[2020-06-12 23:19:27.931689] Element not found
[2020-06-12 23:19:27.931689] Waiting element div#elementSelection input[name='notSelected']
[2020-06-12 23:19:27.947199] Element found

Output explanations

The code waits for a given the CSS selector div#elementSelection input[name='forSelected'] that is checked.
The code found the a checkbox is checked
The code waits for a given the CSS selector div#elementSelection input[name='forSelected'] that is NOT checked.
The code did not find the a checkbox is not checked within 6 seconds
The code waits for a given the CSS selector div#elementSelection input[name='notSelected'] that is checked.
The code did not find the a checkbox is checked
The code waits for a given the CSS selector div#elementSelection input[name='notSelected'] that is NOT checked.
The code found the checkbox is NOT checked within 12 seconds

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_selection(wait_time: int, selector: str, is_selected: bool):
    try:
        print("[{}] Waiting element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.element_located_selection_state_to_be((By.CSS_SELECTOR, selector), is_selected)
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not found".format(str(datetime.now())))

wait_for_element_selection(3, "div#elementSelection input[name='forSelected']", True)
wait_for_element_selection(6, "div#elementSelection input[name='forSelected']", False)
wait_for_element_selection(9, "div#elementSelection input[name='notSelected']", True)
wait_for_element_selection(12, "div#elementSelection input[name='notSelected']", False)
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a checked element using locator.

Web Scraping Using Selenium - Explicit Wait For Element To Be Selected Using Locator

Selenium Wait - Explicit Wait For Element To Be Selected Using Locator

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is to be selected using locator.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Selected Using Locator

Create a file seleniumwaitselected.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_selected(wait_time: int, selector: str):
    try:
        print("[{}] Waiting element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.element_located_to_be_selected((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element selected".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not selected".format(str(datetime.now())))

This method will wait for the element to be selected. It will print a message if the element did loaded or not.

Add this line
```
wait_for_element_selected(3, "div#elementSelected input[name='forSelectedRadio']")
```
This line will call the method we created and will wait within 3 seconds until it displays Element selected.
Add this line
```
wait_for_element_selected(6, "div#elementSelected input[name='notSelectedRadio']")
```
Again will call the method we created and wait for 6 seconds until it gives a message Element not selected.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitselected.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 2 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 23:08:18.550321] Waiting element div#elementSelected input[name='forSelectedRadio']
[2020-06-12 23:08:18.612853] Element selected
[2020-06-12 23:08:18.612853] Waiting element div#elementSelected input[name='notSelectedRadio']
[2020-06-12 23:08:24.855407] Element not selected

Output explanations

The code looks for an element given the CSS selector div#elementSelected input[name='forSelectedRadio'] that is selected
The code found the a selected radio button
The code looks for an element given the CSS selector div#elementSelected input[name='notSelectedRadio'] that is selected
The code does not find the element within 6 seconds because the radio button is not selected

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_selected(wait_time: int, selector: str):
    try:
        print("[{}] Waiting element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.element_located_to_be_selected((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element selected".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not selected".format(str(datetime.now())))

wait_for_element_selected(3, "div#elementSelected input[name='forSelectedRadio']")
wait_for_element_selected(6, "div#elementSelected input[name='notSelectedRadio']")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a selected element using locator.

Web Scraping Using Selenium - Explicit Wait For Element Selection

Selenium Wait - Explicit Wait For Element Selection

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is to be selected.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Selection

Create a file seleniumwaitelementselection.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_selection(wait_time: int, element, is_selected: bool):
    try:
        print("[{}] Waiting element".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_selection_state_to_be(element, is_selected)
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not found".format(str(datetime.now())))

This method will wait for the element to be selected. It will print a message if the element did loaded or not.

Add these lines

check_el = driver.find_element_by_css_selector("div#elementSelection input[name='forSelected']")
uncheck_el = driver.find_element_by_css_selector("div#elementSelection input[name='notSelected']")

This lines will find the checkbox that we will use.

Add this line
```
wait_for_element_selection(3, check_el, True)
```
This line will call the method we created and will wait within 3 seconds until it displays Element found. True parameter means that the checkbox must be checked.
Add this line
```
wait_for_element_selection(6, check_el, False)
```
This line will call the method we created and will wait within 6 seconds until it displays Element not found. False parameter means that the checkbox must NOT be checked.
Add this line
```
wait_for_element_selection(9, uncheck_el, True)
```
This line will call the method we created and will wait within 9 seconds until it displays Element not found. True parameter means that the checkbox must NOT be checked.
Add this line
```
wait_for_element_selection(12, uncheck_el, False)
```
This line will call the method we created and will wait within 12 seconds until it displays Element found. False parameter means that the checkbox must be checked.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitelementselection.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Finds 2 checkboxes and put them in variables
- Call the Method 4 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 23:28:01.282843] Waiting element
[2020-06-12 23:28:01.336237] Element found
[2020-06-12 23:28:01.336237] Waiting element
[2020-06-12 23:28:07.710786] Element not found
[2020-06-12 23:28:07.710786] Waiting element
[2020-06-12 23:28:16.846971] Element not found
[2020-06-12 23:28:16.846971] Waiting element
[2020-06-12 23:28:16.846971] Element found

Output explanations

The code waits for a checkbox element which must be checked.
The code found a checkbox is checked
The code waits for a checkbox element which must be NOT checked.
The code did not find the checkbox within 6 seconds because it is checked.
The code waits for a checkbox element which must be checked.
The code did not find the checkbox within 6 seconds because it is NOT checked.
The code waits for a checkbox element which must be NOT checked.
The code found the checkbox that is NOT checked within 12 seconds

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_selection(wait_time: int, element, is_selected: bool):
    try:
        print("[{}] Waiting element".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_selection_state_to_be(element, is_selected)
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not found".format(str(datetime.now())))

check_el = driver.find_element_by_css_selector("div#elementSelection input[name='forSelected']")
uncheck_el = driver.find_element_by_css_selector("div#elementSelection input[name='notSelected']")

wait_for_element_selection(3, check_el, True)
wait_for_element_selection(6, check_el, False)
wait_for_element_selection(9, uncheck_el, True)
wait_for_element_selection(12, uncheck_el, False)
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a selected element.

Web Scraping Using Selenium - Explicit Wait For Element To Be Selected

Selenium Wait - Explicit Wait For Element To Be Selected

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is to be selected.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Selected

Create a file seleniumwaitelementselected.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_selected(wait_time: int, element):
    try:
        print("[{}] Waiting element".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_to_be_selected(element)
        )
        print("[{}] Element selected".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not selected".format(str(datetime.now())))

This method will wait for the element to be selected. It will print a message if the element did loaded or not.

Add these lines

check_el = driver.find_element_by_css_selector("div#elementSelected input[name='forSelectedRadio']")
uncheck_el = driver.find_element_by_css_selector("div#elementSelected input[name='notSelectedRadio']")

This lines will find the radio buttons that we will use.

Add this line
```
wait_for_element_selected(3, check_el)
```
This line will call the method we created and will wait within 3 seconds until it displays Element selected.
Add this line
```
wait_for_element_selected(6, uncheck_el)
```
Again will call the method we created and wait for 6 seconds until it gives a message Element not selected.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitelementselected.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Finds 2 radio buttons and put them in variables
- Call the Method 2 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 21:29:41.509204] Waiting element
[2020-06-12 21:29:41.527520] Element selected
[2020-06-12 21:29:41.527520] Waiting element
[2020-06-12 21:29:47.616751] Element not selected

Output explanations

The code waits for an selected element.
The code found the a selected radio button
The code looks for another selected element
The code does not find the element within 6 seconds because the radio button is not selected

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_selected(wait_time: int, element):
    try:
        print("[{}] Waiting element".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_to_be_selected(element)
        )
        print("[{}] Element selected".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not selected".format(str(datetime.now())))

check_el = driver.find_element_by_css_selector("div#elementSelected input[name='forSelectedRadio']")
uncheck_el = driver.find_element_by_css_selector("div#elementSelected input[name='notSelectedRadio']")

wait_for_element_selected(3, check_el)
wait_for_element_selected(6, uncheck_el)
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a selected element.

Web Scraping Using Selenium - Explicit Wait For Clickable Element

Selenium Wait - Explicit Wait For Clickable Element

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is clickable.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Clickable

Create a file seleniumwaitclickable.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_clickable(wait_time: int, id: str):
    try:
        print("[{}] Waiting clickable".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_to_be_clickable((By.ID, id))
        )
        print("[{}] Element clickable".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not clickable".format(str(datetime.now())))

This method will wait for the element to be clickable. It will print a message if the element did loaded or not.

Add this line
```
wait_for_element_clickable(3, "clickableBtn")
```
This line will call the method we created and will wait within 3 seconds until it displays Element clickable.
Add this line
```
wait_for_element_clickable(6, "unclickableBtn")
```
Again will call the method we created and wait for 6 seconds until it gives a message Element not clickable.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitclickable.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 2 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 21:17:02.989380] Waiting clickable
[2020-06-12 21:17:06.081625] Element not clickable
[2020-06-12 21:17:06.081625] Waiting clickable
[2020-06-12 21:17:12.254618] Element not clickable

Output explanations

The code waits for an clickable element.
The code found the a clickable button
The code looks for another clickable element
The code does not find the element within 6 seconds because the button is not clickable cause it is disabled

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_clickable(wait_time: int, id: str):
    try:
        print("[{}] Waiting clickable".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_to_be_clickable((By.ID, id))
        )
        print("[{}] Element clickable".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not clickable".format(str(datetime.now())))

wait_for_element_clickable(3, "clickableBtn")
wait_for_element_clickable(6, "unclickableBtn")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a clickable element.

Web Scraping Using Selenium - Explicit Wait For Alert

Selenium Wait - Explicit Wait For Alert

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if there is an alert popup on the page.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Alert

Create a file seleniumwaitalert.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_alert(wait_time: int):
    try:
        print("[{}] Waiting for alert".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(EC.alert_is_present())
        print("[{}] Alert found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Alert did not load".format(str(datetime.now())))

This method will wait for the a popup alert. It will print a message if the alert loaded or not.

Add this line
```
wait_for_alert(3)
```
This line will call the method we created and will wait for 3 seconds until it displays Alert did not load.
Add this line
```
driver.find_element_by_css_selector("#alertExpectation button").click()
```
This line will call the find a button and click it to trigger an alert.
Add this line
```
wait_for_alert(6)
```
Again will call the method we created and wait for 6 seconds until it gives a message Alert found.
Add this line
```
driver.quit()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitalert.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 1 time which prints messages in the console
- Finds a button and clicks the button
- Call the Method 1 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 17:43:25.998479] Waiting for alert
[2020-06-12 17:43:29.005027] Alert did not load
[2020-06-12 17:43:29.089615] Waiting for alert
[2020-06-12 17:43:29.105222] Alert found

Output explanations

The code waits for an alert.
The code did not find any alert
The code again waits for an alert
The code finds the alert because a button was clicked that opens an alert popup

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_alert(wait_time: int):
    try:
        print("[{}] Waiting for alert".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(EC.alert_is_present())
        print("[{}] Alert found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Alert did not load".format(str(datetime.now())))

wait_for_alert(3)
driver.find_element_by_css_selector("#alertExpectation button").click()
wait_for_alert(6)
driver.quit()

Conclusion

Waiting time in selenium can be used for waiting an alert.

Thursday, June 4, 2020

Web Scraping Using Selenium - Explicit Wait For Element Text Value

Selenium Wait - Explicit Wait For Element Text Value

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting a text from value attribute of an element.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For a Element Text

Create a file seleniumwaittextvalue.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_text(wait_time: int, selector: str, text_value: str):
    try:
        print("[{}] Finding text {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.text_to_be_present_in_element_value((By.CSS_SELECTOR, selector), text_value)
        )
        print("[{}] Text found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Text did not load".format(str(datetime.now())))

This method will wait for the elements value attribute text. It will print a message if the element text loaded or not.

Add this line
```
wait_for_text(3, "div#textExpectation > div:nth-of-type(1)", "This is an input")
```
This line will call the method we created and wait for 3 seconds until it gives an error.

Add this line

wait_for_text(6, "div#textExpectation > div:nth-of-type(2) > input", "This is an input")

This line will call the method we created and will display Text found.

Add this line

wait_for_text(9, "div#textExpectation > div:nth-of-type(1) > input", "this is an input")

Again will call the method we created and wait for 9 seconds until it gives an error.

<div id="textExpectation">
    <div>Expected Text</div>
    <div><input type="text" value="This is an input"></div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaittextvalue.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-04 23:09:35.289359] Finding text div#textExpectation > div:nth-of-type(1)
[2020-06-04 23:09:38.512822] Text did not load
[2020-06-04 23:09:38.512822] Finding text div#textExpectation > div:nth-of-type(2) > input
[2020-06-04 23:09:38.543957] Text found
[2020-06-04 23:09:38.543957] Finding text div#textExpectation > div:nth-of-type(1) > input
[2020-06-04 23:09:47.588846] Text did not load

Output explanations

The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(1) and checks the text for that element
The code does not find the element within 3 seconds
The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(2) and checks the text for that element
The code found the text
The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(1) checks the text for that element
The code does not find the element within 9 seconds
```
<div><input type="text" value="This is an input"></div>
```
The matching of the text is case sensitive so the text does not match.

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_text(wait_time: int, selector: str, text_value: str):
    try:
        print("[{}] Finding text {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.text_to_be_present_in_element_value((By.CSS_SELECTOR, selector), text_value)
        )
        print("[{}] Text found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Text did not load".format(str(datetime.now())))

wait_for_text(3, "div#textExpectation > div:nth-of-type(1)", "This is an input")
wait_for_text(6, "div#textExpectation > div:nth-of-type(2) > input", "This is an input")
wait_for_text(9, "div#textExpectation > div:nth-of-type(1) > input", "this is an input")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting elements value attribute text.

Web Scraping Using Selenium - Explicit Wait For Element Text

Selenium Wait - Explicit Wait For Element Text

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting a text from innerHTML of an element.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For a Element Text

Create a file seleniumwaittext.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_text(wait_time: int, selector: str, text_value: str):
    try:
        print("[{}] Finding text {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.text_to_be_present_in_element((By.CSS_SELECTOR, selector), text_value)
        )
        print("[{}] Text found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Text did not load".format(str(datetime.now())))

This method will wait for the elements innerHTML text. It will print a message if the element text loaded or not.

Add this line
```
wait_for_text(3, "div#textExpectation > div:nth-of-type(1)", "Expected Text")
```
This line will call the method we created and will display Text found.
Add this line
```
wait_for_text(6, "div#textExpectation > div:nth-of-type(2)", "Expected Text"))
```
This line will call the method we created and wait for 6 seconds until it gives an error.

Add this line

wait_for_text(9, "div#textExpectation > div:nth-of-type(1)", "expected text")

Again will call the method we created and wait for 9 seconds until it gives an error.

<div id="textExpectation">
    <div>Expected Text</div>
    <div><input type="text" value="This is an input"></div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaittext.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-04 13:44:09.697415] Finding text div#textExpectation > div:nth-of-type(1)
[2020-06-04 13:44:09.782072] Text found
[2020-06-04 13:44:09.782072] Finding text div#textExpectation > div:nth-of-type(2)
[2020-06-04 13:44:16.139792] Text did not load
[2020-06-04 13:44:16.139792] Finding text div#textExpectation > div:nth-of-type(1)
[2020-06-04 13:44:25.508062] Text did not load

Output explanations

The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(1) and checks the text for that element
The code found the text
The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(2) and checks the text for that element
The code does not find the element within 6 seconds
The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(1) checks the text for that element
The code does not find the element within 9 seconds
```
<div>Expected Text</div>
```
The matching of the text is case sensitive so the text does not match.

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_text(wait_time: int, selector: str, text_value: str):
    try:
        print("[{}] Finding text {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.text_to_be_present_in_element((By.CSS_SELECTOR, selector), text_value)
        )
        print("[{}] Text found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Text did not load".format(str(datetime.now())))

wait_for_text(3, "div#textExpectation > div:nth-of-type(1)", "Expected Text")
wait_for_text(6, "div#textExpectation > div:nth-of-type(2)", "Expected Text")
wait_for_text(9, "div#textExpectation > div:nth-of-type(1)", "expected text")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting elements innerHTML text.

Monday, June 1, 2020

Web Scraping Using Selenium - Explicit Wait For Visibility of All Elements

Selenium Wait - Explicit Wait For Visibility of All Elements

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for visibility of elements. All elements for a locator must be visible (can be seen or has at least 1px width).

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Visibility of Any Element Given Its CSS Selector

Create a file seleniumwaitvisibilityall.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_all_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for any elements given its selector to become visible given the waiting time. It will print a message if the element is visible or not.

Add this line
```
wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
```
Again will call the method with a non visible element this time it is 9 seconds.

Add this line

wait_for_the_elements(12, "div#hiddenElements > div")

Again will call the method with a non visible and visible elements this time it is 12 seconds.

<div id="hiddenElements">
    <div style="display: none;">This is a hidden element</div>
    <div style="width: 0px; height: 0px;"></div>
    <div>Visible div</div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitvisibilityall.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 4 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 23:41:56.278252] Finding element div#hiddenElements > div:nth-of-type(3)
[2020-06-01 23:41:56.331652] Element found
[2020-06-01 23:41:56.331652] Finding element div#hiddenElements > div:nth-of-type(1)
[2020-06-01 23:42:02.804026] Element did not load
[2020-06-01 23:42:02.804026] Finding element div#hiddenElements > div:nth-of-type(2)
[2020-06-01 23:42:12.094494] Element did not load
[2020-06-01 23:42:12.094494] Finding element div#hiddenElements > div
[2020-06-01 23:42:24.440167] Element did not load

Output explanations

The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(3) and checks visibility
The code found the element
The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(1) and checks visibility
The code does not find the element within 6 seconds
The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(2) and checks visibility
The code does not find the element within 9 seconds
The code looks for all elements given the CSS selector div#hiddenElements > div and checks visibility
The code does not find the element within 12 seconds becuase only 1 div is visible

<div style="display: none;">This is a hidden element</div>
<div style="width: 0px; height: 0px;"></div>
<div>Visible div</div>

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_all_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
wait_for_the_elements(12, "div#hiddenElements > div")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting all element's visibility via locators.

Web Scraping Using Selenium - Explicit Wait For Visibility of Any Element

Selenium Wait - Explicit Wait For Visibility of Any Element

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for visibility of elements. Any elements for a locator must be visible (can be seen or has at least 1px width).

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Visibility of Any Element Given Its CSS Selector

Create a file seleniumwaitvisibilityany.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_any_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for any elements given its selector to become visible given the waiting time. It will print a message if the element is visible or not.

Add this line
```
wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
```
Again will call the method with a non visible element this time it is 9 seconds.

Add this line

wait_for_the_elements(12, "div#hiddenElements > div")

Again will call the method with a non visible and visible elements this time it is 12 seconds.

<div id="hiddenElements">
    <div style="display: none;">This is a hidden element</div>
    <div style="width: 0px; height: 0px;"></div>
    <div>Visible div</div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitvisibilityany.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 4 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 23:25:27.525543] Finding element div#hiddenElements > div:nth-of-type(3)
[2020-06-01 23:25:27.594599] Element found
[2020-06-01 23:25:27.594599] Finding element div#hiddenElements > div:nth-of-type(1)
[2020-06-01 23:25:33.604729] Element did not load
[2020-06-01 23:25:33.604729] Finding element div#hiddenElements > div:nth-of-type(2)
[2020-06-01 23:25:42.787525] Element did not load
[2020-06-01 23:25:42.787525] Finding element div#hiddenElements > div
[2020-06-01 23:25:42.891207] Element found

Output explanations

The code looks for any element given the CSS selector div#hiddenElements > div:nth-of-type(3) and checks visibility
The code found the element
The code looks for any element given the CSS selector div#hiddenElements > div:nth-of-type(1) and checks visibility
The code does not find the element within 6 seconds
The code looks for any element given the CSS selector div#hiddenElements > div:nth-of-type(2) and checks visibility
The code does not find the element within 9 seconds
The code looks for any element given the CSS selector div#hiddenElements > div and checks visibility

The code found the a visible element within 12 seconds because the third div found is visible

<div style="display: none;">This is a hidden element</div>
<div style="width: 0px; height: 0px;"></div>
<div>Visible div</div>

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_any_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
wait_for_the_elements(12, "div#hiddenElements > div")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting any element's visibility via locators.

Web Scraping Using Selenium - Explicit Wait For Visibility of An Element

Selenium Wait - Explicit Wait For Visibility of An Element

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for visibility of elements. The element must be visible (can be seen or has at least 1px width).

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Visibility of An Element Given Its CSS Selector

Create a file seleniumwaitvisibilityone.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_the_element(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_element_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for the element given its selector to become visible given the waiting time. It will print a message if the element is visible or not.

Add this line
```
wait_for_the_element(3, "div#hiddenElements > div:nth-of-type(3)")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_element(6, "div#hiddenElements > div:nth-of-type(1)")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_element(9, "div#hiddenElements > div:nth-of-type(2)")
```
Again will call the method with a non visible element this time it is 9 seconds.

Add this line

wait_for_the_element(12, "div#hiddenElements > div")

Again will call the method with a non visible element this time it is 12 seconds. The selector finds the first div.

<div id="hiddenElements">
    <div style="display: none;">This is a hidden element</div>
    <div style="width: 0px; height: 0px;"></div>
    <div>Visible div</div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitvisibilityone.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 4 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 22:58:40.336003] Finding element div#hiddenElements > div:nth-of-type(3)
[2020-06-01 22:58:40.414464] Element found
[2020-06-01 22:58:40.414464] Finding element div#hiddenElements > div:nth-of-type(1)
[2020-06-01 22:58:46.421484] Element did not load
[2020-06-01 22:58:46.421484] Finding element div#hiddenElements > div:nth-of-type(2)
[2020-06-01 22:58:55.767107] Element did not load
[2020-06-01 22:58:55.767107] Finding element div#hiddenElements > div
[2020-06-01 22:59:08.054460] Element did not load

Output explanations

The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(3) and checks visibility
The code found the element
The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(1) and checks visibility
The code does not find the element within 6 seconds
The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(2) and checks visibility
The code does not find the element within 9 seconds
The code looks for an element given the CSS selector div#hiddenElements > div and checks visibility
The code does not find the element within 12 seconds because the first div found is not visible
```
<div style="display: none;">This is a hidden element</div>
```
This is not the same is wait by visibility_of because the element does not need ti be present.

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_the_element(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_element_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_element(3, "div#hiddenElements > div:nth-of-type(3)")
wait_for_the_element(6, "div#hiddenElements > div:nth-of-type(1)")
wait_for_the_element(9, "div#hiddenElements > div:nth-of-type(2)")
wait_for_the_element(12, "div#hiddenElements > div")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting an element visibility without checking its presence.

Web Scraping Using Selenium - Explicit Wait For Visibility of Elements

Selenium Wait - Explicit Wait For Visibility of Elements

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for visibility of elements. The element must be loaded and visible (can be seen or has at least 1px width).

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Visibility of Elements Given Its CSS Selector

Create a file seleniumwaitvisibility.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        element = driver.find_element_by_css_selector(selector)
        WebDriverWait(driver, wait_time).until(EC.visibility_of(element))
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will find the element first given its selector then wait for that element to be visible given the waiting time. It will print a message if the element is visble or not.

Add this line
```
wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
```
Again will call the method we created and wait for 9 seconds until it gives an error.

Add this line

wait_for_the_elements(12, "div#hiddenElements > div")

Again will call the method with a non visible element this time it is 12 seconds. The selector finds the first div.

<div id="hiddenElements">
    <div style="display: none;">This is a hidden element</div>
    <div style="width: 0px; height: 0px;"></div>
    <div>Visible div</div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitvisibility.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 4 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 22:57:20.918719] Finding element div#hiddenElements > div:nth-of-type(3)
[2020-06-01 22:57:20.972116] Element found
[2020-06-01 22:57:20.972116] Finding element div#hiddenElements > div:nth-of-type(1)
[2020-06-01 22:57:27.329329] Element did not load
[2020-06-01 22:57:27.329329] Finding element div#hiddenElements > div:nth-of-type(2)
[2020-06-01 22:57:36.410538] Element did not load
[2020-06-01 22:57:36.410538] Finding element div#hiddenElements > div
[2020-06-01 22:57:48.562325] Element did not load

Output explanations

The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(3) and checks visibility
The code found the element
The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(1) and checks visibility
The code does not find the element within 6 seconds
The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(2) and checks visibility
The code does not find the element within 9 seconds
The code looks for an element given the CSS selector div#hiddenElements > div and checks visibility
The code does not find the element within 12 seconds because the first div found is not visible
```
<div style="display: none;">This is a hidden element</div>
```

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        element = driver.find_element_by_css_selector(selector)
        WebDriverWait(driver, wait_time).until(EC.visibility_of(element))
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
wait_for_the_elements(12, "div#hiddenElements > div")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting elements visibility.

Sunday, May 31, 2020

Web Scraping Using Selenium - Explicit Wait for Presence of An Element

Selenium Wait - Explicit Wait For Presence of An Element

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for presence of an element given its locator. This is a way to check if the element already loaded.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Presence of An Elements Given Its Locator

Create a file seleniumwaitpresenceone.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex).

Add this function as is

def wait_for_the_element(wait_time: int, el_id: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), el_id))
        WebDriverWait(driver, wait_time).until(
            EC.presence_of_all_elements_located((By.ID, el_id))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for an element using locator to load at a given waiting time. It will print a message if the element loaded or not.

Add this line
```
wait_for_the_element(3, "navMenuId")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_element(6, "noneExistentId")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_element(9, "anotherNoneExistentId")
```
Again will call the method with a non existing element this time it is 9 seconds.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitpresenceone.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 13:52:53.817328] Finding element navMenuId
[2020-06-01 13:52:53.886479] Element found
[2020-06-01 13:52:53.886479] Finding element noneExistentId
[2020-06-01 13:53:00.019786] Element did not load
[2020-06-01 13:53:00.019786] Finding element anotherNoneExistentId
[2020-06-01 13:53:09.055716] Element did not load

Output explanations

The code looks for an element given the id navMenuId
The code found the element
The code looks for an element given the id noneExistentId
The code does not find the element within 6 seconds
The code looks for an element given the id anotherNoneExistentId
The code does not find the element within 9 seconds

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")

def wait_for_the_element(wait_time: int, el_id: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), el_id))
        WebDriverWait(driver, wait_time).until(
            EC.presence_of_all_elements_located((By.ID, el_id))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_element(3, "navMenuId")
wait_for_the_element(6, "noneExistentId")
wait_for_the_element(9, "anotherNoneExistentId")
driver.close()

Conclusion

Waiting time in selenium can be set to wait for an element given its locator.

Web Scraping Using Selenium - Explicit Wait for Presence of Any Elements

Selenium Wait - Explicit Wait For Presence of Any Elements

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for presence of any elements given a locator. This is a way to check if the elements already load.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Presence of Any Elements Given a Locator

Create a file seleniumwaitpresenceall.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex).

Add this function as is

def wait_for_the_elements(wait_time: int, el_name: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), el_name))
        WebDriverWait(driver, wait_time).until(
            EC.presence_of_all_elements_located((By.TAG_NAME, el_name))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for the any element given a locator to load at a given waiting time. It will print a message if any element loaded or not.

Add this line
```
wait_for_the_elements(3, "nav")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_elements(6, "table")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_elements(9, "ul")
```
Again will call the method with a non existing element this time it is 9 seconds.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitpresenceall.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 01:02:57.360303] Finding element nav
[2020-06-01 01:02:57.375942] Element found
[2020-06-01 01:02:57.375942] Finding element table
[2020-06-01 01:03:03.456108] Element did not load
[2020-06-01 01:03:03.456108] Finding element ul
[2020-06-01 01:03:12.497395] Element did not load

Output explanations

The code looks for any element given the tag nav
The code found the element
The code looks for any element given the tag table
The code does not find the element within 6 seconds
The code looks for any element given the tag ul
The code does not find the element within 9 seconds

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")

def wait_for_the_elements(wait_time: int, el_name: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), el_name))
        WebDriverWait(driver, wait_time).until(
            EC.presence_of_all_elements_located((By.TAG_NAME, el_name))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_elements(3, "nav")
wait_for_the_elements(6, "table")
wait_for_the_elements(9, "ul")
driver.close()

Conclusion

Waiting time in selenium can be set to wait for any element given a locator.

Web Scraping Using Selenium - Explicit Wait for Title Contains Words

Selenium Wait - Explicit Wait For Title Contains

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for title.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Title That Contains Words

Create a file seleniumwaittitlecontains.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex).

Add this function as is

def wait_for_the_title(wait_time: int, doc_title: str):
    try:
        print("[{}] Waiting for title {}".format(str(datetime.now()), doc_title))
        WebDriverWait(driver, wait_time).until(EC.title_contains(doc_title))
        print("[{}] Title found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Error waiting for title".format(str(datetime.now())))

This method will wait for the a document title that contains a word or a phrase to load at a given waiting time. It will print a message if the document title loaded loaded or not.

Add this line
```
wait_for_the_title(3, "Do It Simpler")
```
This line will call the method we created and will display Title found.
Add this line
```
wait_for_the_title(6, "Not the title")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_title(9, "Another wrong title")
```
Again will call the method with a non existing element this time it is 9 seconds.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaittitlecontains.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 00:13:13.537620] Waiting for title Do It Simpler
[2020-06-01 00:13:13.553244] Title found
[2020-06-01 00:13:13.553244] Waiting for title Not the title
[2020-06-01 00:13:19.581967] Error waiting for title
[2020-06-01 00:13:19.581967] Waiting for title Another wrong title
[2020-06-01 00:13:28.635654] Error waiting for title

Output explanations

The code looks for the title that contains Do It Simpler
The code found the title
The code looks for the title that contains Not the title
The code does not find the title within 6 seconds
The code looks for the title that contains Another wrong title
The code does not find the title within 9 seconds

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")

def wait_for_the_title(wait_time: int, doc_title: str):
    try:
        print("[{}] Waiting for title {}".format(str(datetime.now()), doc_title))
        WebDriverWait(driver, wait_time).until(EC.title_contains(doc_title))
        print("[{}] Title found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Error waiting for title".format(str(datetime.now())))

wait_for_the_title(3, "Do It Simpler")
wait_for_the_title(6, "Not the title")
wait_for_the_title(9, "Another wrong title")
driver.close()

Conclusion

Waiting time in selenium can be set to wait for a title that contains words or phrase.