Basic Programming - Do It Simpler: firefox

Showing posts with label firefox. Show all posts

Friday, June 12, 2020

Web Scraping Using Selenium - Explicit Wait For Element Selection Using Locator

Selenium Wait - Explicit Wait For Element Selection Using Locator

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is to be selected using locator.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Selection Using Locator

Create a file seleniumwaitselection.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_selection(wait_time: int, selector: str, is_selected: bool):
    try:
        print("[{}] Waiting element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.element_located_selection_state_to_be((By.CSS_SELECTOR, selector), is_selected)
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not found".format(str(datetime.now())))

This method will wait for the element to be checked. It will print a message if the element did loaded or not.

Add this line
```
wait_for_element_selection(3, "div#elementSelection input[name='forSelected']", True)
```
This line will call the method we created and will wait within 3 seconds until it displays Element found. True parameter means that the checkbox must be checked.
Add this line
```
wait_for_element_selection(6, "div#elementSelection input[name='forSelected']", False)
```
This line will call the method we created and will wait within 6 seconds until it displays Element not found. False parameter means that the checkbox must NOT be checked.
Add this line
```
wait_for_element_selection(9, "div#elementSelection input[name='notSelected']", True)
```
This line will call the method we created and will wait within 9 seconds until it displays Element not found. True parameter means that the checkbox must NOT be checked.
Add this line
```
wait_for_element_selection(12, "div#elementSelection input[name='notSelected']", False)
```
This line will call the method we created and will wait within 12 seconds until it displays Element found. False parameter means that the checkbox must be checked.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitselection.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 4 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 23:19:12.181093] Waiting element div#elementSelection input[name='forSelected']
[2020-06-12 23:19:12.250153] Element found
[2020-06-12 23:19:12.250153] Waiting element div#elementSelection input[name='forSelected']
[2020-06-12 23:19:18.562138] Element not found
[2020-06-12 23:19:18.562138] Waiting element div#elementSelection input[name='notSelected']
[2020-06-12 23:19:27.931689] Element not found
[2020-06-12 23:19:27.931689] Waiting element div#elementSelection input[name='notSelected']
[2020-06-12 23:19:27.947199] Element found

Output explanations

The code waits for a given the CSS selector div#elementSelection input[name='forSelected'] that is checked.
The code found the a checkbox is checked
The code waits for a given the CSS selector div#elementSelection input[name='forSelected'] that is NOT checked.
The code did not find the a checkbox is not checked within 6 seconds
The code waits for a given the CSS selector div#elementSelection input[name='notSelected'] that is checked.
The code did not find the a checkbox is checked
The code waits for a given the CSS selector div#elementSelection input[name='notSelected'] that is NOT checked.
The code found the checkbox is NOT checked within 12 seconds

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_selection(wait_time: int, selector: str, is_selected: bool):
    try:
        print("[{}] Waiting element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.element_located_selection_state_to_be((By.CSS_SELECTOR, selector), is_selected)
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not found".format(str(datetime.now())))

wait_for_element_selection(3, "div#elementSelection input[name='forSelected']", True)
wait_for_element_selection(6, "div#elementSelection input[name='forSelected']", False)
wait_for_element_selection(9, "div#elementSelection input[name='notSelected']", True)
wait_for_element_selection(12, "div#elementSelection input[name='notSelected']", False)
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a checked element using locator.

Web Scraping Using Selenium - Explicit Wait For Element To Be Selected Using Locator

Selenium Wait - Explicit Wait For Element To Be Selected Using Locator

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is to be selected using locator.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Selected Using Locator

Create a file seleniumwaitselected.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_selected(wait_time: int, selector: str):
    try:
        print("[{}] Waiting element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.element_located_to_be_selected((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element selected".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not selected".format(str(datetime.now())))

This method will wait for the element to be selected. It will print a message if the element did loaded or not.

Add this line
```
wait_for_element_selected(3, "div#elementSelected input[name='forSelectedRadio']")
```
This line will call the method we created and will wait within 3 seconds until it displays Element selected.
Add this line
```
wait_for_element_selected(6, "div#elementSelected input[name='notSelectedRadio']")
```
Again will call the method we created and wait for 6 seconds until it gives a message Element not selected.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitselected.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 2 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 23:08:18.550321] Waiting element div#elementSelected input[name='forSelectedRadio']
[2020-06-12 23:08:18.612853] Element selected
[2020-06-12 23:08:18.612853] Waiting element div#elementSelected input[name='notSelectedRadio']
[2020-06-12 23:08:24.855407] Element not selected

Output explanations

The code looks for an element given the CSS selector div#elementSelected input[name='forSelectedRadio'] that is selected
The code found the a selected radio button
The code looks for an element given the CSS selector div#elementSelected input[name='notSelectedRadio'] that is selected
The code does not find the element within 6 seconds because the radio button is not selected

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_selected(wait_time: int, selector: str):
    try:
        print("[{}] Waiting element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.element_located_to_be_selected((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element selected".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not selected".format(str(datetime.now())))

wait_for_element_selected(3, "div#elementSelected input[name='forSelectedRadio']")
wait_for_element_selected(6, "div#elementSelected input[name='notSelectedRadio']")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a selected element using locator.

Web Scraping Using Selenium - Explicit Wait For Element Selection

Selenium Wait - Explicit Wait For Element Selection

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is to be selected.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Selection

Create a file seleniumwaitelementselection.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_selection(wait_time: int, element, is_selected: bool):
    try:
        print("[{}] Waiting element".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_selection_state_to_be(element, is_selected)
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not found".format(str(datetime.now())))

This method will wait for the element to be selected. It will print a message if the element did loaded or not.

Add these lines

check_el = driver.find_element_by_css_selector("div#elementSelection input[name='forSelected']")
uncheck_el = driver.find_element_by_css_selector("div#elementSelection input[name='notSelected']")

This lines will find the checkbox that we will use.

Add this line
```
wait_for_element_selection(3, check_el, True)
```
This line will call the method we created and will wait within 3 seconds until it displays Element found. True parameter means that the checkbox must be checked.
Add this line
```
wait_for_element_selection(6, check_el, False)
```
This line will call the method we created and will wait within 6 seconds until it displays Element not found. False parameter means that the checkbox must NOT be checked.
Add this line
```
wait_for_element_selection(9, uncheck_el, True)
```
This line will call the method we created and will wait within 9 seconds until it displays Element not found. True parameter means that the checkbox must NOT be checked.
Add this line
```
wait_for_element_selection(12, uncheck_el, False)
```
This line will call the method we created and will wait within 12 seconds until it displays Element found. False parameter means that the checkbox must be checked.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitelementselection.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Finds 2 checkboxes and put them in variables
- Call the Method 4 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 23:28:01.282843] Waiting element
[2020-06-12 23:28:01.336237] Element found
[2020-06-12 23:28:01.336237] Waiting element
[2020-06-12 23:28:07.710786] Element not found
[2020-06-12 23:28:07.710786] Waiting element
[2020-06-12 23:28:16.846971] Element not found
[2020-06-12 23:28:16.846971] Waiting element
[2020-06-12 23:28:16.846971] Element found

Output explanations

The code waits for a checkbox element which must be checked.
The code found a checkbox is checked
The code waits for a checkbox element which must be NOT checked.
The code did not find the checkbox within 6 seconds because it is checked.
The code waits for a checkbox element which must be checked.
The code did not find the checkbox within 6 seconds because it is NOT checked.
The code waits for a checkbox element which must be NOT checked.
The code found the checkbox that is NOT checked within 12 seconds

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_selection(wait_time: int, element, is_selected: bool):
    try:
        print("[{}] Waiting element".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_selection_state_to_be(element, is_selected)
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not found".format(str(datetime.now())))

check_el = driver.find_element_by_css_selector("div#elementSelection input[name='forSelected']")
uncheck_el = driver.find_element_by_css_selector("div#elementSelection input[name='notSelected']")

wait_for_element_selection(3, check_el, True)
wait_for_element_selection(6, check_el, False)
wait_for_element_selection(9, uncheck_el, True)
wait_for_element_selection(12, uncheck_el, False)
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a selected element.

Web Scraping Using Selenium - Explicit Wait For Element To Be Selected

Selenium Wait - Explicit Wait For Element To Be Selected

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is to be selected.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Selected

Create a file seleniumwaitelementselected.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_selected(wait_time: int, element):
    try:
        print("[{}] Waiting element".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_to_be_selected(element)
        )
        print("[{}] Element selected".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not selected".format(str(datetime.now())))

This method will wait for the element to be selected. It will print a message if the element did loaded or not.

Add these lines

check_el = driver.find_element_by_css_selector("div#elementSelected input[name='forSelectedRadio']")
uncheck_el = driver.find_element_by_css_selector("div#elementSelected input[name='notSelectedRadio']")

This lines will find the radio buttons that we will use.

Add this line
```
wait_for_element_selected(3, check_el)
```
This line will call the method we created and will wait within 3 seconds until it displays Element selected.
Add this line
```
wait_for_element_selected(6, uncheck_el)
```
Again will call the method we created and wait for 6 seconds until it gives a message Element not selected.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitelementselected.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Finds 2 radio buttons and put them in variables
- Call the Method 2 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 21:29:41.509204] Waiting element
[2020-06-12 21:29:41.527520] Element selected
[2020-06-12 21:29:41.527520] Waiting element
[2020-06-12 21:29:47.616751] Element not selected

Output explanations

The code waits for an selected element.
The code found the a selected radio button
The code looks for another selected element
The code does not find the element within 6 seconds because the radio button is not selected

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_selected(wait_time: int, element):
    try:
        print("[{}] Waiting element".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_to_be_selected(element)
        )
        print("[{}] Element selected".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not selected".format(str(datetime.now())))

check_el = driver.find_element_by_css_selector("div#elementSelected input[name='forSelectedRadio']")
uncheck_el = driver.find_element_by_css_selector("div#elementSelected input[name='notSelectedRadio']")

wait_for_element_selected(3, check_el)
wait_for_element_selected(6, uncheck_el)
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a selected element.

Web Scraping Using Selenium - Explicit Wait For Clickable Element

Selenium Wait - Explicit Wait For Clickable Element

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if the element is clickable.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Clickable

Create a file seleniumwaitclickable.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_element_clickable(wait_time: int, id: str):
    try:
        print("[{}] Waiting clickable".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_to_be_clickable((By.ID, id))
        )
        print("[{}] Element clickable".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not clickable".format(str(datetime.now())))

This method will wait for the element to be clickable. It will print a message if the element did loaded or not.

Add this line
```
wait_for_element_clickable(3, "clickableBtn")
```
This line will call the method we created and will wait within 3 seconds until it displays Element clickable.
Add this line
```
wait_for_element_clickable(6, "unclickableBtn")
```
Again will call the method we created and wait for 6 seconds until it gives a message Element not clickable.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitclickable.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 2 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 21:17:02.989380] Waiting clickable
[2020-06-12 21:17:06.081625] Element not clickable
[2020-06-12 21:17:06.081625] Waiting clickable
[2020-06-12 21:17:12.254618] Element not clickable

Output explanations

The code waits for an clickable element.
The code found the a clickable button
The code looks for another clickable element
The code does not find the element within 6 seconds because the button is not clickable cause it is disabled

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_element_clickable(wait_time: int, id: str):
    try:
        print("[{}] Waiting clickable".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(
            EC.element_to_be_clickable((By.ID, id))
        )
        print("[{}] Element clickable".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element not clickable".format(str(datetime.now())))

wait_for_element_clickable(3, "clickableBtn")
wait_for_element_clickable(6, "unclickableBtn")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting a clickable element.

Web Scraping Using Selenium - Explicit Wait For Alert

Selenium Wait - Explicit Wait For Alert

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting if there is an alert popup on the page.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Alert

Create a file seleniumwaitalert.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_alert(wait_time: int):
    try:
        print("[{}] Waiting for alert".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(EC.alert_is_present())
        print("[{}] Alert found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Alert did not load".format(str(datetime.now())))

This method will wait for the a popup alert. It will print a message if the alert loaded or not.

Add this line
```
wait_for_alert(3)
```
This line will call the method we created and will wait for 3 seconds until it displays Alert did not load.
Add this line
```
driver.find_element_by_css_selector("#alertExpectation button").click()
```
This line will call the find a button and click it to trigger an alert.
Add this line
```
wait_for_alert(6)
```
Again will call the method we created and wait for 6 seconds until it gives a message Alert found.
Add this line
```
driver.quit()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitalert.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 1 time which prints messages in the console
- Finds a button and clicks the button
- Call the Method 1 time which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-12 17:43:25.998479] Waiting for alert
[2020-06-12 17:43:29.005027] Alert did not load
[2020-06-12 17:43:29.089615] Waiting for alert
[2020-06-12 17:43:29.105222] Alert found

Output explanations

The code waits for an alert.
The code did not find any alert
The code again waits for an alert
The code finds the alert because a button was clicked that opens an alert popup

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_alert(wait_time: int):
    try:
        print("[{}] Waiting for alert".format(str(datetime.now())))
        WebDriverWait(driver, wait_time).until(EC.alert_is_present())
        print("[{}] Alert found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Alert did not load".format(str(datetime.now())))

wait_for_alert(3)
driver.find_element_by_css_selector("#alertExpectation button").click()
wait_for_alert(6)
driver.quit()

Conclusion

Waiting time in selenium can be used for waiting an alert.

Thursday, June 4, 2020

Web Scraping Using Selenium - Explicit Wait For Element Text Value

Selenium Wait - Explicit Wait For Element Text Value

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting a text from value attribute of an element.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For a Element Text

Create a file seleniumwaittextvalue.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_text(wait_time: int, selector: str, text_value: str):
    try:
        print("[{}] Finding text {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.text_to_be_present_in_element_value((By.CSS_SELECTOR, selector), text_value)
        )
        print("[{}] Text found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Text did not load".format(str(datetime.now())))

This method will wait for the elements value attribute text. It will print a message if the element text loaded or not.

Add this line
```
wait_for_text(3, "div#textExpectation > div:nth-of-type(1)", "This is an input")
```
This line will call the method we created and wait for 3 seconds until it gives an error.

Add this line

wait_for_text(6, "div#textExpectation > div:nth-of-type(2) > input", "This is an input")

This line will call the method we created and will display Text found.

Add this line

wait_for_text(9, "div#textExpectation > div:nth-of-type(1) > input", "this is an input")

Again will call the method we created and wait for 9 seconds until it gives an error.

<div id="textExpectation">
    <div>Expected Text</div>
    <div><input type="text" value="This is an input"></div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaittextvalue.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-04 23:09:35.289359] Finding text div#textExpectation > div:nth-of-type(1)
[2020-06-04 23:09:38.512822] Text did not load
[2020-06-04 23:09:38.512822] Finding text div#textExpectation > div:nth-of-type(2) > input
[2020-06-04 23:09:38.543957] Text found
[2020-06-04 23:09:38.543957] Finding text div#textExpectation > div:nth-of-type(1) > input
[2020-06-04 23:09:47.588846] Text did not load

Output explanations

The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(1) and checks the text for that element
The code does not find the element within 3 seconds
The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(2) and checks the text for that element
The code found the text
The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(1) checks the text for that element
The code does not find the element within 9 seconds
```
<div><input type="text" value="This is an input"></div>
```
The matching of the text is case sensitive so the text does not match.

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_text(wait_time: int, selector: str, text_value: str):
    try:
        print("[{}] Finding text {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.text_to_be_present_in_element_value((By.CSS_SELECTOR, selector), text_value)
        )
        print("[{}] Text found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Text did not load".format(str(datetime.now())))

wait_for_text(3, "div#textExpectation > div:nth-of-type(1)", "This is an input")
wait_for_text(6, "div#textExpectation > div:nth-of-type(2) > input", "This is an input")
wait_for_text(9, "div#textExpectation > div:nth-of-type(1) > input", "this is an input")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting elements value attribute text.

Web Scraping Using Selenium - Explicit Wait For Element Text

Selenium Wait - Explicit Wait For Element Text

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for waiting a text from innerHTML of an element.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For a Element Text

Create a file seleniumwaittext.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_text(wait_time: int, selector: str, text_value: str):
    try:
        print("[{}] Finding text {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.text_to_be_present_in_element((By.CSS_SELECTOR, selector), text_value)
        )
        print("[{}] Text found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Text did not load".format(str(datetime.now())))

This method will wait for the elements innerHTML text. It will print a message if the element text loaded or not.

Add this line
```
wait_for_text(3, "div#textExpectation > div:nth-of-type(1)", "Expected Text")
```
This line will call the method we created and will display Text found.
Add this line
```
wait_for_text(6, "div#textExpectation > div:nth-of-type(2)", "Expected Text"))
```
This line will call the method we created and wait for 6 seconds until it gives an error.

Add this line

wait_for_text(9, "div#textExpectation > div:nth-of-type(1)", "expected text")

Again will call the method we created and wait for 9 seconds until it gives an error.

<div id="textExpectation">
    <div>Expected Text</div>
    <div><input type="text" value="This is an input"></div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaittext.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex#/seleniumwait
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-04 13:44:09.697415] Finding text div#textExpectation > div:nth-of-type(1)
[2020-06-04 13:44:09.782072] Text found
[2020-06-04 13:44:09.782072] Finding text div#textExpectation > div:nth-of-type(2)
[2020-06-04 13:44:16.139792] Text did not load
[2020-06-04 13:44:16.139792] Finding text div#textExpectation > div:nth-of-type(1)
[2020-06-04 13:44:25.508062] Text did not load

Output explanations

The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(1) and checks the text for that element
The code found the text
The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(2) and checks the text for that element
The code does not find the element within 6 seconds
The code looks for an element given the CSS selector div#textExpectation > div:nth-of-type(1) checks the text for that element
The code does not find the element within 9 seconds
```
<div>Expected Text</div>
```
The matching of the text is case sensitive so the text does not match.

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_text(wait_time: int, selector: str, text_value: str):
    try:
        print("[{}] Finding text {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.text_to_be_present_in_element((By.CSS_SELECTOR, selector), text_value)
        )
        print("[{}] Text found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Text did not load".format(str(datetime.now())))

wait_for_text(3, "div#textExpectation > div:nth-of-type(1)", "Expected Text")
wait_for_text(6, "div#textExpectation > div:nth-of-type(2)", "Expected Text")
wait_for_text(9, "div#textExpectation > div:nth-of-type(1)", "expected text")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting elements innerHTML text.

Monday, June 1, 2020

Web Scraping Using Selenium - Explicit Wait For Visibility of All Elements

Selenium Wait - Explicit Wait For Visibility of All Elements

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for visibility of elements. All elements for a locator must be visible (can be seen or has at least 1px width).

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Visibility of Any Element Given Its CSS Selector

Create a file seleniumwaitvisibilityall.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_all_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for any elements given its selector to become visible given the waiting time. It will print a message if the element is visible or not.

Add this line
```
wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
```
Again will call the method with a non visible element this time it is 9 seconds.

Add this line

wait_for_the_elements(12, "div#hiddenElements > div")

Again will call the method with a non visible and visible elements this time it is 12 seconds.

<div id="hiddenElements">
    <div style="display: none;">This is a hidden element</div>
    <div style="width: 0px; height: 0px;"></div>
    <div>Visible div</div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitvisibilityall.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 4 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 23:41:56.278252] Finding element div#hiddenElements > div:nth-of-type(3)
[2020-06-01 23:41:56.331652] Element found
[2020-06-01 23:41:56.331652] Finding element div#hiddenElements > div:nth-of-type(1)
[2020-06-01 23:42:02.804026] Element did not load
[2020-06-01 23:42:02.804026] Finding element div#hiddenElements > div:nth-of-type(2)
[2020-06-01 23:42:12.094494] Element did not load
[2020-06-01 23:42:12.094494] Finding element div#hiddenElements > div
[2020-06-01 23:42:24.440167] Element did not load

Output explanations

The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(3) and checks visibility
The code found the element
The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(1) and checks visibility
The code does not find the element within 6 seconds
The code looks for all elements given the CSS selector div#hiddenElements > div:nth-of-type(2) and checks visibility
The code does not find the element within 9 seconds
The code looks for all elements given the CSS selector div#hiddenElements > div and checks visibility
The code does not find the element within 12 seconds becuase only 1 div is visible

<div style="display: none;">This is a hidden element</div>
<div style="width: 0px; height: 0px;"></div>
<div>Visible div</div>

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_all_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
wait_for_the_elements(12, "div#hiddenElements > div")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting all element's visibility via locators.

Web Scraping Using Selenium - Explicit Wait For Visibility of Any Element

Selenium Wait - Explicit Wait For Visibility of Any Element

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for visibility of elements. Any elements for a locator must be visible (can be seen or has at least 1px width).

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Visibility of Any Element Given Its CSS Selector

Create a file seleniumwaitvisibilityany.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_any_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for any elements given its selector to become visible given the waiting time. It will print a message if the element is visible or not.

Add this line
```
wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
```
Again will call the method with a non visible element this time it is 9 seconds.

Add this line

wait_for_the_elements(12, "div#hiddenElements > div")

Again will call the method with a non visible and visible elements this time it is 12 seconds.

<div id="hiddenElements">
    <div style="display: none;">This is a hidden element</div>
    <div style="width: 0px; height: 0px;"></div>
    <div>Visible div</div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitvisibilityany.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 4 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 23:25:27.525543] Finding element div#hiddenElements > div:nth-of-type(3)
[2020-06-01 23:25:27.594599] Element found
[2020-06-01 23:25:27.594599] Finding element div#hiddenElements > div:nth-of-type(1)
[2020-06-01 23:25:33.604729] Element did not load
[2020-06-01 23:25:33.604729] Finding element div#hiddenElements > div:nth-of-type(2)
[2020-06-01 23:25:42.787525] Element did not load
[2020-06-01 23:25:42.787525] Finding element div#hiddenElements > div
[2020-06-01 23:25:42.891207] Element found

Output explanations

The code looks for any element given the CSS selector div#hiddenElements > div:nth-of-type(3) and checks visibility
The code found the element
The code looks for any element given the CSS selector div#hiddenElements > div:nth-of-type(1) and checks visibility
The code does not find the element within 6 seconds
The code looks for any element given the CSS selector div#hiddenElements > div:nth-of-type(2) and checks visibility
The code does not find the element within 9 seconds
The code looks for any element given the CSS selector div#hiddenElements > div and checks visibility

The code found the a visible element within 12 seconds because the third div found is visible

<div style="display: none;">This is a hidden element</div>
<div style="width: 0px; height: 0px;"></div>
<div>Visible div</div>

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_the_elements(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_any_elements_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_elements(3, "div#hiddenElements > div:nth-of-type(3)")
wait_for_the_elements(6, "div#hiddenElements > div:nth-of-type(1)")
wait_for_the_elements(9, "div#hiddenElements > div:nth-of-type(2)")
wait_for_the_elements(12, "div#hiddenElements > div")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting any element's visibility via locators.

Web Scraping Using Selenium - Explicit Wait For Visibility of An Element

Selenium Wait - Explicit Wait For Visibility of An Element

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for visibility of elements. The element must be visible (can be seen or has at least 1px width).

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Visibility of An Element Given Its CSS Selector

Create a file seleniumwaitvisibilityone.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex#/seleniumwait).

Add this function as is

def wait_for_the_element(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_element_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for the element given its selector to become visible given the waiting time. It will print a message if the element is visible or not.

Add this line
```
wait_for_the_element(3, "div#hiddenElements > div:nth-of-type(3)")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_element(6, "div#hiddenElements > div:nth-of-type(1)")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_element(9, "div#hiddenElements > div:nth-of-type(2)")
```
Again will call the method with a non visible element this time it is 9 seconds.

Add this line

wait_for_the_element(12, "div#hiddenElements > div")

Again will call the method with a non visible element this time it is 12 seconds. The selector finds the first div.

<div id="hiddenElements">
    <div style="display: none;">This is a hidden element</div>
    <div style="width: 0px; height: 0px;"></div>
    <div>Visible div</div>
</div>

The above HTML is taken from the website.

Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaitvisibilityone.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 4 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-06-01 22:58:40.336003] Finding element div#hiddenElements > div:nth-of-type(3)
[2020-06-01 22:58:40.414464] Element found
[2020-06-01 22:58:40.414464] Finding element div#hiddenElements > div:nth-of-type(1)
[2020-06-01 22:58:46.421484] Element did not load
[2020-06-01 22:58:46.421484] Finding element div#hiddenElements > div:nth-of-type(2)
[2020-06-01 22:58:55.767107] Element did not load
[2020-06-01 22:58:55.767107] Finding element div#hiddenElements > div
[2020-06-01 22:59:08.054460] Element did not load

Output explanations

The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(3) and checks visibility
The code found the element
The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(1) and checks visibility
The code does not find the element within 6 seconds
The code looks for an element given the CSS selector div#hiddenElements > div:nth-of-type(2) and checks visibility
The code does not find the element within 9 seconds
The code looks for an element given the CSS selector div#hiddenElements > div and checks visibility
The code does not find the element within 12 seconds because the first div found is not visible
```
<div style="display: none;">This is a hidden element</div>
```
This is not the same is wait by visibility_of because the element does not need ti be present.

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex#/seleniumwait")

def wait_for_the_element(wait_time: int, selector: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), selector))
        WebDriverWait(driver, wait_time).until(
            EC.visibility_of_element_located((By.CSS_SELECTOR, selector))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_element(3, "div#hiddenElements > div:nth-of-type(3)")
wait_for_the_element(6, "div#hiddenElements > div:nth-of-type(1)")
wait_for_the_element(9, "div#hiddenElements > div:nth-of-type(2)")
wait_for_the_element(12, "div#hiddenElements > div")
driver.close()

Conclusion

Waiting time in selenium can be used for waiting an element visibility without checking its presence.

Sunday, May 31, 2020

Web Scraping Using Selenium - Navigation Using Browser History

Selenium Navigation - History

Navigations in selenium can be done in different ways. In this tutorial, we will use the back and forward button of the browser as a form of navigation.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Navigation History - Back and Forward

Create a file seleniumnavhistory.py and paste the following codes
```
from selenium import webdriver
import time
```
The codes above imports the required library that we will use.
Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will execute a script to go to the webpage https://slackingslacker.github.io/seleniumindex.
Add this line
```
time.sleep(3)
```
We will pause the program for 3 seconds.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex#/about")
```
The line will execute a script to go to the webpage https://slackingslacker.github.io/seleniumindex#/about.
Add this line
```
time.sleep(3)
```
We will pause the program for 3 seconds. At this time we have history in the browser.
Add this line
```
driver.back()
```
The line will go back to the previous page which is https://slackingslacker.github.io/seleniumindex.
Add this line
```
time.sleep(10)
```
We will pause the program for 10 seconds. You may have noticed that the current page is the main page.
Add this line
```
driver.forward()
```
The line will go back to the next page which is https://slackingslacker.github.io/seleniumindex#/about.
Add this line
```
time.sleep(10)
```
We will pause the program for 10 seconds. You may have noticed that the current page is the about page.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumnavhistory.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Halts for 3 seconds
- Browser goes to https://slackingslacker.github.io/seleniumindex#/about
- Halts for 3 seconds
- Browser goes to https://slackingslacker.github.io/seleniumindex using the back of history
- Halts for 10 seconds
- Browser goes to https://slackingslacker.github.io/seleniumindex#/about using the forward of history
- Halts for 10 seconds
- Closes the browser

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")
time.sleep(3)
driver.get("https://slackingslacker.github.io/seleniumindex#/about")
time.sleep(3)

driver.back()
time.sleep(10)

driver.forward()
time.sleep(10)

driver.close()

Conclusion

Navigation in selenium can be done using browser history.

Web Scraping Using Selenium - Navigation with Anchor Element

Selenium Navigation - Anchor Navigation

Navigations in selenium can be done in different ways. In this tutorial, we will use the click functionality of an anchor tag.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Navigation Using Anchor Element

Create a file seleniumnavachor.py and paste the following codes
```
from selenium import webdriver
import time
```
The codes above imports the required library that we will use.
Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex).
Add this line
```
time.sleep(5)
```
We will pause the program for 5 seconds.
Add this line
```
el = driver.find_element_by_css_selector("div[class='navbar-start'] > a:last-of-type")
```
This code will find an element using a CSS selector. We will tackle this more in the future. The element that we are looking for is the About link in the Menu Bar at the top
Add this line
```
el.click()
```
This will click the About in the menu.
Add this line
```
time.sleep(10)
```
We will pause the program for 10 seconds to see that it loaded the about page.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumnavachor.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Halts for 5 seconds
- Finds the anchor tag for the About in the top menu
- Clicks the link
- Halts for 10 seconds
- Closes the browser

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")
time.sleep(5)

el = driver.find_element_by_css_selector("div[class='navbar-start'] > a:last-of-type")
el.click()
time.sleep(10)

driver.close()

Conclusion

Navigation in selenium can be done using an anchor element.

Tuesday, May 26, 2020

Web Scraping Using Selenium - Explicit Wait for Title Element

Selenium Wait - Explicit Wait For Title

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality for title. Good example is loading a blog or a product page with a specific title.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait For Specific Title

Create a file seleniumwaittitle.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex).

Add this function as is

def wait_for_the_title(wait_time: int, doc_title: str):
    try:
        print("[{}] Waiting for title {}".format(str(datetime.now()), doc_title))
        WebDriverWait(driver, wait_time).until(EC.title_is(doc_title))
        print("[{}] Title found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Error waiting for title".format(str(datetime.now())))

This method will wait for the an specific document title to load at a given waiting time. It will print a message if the document title loaded loaded or not.

Add this line
```
wait_for_the_title(3, "Do It Simpler - VUE - Bulma For Scraping")
```
This line will call the method we created and will display Title found.
Add this line
```
wait_for_the_title(6, "Not the title")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_title(9, "Another wrong title")
```
Again will call the method with a non existing element this time it is 9 seconds.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumwaittitle.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-05-31 23:58:37.029019] Waiting for title Do It Simpler - VUE - Bulma For Scraping
[2020-05-31 23:58:37.044668] Title found
[2020-05-31 23:58:37.044668] Waiting for title Not the title
[2020-05-31 23:58:43.274813] Error waiting for title
[2020-05-31 23:58:43.274813] Waiting for title Another wrong title
[2020-05-31 23:58:52.510780] Error waiting for title

Output explanations

The code looks for the title that is equal to Do It Simpler - VUE - Bulma For Scraping
The code found the title
The code looks for the title that is equal to Not the title
The code does not find the title within 6 seconds
The code looks for the title that is equal to Another wrong title
The code does not find the title within 9 seconds

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")

def wait_for_the_title(wait_time: int, doc_title: str):
    try:
        print("[{}] Waiting for title {}".format(str(datetime.now()), doc_title))
        WebDriverWait(driver, wait_time).until(EC.title_is(doc_title))
        print("[{}] Title found".format(str(datetime.now())))
    except TimeoutException as e:
        print("[{}] Error waiting for title".format(str(datetime.now())))

wait_for_the_title(3, "Do It Simpler - VUE - Bulma For Scraping")
wait_for_the_title(6, "Not the title")
wait_for_the_title(9, "Another wrong title")
driver.close()

Conclusion

Waiting time in selenium can be set to wait for a specific title.

Monday, May 25, 2020

Web Scraping Using Selenium - Explicit Wait

Selenium Wait - Explicit Wait

Waiting in selenium can be done in different ways. In this tutorial, we will use the explicit wait functionality.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Explicit Wait

Create a file seleniumexplicitwait.py and paste the following codes

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from datetime import datetime

The codes above imports the required library that we will use.

Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex).

Add this function as is

def wait_for_the_element(wait_time: int, el_id: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), el_id))
        WebDriverWait(driver, wait_time).until(
            EC.presence_of_element_located((By.ID, el_id))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except Exception as e:
        print("[{}] Element did not load".format(str(datetime.now())))

This method will wait for the element to load at a given waiting time. It will print a message if the element loaded or not.

Add this line
```
wait_for_the_element(3, "navMenuId")
```
This line will call the method we created and will display Element found.
Add this line
```
wait_for_the_element(6, "noneExistentId")
```
This line will call the method we created and wait for 6 seconds until it gives an error.
Add this line
```
wait_for_the_element(9, "anotherNoneExistentId")
```
Again will call the method with a non existing element this time it is 9 seconds.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumexplicitwait.py. It should do the following:
- Open the firefox browser
- Browser goes to https://slackingslacker.github.io/seleniuminde
- Call the Method 3 times which prints messages in the console
- Closes the browser

Program Sample Output

[2020-05-31 23:55:38.889027] Finding element navMenuId
[2020-05-31 23:55:38.920301] Element found
[2020-05-31 23:55:38.920301] Finding element noneExistentId
[2020-05-31 23:55:44.931274] Element did not load
[2020-05-31 23:55:44.931274] Finding element anotherNoneExistentId
[2020-05-31 23:55:53.968452] Element did not load

Output explanations

The code looks for an element with id navMenuId
The code found the element
The code looks for an element with id noneExistentId
The code does not find the element within 6 seconds
The code looks for an element with id anotherNoneExistentId
The code does not find the element within 9 seconds

As you may have noticed that the waiting time varies to what you have supplied to the WebDriverWait class.

Final Selenium Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from datetime import datetime

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")

def wait_for_the_element(wait_time: int, el_id: str):
    try:
        print("[{}] Finding element {}".format(str(datetime.now()), el_id))
        WebDriverWait(driver, wait_time).until(
            EC.presence_of_element_located((By.ID, el_id))
        )
        print("[{}] Element found".format(str(datetime.now())))
    except Exception as e:
        print("[{}] Element did not load".format(str(datetime.now())))

wait_for_the_element(3, "navMenuId")
wait_for_the_element(6, "noneExistentId")
wait_for_the_element(9, "anotherNoneExistentId")
driver.close()

Conclusion

Waiting time in selenium can be set per element.

Saturday, May 23, 2020

Web Scraping Using Selenium - Navigation

Selenium Navigation - driver.get

Navigations in selenium can be done in different ways. In this tutorial, we will use the driver.get functionality.

But before that, please make sure you have read the first blog on this series to do the prerequisites.

Selenium Navigation Code Using driver.get

Create a file seleniumnav.py and paste the following codes
```
from selenium import webdriver
import time
```
The codes above imports the required library that we will use.
Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox.
Add this line
```
driver.get("https://www.google.com")
```
The line will got to the website (https://www.google.com).
Add this line
```
time.sleep(10)
```
We will pause the program for 10 seconds.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex). This is way to navigate to different pages of the site or different websites.
Add this line
```
time.sleep(10)
```
We will pause the program for 10 seconds.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumnav.py. It should do the following:
- Open the firefox browser
- Browser goes to www.google.com
- Halts for 10 seconds
- Browser goes to https://slackingslacker.github.io/seleniumindex
- Halts for 10 seconds
- Closes the browser

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://www.google.com")
time.sleep(10)

driver.get("https://slackingslacker.github.io/seleniumindex")
time.sleep(10)

driver.close()

Conclusion

Navigation from page to page or website to website can be done using driver.get functionality

Saturday, May 16, 2020

Web Scraping Using Selenium - Intro

Selenium Introduction

The purpose of these series is to use the different functionalities in the selenium documentation. Each functionality will have a different example in order for us to better understand when and where to use that functionality.

Background

Selenium is usually used for automated testing but can also be used for scraping websites. Since most modern websites are created as Single-Page Applications (SPA), the page is lazy loaded. It means that the basic structure like HTML and CSS are loaded first before the data were loaded. The data were loaded afterwards through API calls and Javascript. Scrapers such as requests library from Python (tutorials can be found here) or guzzle from PHP cannot directly interact with javascripts. There can be workarounds to handle those javascript interactions but sometimes it is a dead end. Selenium solves this kind of problems by interacting to the website using browser so it is just like a person controlling the website. Selenium is mostly used for testing website and also can be user for scraping.

Getting the Softwares required for Selenium

We will now start to code. But first lets make sure that we have the required.

Install python. You can download python here depending on your OS. The installation of python will depend on the OS that you are using
Install the required library, in this case requests.You can run the command.
```
pip install selenium
```
or
```
python -m pip install selenium
```
(Optional) You can download pycharm here as to make your coding faster. I will be using pycharm in doing these tutorials but you can also use notepad and command lines.
Download the browser drivers and paste it to the directory where it is accessible to the app. you can it paste the library later. I will be using firefox for most of the tutorials as the geckodriver and firefox are compatible even when the browser were updated.
- chrome driver download
- firefox driver download
(Optional) Install the chrome and firefox browsers.

Coding My First Selenium Program

Create a directory where you will put your codes
Copy the drivers that you have downloaded and paste them in the directory you've created.
Create a file seleniumintro.py and paste the following codes
```
from selenium import webdriver
import time
```
The codes above imports the required library that we will use.
Add this line
```
driver = webdriver.Firefox(executable_path="geckodriver.exe")
```
The code above will create a webdriver instance for Firefox
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex).
Add this line
```
time.sleep(5)
```
We will pause the program for 5 seconds. This is to ensure that you can see whats happening in the browser.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Add this line
```
driver = webdriver.Chrome(executable_path="chromedriver.exe")
```
This time we are going to use chrome browser to access the website.
Add this line
```
driver.get("https://slackingslacker.github.io/seleniumindex")
```
The line will got to the website (https://slackingslacker.github.io/seleniumindex) on the chrome.
Add this line
```
time.sleep(5)
```
We will again pause the program for 5 seconds.
Add this line
```
driver.close()
```
The line will close the webdriver as well as the browser.
Run the seleniumsimple.py.
```
python seleniumintro.py
```
or Run on pycharm

It should do the following:
1. Opens the firefox browser.(assuming you have firefox installed.)
2. Browser goes to the website https://slackingslacker.github.io/seleniumindex
3. Halts for 5 seconds
4. Close firefox browser
5. Opens the chrome browser (assuming you have chrome installed.)
6. Browser goes to the website https://slackingslacker.github.io/seleniumindex
7. Wait for 5 seconds
8. Close chrome browser

Final Selenium Code

from selenium import webdriver
import time

driver = webdriver.Firefox(executable_path="geckodriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")
time.sleep(5)
driver.close()

driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.get("https://slackingslacker.github.io/seleniumindex")
time.sleep(5)
driver.close()

Conclusion

Using selenium, we can open the browser and automatically use the functionalities of a website. With just a few lines of codes, we can easily use selenium.