Python – How to retrieve tables from the dynamic website Python Selenium

How to retrieve tables from the dynamic website Python Selenium… here is a solution to the problem.

How to retrieve tables from the dynamic website Python Selenium

I

want to retrieve all the information from the table of the dynamic website, I have the following code:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
import sys
reload(sys)
import re
import csv
from time import sleep
sys.setdefaultencoding('utf-8') #added since it would give error for certain values when using str(i)

chrome_options = webdriver. ChromeOptions()
chrome_options.add_argument('--headless')
prefs = {'profile.managed_default_content_settings.images':2}
chrome_options.add_experimental_option("prefs", prefs)
driver = webdriver. Chrome(chrome_options=chrome_options) 

maxcr = 1379
listofrows = []

url = "http://biggestbook.com/ui/catalog.html#/itemDetail?itemId=HERY4832YER01&uom=CT"
print(url) 
driver.get(url)
wait = WebDriverWait(driver,10)
# Trying to get the table 
tableloadwait = (wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".panel-body"))))
table = driver.find_elements_by_css_selector(".panel-body")
print(table)
RowsOfTable = table.get_attribute("tr")

However, I keep getting error messages, but so far it hasn’t worked. How do I retrieve information for a table?
Thank you very much!

Mistake:
RowsOfTable = table.get_attribute(“tr”)
The AttributeError: ‘list’ object does not have the property ‘get_attribute’

Solution

This is the code to get the product details

tableloadwait = (wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".panel-body"))))
driver.find_element_by_xpath("//span[contains(.,'Product Details')]").click()
rows = driver.find_elements_by_xpath("//span[contains(.,'Product Details')]/ancestor::div[@class='accordion-top-border']//tr[(@ng-repeat='attr in attributes' or @ng-repeat=' field in fields') and @class='visible-xs']")

for rowNum in range(len(rows)):
    print(rows[rowNum].get_attribute('innerText'))
driver.quit()

We have to trim or break the value according to your requirements.

If you want to get data based on line text, use the following.

upcData = driver.find_element_by_xpath("//strong[.='UPC']/parent::td").get_attribute('innerText').replace('UPC','').replace('\n','').replace('    ','')

Related Problems and Solutions