I cannot able to webscrape rating percentage from amazon product page. I am getting only null values. Here is my code
from typing import Text
from bs4 import BeautifulSoup
import requests
import pandas as pd
from datetime import date
import os
url='https://www.amazon.in/dp/B09BJQCTMX?ref=myi_title_dp'
req = requests.get(url)
content=BeautifulSoup(req.content,"lxml")
data = content.findAll('a',class_='a-link-normal')
print(data)I have provided correct class name but only null values in retrieved.
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
Try to put headers inside request call and data will be present inside content
headers={"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"}
url='https://www.amazon.in/dp/B09BJQCTMX?ref=myi_title_dp'
req = requests.get(url,headers=headers)
content=BeautifulSoup(req.content,"lxml")Now use appropriate class to find data
text_data=content.find("span",class_="a-list-item").get_text(strip=True)
href_data=content.find("span",class_="a-list-item").find("a")['href']Image:
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0