soup.find("tagName", { "id" : "articlebody" })
Why does this NOT return the <div id="articlebody"> ... </div> tags and stuff in between? It returns nothing. And I know for a fact it exists because I’m staring right at it from
soup.prettify()
soup.find("div", { "id" : "articlebody" }) also does not work.
(EDIT: I found that BeautifulSoup wasn’t correctly parsing my page, which probably meant the page I was trying to parse isn’t properly formatted in SGML or whatever)
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
You should post your example document, because the code works fine:
>>> import BeautifulSoup
>>> soup = BeautifulSoup.BeautifulSoup('<html><body><div id="articlebody"> ... </div></body></html')
>>> soup.find("div", {"id": "articlebody"})
<div id="articlebody"> ... </div>
Finding <div>s inside <div>s works as well:
>>> soup = BeautifulSoup.BeautifulSoup('<html><body><div><div id="articlebody"> ... </div></div></body></html')
>>> soup.find("div", {"id": "articlebody"})
<div id="articlebody"> ... </div>
Method 2
To find an element by its id:
div = soup.find(id="articlebody")
Method 3
Beautiful Soup 4 supports most CSS selectors with the .select() method, therefore you can use an id selector such as:
soup.select('#articlebody')
If you need to specify the element’s type, you can add a type selector before the id selector:
soup.select('div#articlebody')
The .select() method will return a collection of elements, which means that it would return the same results as the following .find_all() method example:
soup.find_all('div', id="articlebody")
# or
soup.find_all(id="articlebody")
If you only want to select a single element, then you could just use the .find() method:
soup.find('div', id="articlebody")
# or
soup.find(id="articlebody")
Method 4
I think there is a problem when the ‘div’ tags are too much nested. I am trying to parse some contacts from a facebook html file, and the Beautifulsoup is not able to find tags “div” with class “fcontent”.
This happens with other classes as well. When I search for divs in general, it turns only those that are not so much nested.
The html source code can be any page from facebook of the friends list of a friend of you (not the one of your friends). If someone can test it and give some advice I would really appreciate it.
This is my code, where I just try to print the number of tags “div” with class “fcontent”:
from BeautifulSoup import BeautifulSoup
f = open('/Users/myUserName/Desktop/contacts.html')
soup = BeautifulSoup(f)
list = soup.findAll('div', attrs={'class':'fcontent'})
print len(list)
Method 5
Most probably because of the default beautifulsoup parser has problem. Change a different parser, like ‘lxml’ and try again.
Method 6
In the beautifulsoup source this line allows divs to be nested within divs; so your concern in lukas’ comment wouldn’t be valid.
NESTABLE_BLOCK_TAGS = ['blockquote', 'div', 'fieldset', 'ins', 'del']
What I think you need to do is to specify the attrs you want such as
source.find('div', attrs={'id':'articlebody'})
Method 7
have you tried soup.findAll("div", {"id": "articlebody"})?
sounds crazy, but if you’re scraping stuff from the wild, you can’t rule out multiple divs…
Method 8
I used:
soup.findAll('tag', attrs={'attrname':"attrvalue"})
As my syntax for find/findall; that said, unless there are other optional parameters between the tag and attribute list, this shouldn’t be different.
Method 9
Here is a code fragment
soup = BeautifulSoup(:"index.html")
titleList = soup.findAll('title')
divList = soup.findAll('div', attrs={ "class" : "article story"})
As you can see I find all tags and then I find all tags with class=”article” inside
Method 10
Happened to me also while trying to scrape Google.
I ended up using pyquery.
Install:
pip install pyquery
Use:
from pyquery import PyQuery
pq = PyQuery('<html><body><div id="articlebody"> ... </div></body></html')
tag = pq('div#articlebody')
Method 11
The Id property is always uniquely identified. That means you can use it directly without even specifying the element. Therefore, it is a plus point if your elements have it to parse through the content.
divEle = soup.find(id = "articlebody")
Method 12
from bs4 import BeautifulSoup
from requests_html import HTMLSession
url = 'your_url'
session = HTMLSession()
resp = session.get(url)
# if element with id "articlebody" is dynamic, else need not to render
resp.html.render()
soup = bs(resp.html.html, "lxml")
soup.find("div", {"id": "articlebody"})
Method 13
soup.find("tagName",attrs={ "id" : "articlebody" })
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0