pull data with HtmlAgilityPack Web Scraping c#

What could be the reason that it returns null in selectnodes?

HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("https://www.wired.com/most-popular/");
var headerNames = doc.DocumentNode.SelectNodes("//a[@class='archive-item-component__title']").ToList();
string listData = "";
foreach(var item in headerNames)
{
   listData += Environment.NewLine + item.InnerText;   
}
Console.WriteLine(listData);

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

The reason is that ‘archive-item-component__title’ class in the web site is specified for h2 tags, not for a tags.

Selecting your headerNames like this works:

doc.DocumentNode.SelectNodes("//h2[@class='archive-item-component__title']");

PS: you don’t need to call .ToList() on the SelectNodes results, you can loop through the HtmlNodeCollection as is.


All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

0 0 votes
Article Rating
Subscribe
Notify of
guest

0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x