How to get rendered html (processed by Javascript) in WebBrowser control?

I have an ASP.NET page and some custom class that fetches a specified webpage and returns that page body back.

protected String GetHtml()
{
    Thread thread = new Thread(new ThreadStart(GetHtmlWorker));
    thread.SetApartmentState(ApartmentState.STA);
    thread.Start();
    thread.Join();
    return docHtml;
}

protected void GetHtmlWorker()
{
    using (WebBrowser browser = new WebBrowser())
    {
        browser.ScriptErrorsSuppressed = true;
        browser.Navigate(_url);
        // Wait for control to load page
        while (browser.ReadyState != WebBrowserReadyState.Complete)
            Application.DoEvents();
        docHtml = browser.DocumentText;
    }
}

But what I need is to get DOM HTML instead of the page source because I do some extra operations over DOM by jQuery.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

Here is one solution I found to get to the rendered HTML(DOM) after javascript was run:

Place a WebBrowser control named webBrowser1 on the Form of class Form1.
[Form1.cs[Design]]

Then for code use:
[Form1.cs]

using System;
using System.Runtime.InteropServices;
using System.Windows.Forms;

namespace WebBrowserTest
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
            this.webBrowser1.ObjectForScripting = new MyScript();
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            webBrowser1.Navigate("http://localhost:6489/Default.aspx");
        }

        private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            webBrowser1.Navigate("javascript: window.external.CallServerSideCode();");
        }

        [ComVisible(true)]
        public class MyScript
        {
            public void CallServerSideCode()
            {
                var doc = ((Form1)Application.OpenForms[0]).webBrowser1.Document;
            }
        }
    }
}

Change the webBrowser1.Navigate(“http://localhost:6489/Default.aspx“) parameter in Form1_Load to the page whose DOM after being processed by javascript you wish to obtain.

You can access the modified DOM in the CallServerSideCode() method, for example:

doc.GetElementById("myDataTable");

Or you can access the rendered HTML like this:

var renderedHtml = doc.GetElementsByTagName("HTML")[0].OuterHtml;

Method 2

As George said in one of the comments, in theory you can just get the DOM in webBrowser1_DocumentCompleted by just using:

webBrowser1.Document.GetElementsByTagName("HTML")[0].OuterHtml;

Method 3

First a little background. I have been trying to scrape information from a web page. The content of this webpage is dynamic. What I mean by dynamic is that the web page loads more information as you scroll down to the bottom of the page. The HTML content changes as you scroll to the bottom of the page. Unfortunately the Web Browser Object does not update this information automatically. It still has the original document that it first loaded via the webbrowser.navigate function. The updated information is available to the HTMLElementCollection.

The following code did not work for me.

webBrowser1.Document.GetElementsByTagName("HTML")[0].OuterHtml

I broke up the above statement as follows

    Dim eCollections As HtmlElementCollection
    Dim strDoc As String
    eCollections = WB.Document.GetElementsByTagName("HTML")
    strDoc = eCollections(0).OuterHtml

Worked like a charm. Hope this helps someone too.

Method 4

Another way would be to set a timer on the form, then when the timer hits, the page will have re-rendered and you can parse the page.

Method 5

You can get

webBrowser1.Document.Body.OuterHtml

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0