Totzkeeeeee's Blog

Just because I can...

  Home  |   Contact  |   Syndication    |   Login
  216 Posts | 4 Stories | 345 Comments | 321 Trackbacks

News


My blog is worth $14,678.04.
How much is your blog worth?

Tag Cloud


Article Categories

Archives

Post Categories

Blog Roll

Cool Sites

Accessing data from a URL is a fairly straightforward exercise.  Many services have implemented a REST style interface to allow queries to be submitted and the results returned to the client, usually as XML. 

For example:

StringBuilder restURL = new StringBuilder();
HttpWebRequest restRequest;
HttpWebResponse restResponse;
XmlDocument xDoc =
new XmlDocument();

// build the URL String - dr is a DataReader
// ADO.NET code omitted for clarity
restURL.AppendFormat("http://myServer/rest/item?catalogNumber={0}&itemOwner={1}", dr["ItemNumber"], dr["ItemOwner"]);

// use the static Create method of the WebRequest object
// casting the returned WebRequest to an HttpWebRequest
restRequest = (HttpWebRequest) WebRequest.Create(restURL.ToString());

// use the GetResponse method to obtain a WebResponse object
// for the request casting to an HttpWebResponse
restResponse = (HttpWebResponse) restRequest.GetResponse();

// since we are expecting XML back, we can load the
// XML document directly from the GetResponseStream()
// method of the HttpWebResponse
xDoc.Load(restResponse.GetResponseStream());

// do something really cool with the Document :-)

Now, what if we had the following URL http://myServer/rest/secure/item?catalogNumber=12345657&itemOwner=ME and that particular path was secured using Basic Authentication?  For an in-depth look at Basic and Digest Authentication, or if you can't sleep, you can read RFC-2617 HTTP Authentication: Basic and Digest Access Authentication

If we were to hit that link with a browser, we'd simply be presented with a logon dialog, fill in our credentials, and click OK.  The problem is, we don't have a browser to gather our credetials and pass them back to the webserver.  On top of that, the webserver is expecting the username and password to be Base64 encoded.  Fortunately, everything we need to solve this dilema is right in the .NET Framework.

According to the RFC, we need to pass back the Authorization header in the following form: Authorization: Basic userid:password with the “userid:password“ portion Base64 encoded.  Adding the Authorization header is easy.  The HttpWebRequest object exposes the Headers collection and we can call the Add method of that collection passing in the name of the header we wish to add along with the value.  So, to use the example given in the RFC, if we wish to pass back the userid “Aladin“ and the password “open sesame“ then we can do the following:

restRequest.Headers.Add("Authorization", "Basic " + "QWxhZGRpbjpvcGVuIHNlc2FtZQ==";

The gobbledy-goop on the end is actually the Base64 encoding of Aladin:open sesame.  No need to panic.  We can again turn to the .NET Framework to make this conversion simple.  As one might expect, the System.Convert class contains the static method ToBase64String.  ToBase64String expects an array of unsigned integers.  What the? Why is nothing every simple? 

Now we need to turn “Aladin:open sesame” into an array of unsigned integers.  Fortunately, System.Encoding contains just the thing we need.  The System.Encoding.ASCII.GetBytes method takes a string as a parameter and returns a byte array containing the encoded version of the string.  We feed that into Convert.ToBase64String and we're ready to go. We end up with this:

restRequest.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(“Aladin:open sesame“)));

We just add that line before the call to GetResponse and everything works out.  We just need to take care of error handling and then we're all set.

Of course, we'll put all of this code inside a try{}catch{} block. 

// catch web related exceptions

catch(WebException webEx)

{

      StringBuilder errorString = new StringBuilder();

      // Protocol errors are things like

      // 404 - Not Found

      // 401 - Unauthorized etc...

      if(webEx.Status == WebExceptionStatus.ProtocolError)

      {

 

            // we can get the actual status code from the original response

            // that is exposed through the Response property of the WebException class

            errorString.AppendFormat("Status Code : {0}", ((HttpWebResponse)webEx.Response).StatusCode);

 

            // the same goes for the description of the returned status

            errorString.AppendFormat("Status Description : {0}", ((HttpWebResponse)webEx.Response).StatusDescription);

      }

}

So, there you go.  I hope that this will be of some value to you in the future.

Dave
Just because I can... (living the hell so that you don't have to)

 

posted on Monday, February 28, 2005 6:33 PM

Feedback

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 3/4/2005 11:52 AM Danli
Hi there,

I am trying to ahieve the same thing you described here, getting page content from a https:// url that requires Basic Auth. I coded exactly the same as you did in your post, but the problem, I always got myWebResponse.ContentLength = 0 while the statuscode always says "ok".

I tried a couple of other ways to pass the credential too, e.g. NetworkCredential, but unfornately, none of them worked, still zero content returned.

p.s. I have x509certificate stuff in my code too, so cerfiticate should not be an issue.

Then why no content is returned??? If you have any input/suggestions ... I GREATLY

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 3/4/2005 11:53 AM Danli
APPRECIATE it !!!!!!!!!


# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 3/4/2005 4:22 PM David Totzke
Hello Danli,

First, try it with your browser to make sure that your request actually returns results. :-)

You shouldn't need to write code for x.509 certificates. That is probably what is mixing you up. The .NET Framework includes support for the http://, https://, and file:// protocols. Try commenting out all of the x.509 code and see what happens. It is likely trying to authenticate you using the certificate rather than basic. The server and client will always negotiate the most secure authentication method that both parties support and the rest will be ignored.

I know that if I added the Authorization header and added in a NetworkCredential object, I got nothing back as well even thought the auth header was correct.

good luck.

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 3/24/2005 12:34 PM Danli
Ok David,

>> First, try it with your browser to make sure that your request actually returns results. :-)

Yes, I get search results when I search (both public and private search over public and private content) in web browser.

I removed the x.509 stuff, but no help. No response data is returned when I tried to get page content from a https:// url that requires Basic Auth.

>>I know that if I added the Authorization header and added >>in a NetworkCredential object, I got nothing back as well >>even thought the auth header was correct.

I did try both (adding the auth header and adding a NetworkCredential obj), but you are right, they don't help to get content back at all even though the auth header was correct.

I am stuck in the middle of nowhere. I don't know what else I can try. If you have any suggestions, I would greatly appreciate it!!

Thanks!

Danli




# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 5/8/2005 10:48 PM Kevin
Hey, thanks a bunch. I was banging my head against 401 errors using the "standard" approach of setting up the CredentialCache of the request. I never did get that to work. Short-circuiting that process with your method worked the first time, out of the box. Thanks. :)

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 5/28/2005 3:40 PM Dorothy
Thank you so much. I have also been getting 401 errors using Credentials. Your method worked for me!! Much appreciated!

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 6/13/2005 11:58 PM Adam
Thank you very much Totzkeeeeee.

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 6/23/2005 8:18 AM hoope
I use the same approach as described here, but the browser asks me the authentication last 2 times of 3 possible. Somehow the browser doesn't remember the username and password. What might be wrong?

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 7/8/2005 9:56 AM Joseph Karbarz
Thanks Totzkeeeeee, this is the closest I've come to find relevant info, BUT, no cigar. I continue to receive 401 error on .GetResponse. I'm tweeked out and am stuck. My latest attempt was to add:

req.PreAuthenticate = true;

I've tried Anonymnous access in IIS; to no avail. I'm assuming that it's IIS that has a problem with authenticating. I'm sending a SOAP envelope to Active Directory for a user profile. Could AD have anything to do with this? I suppose I can try an HttpWebRequest to a basic page; I might get further?

Any ideas would be greatly appreciated.

Thank again,
joe


# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 8/16/2005 3:19 AM Aibo
I was banging my head with credential last 2 days. You saved my time. Thanks a lot! keep it up buddy.

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 1/11/2006 7:04 AM ryan
This is good info and works for pulling data back, but I want to do a Response.Redirect(uri). Is there a way to use this code to do that without getting prompted?

thanks a lot!

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 12/18/2006 8:10 PM Billy
GOOD STUFF, JUST WHAT I WAS LOOKING FOR :)

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 2/3/2008 9:56 PM Sean
Wow, beautiful! I needed to use this for Twitter when I locked down my user profile.

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 3/14/2008 12:38 PM SeanG
This was very helpful! Thanks!

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 8/15/2008 9:44 AM tob
Ok, nice article. That did it for me :-)

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 10/13/2008 12:02 PM kral
Thanks the for article

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 12/10/2008 11:59 PM Me Too
Thank you so much! I have been searching for solution for many hours. Your post saves me.


# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 4/7/2009 10:39 AM SSJ
The same thing can be achieved using this code:
webRequest.Credentials = new NetworkCredential("username", "password");

Hope it helps.

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 4/17/2009 4:22 AM Asmita
Thanks, It worked for me

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 4/24/2009 6:27 AM Niky
>>The same thing can be achieved using this code:
>>webRequest.Credentials = new NetworkCredential("username", "password");
>>
>>Hope it helps.

Thanks this one worked for me.
The above code
restRequest.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(“Aladin:open sesame“)));

didn't work for me with IIS.

# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 6/25/2009 10:14 AM wayne
both code the header add and then the credentials worked for me -- however, now I have to actually display the results in the AxWebBrowser browser that contains a flash component. And I am challenged again.


# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 1/9/2010 7:34 AM Kalyan
Hi, thanks for this piece of information. It was a life saver.
Though this should be used as a last resort among three different approaches..

1. Try NetWorkCredential
NetworkCredential cred = new NetworkCredential(username, password);
MyHttpRequest.Credentials = cred;

2. Try CredentialCache
CredentialCache cache = new CredentialCache();
cache.Add(new Uri(URL), "Basic", cred);
MyHttpRequest.Credentials = new NetworkCredential(username, password);

3. Try the AddHeader as explained in your post.

Options 1 and 2 should work with most IIS servers.


# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 4/14/2010 3:32 PM Gopala
If your webserver has siteminder basic authentication, then you would need to pass SMCHALLENGE Cookie in the header.

restRequest.Headers.Add("Cookie", "SMCHALLENGE=YES");


# re: HOWTO: Programmatically Accessing Web Content using Basic Authentication 4/29/2010 5:52 AM ujha
hi there ,
i need the header Authorization to give some operator seluler to be value for them .
i use this source :
----
Dim Auth As String = System.Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes("" & objSIDInfo.User & ":" & objSIDInfo.Pass & ""))
objRequest.Method = "POST"
objRequest.Headers.Add("Authorization", "Basic " + Auth)

-----

But, when i test , i couldnt get the value Authorization there.

are there some miss from my source code ??

thank you for the answerd before ..


-ujha-

Comments have been closed on this topic.