2

I am trying to download some data in the form of json from a SharePoint API. I was able to do it using WGET:

wget --user user --password password --header='Accept: application/json;odata=verbose' "URL"

and I am trying to do the same thing with CURL.

I tried using different headers and authentication mechanisms, but it always gives me the bad request error. Below are the commands I tried:

curl -v --user user:password "URL" curl -v --user user:password -H 'Accept: application/json;odata=verbose' "URL" curl -v --ntlm --user user:password -H 'Accept: application/xml' "URL" curl -v --anyauth --user user:password -H 'Accept: application/xml' "URL"

Below is the output I get:

* About to connect() to sharepoint2.domain.com port 80 (#0)
*   Trying xx.xx.xx.xx... connected
* Connected to sharepoint2.domain.com (xx.xx.xx.xx) port 80 (#0)
> GET <path> HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: sharepoint2.domain.com
> Accept: application/json;odata=verbose
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< Date: Mon, 12 Feb 2018 21:45:06 GMT
< Connection: close
< Content-Length: 311
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request</h2>
<hr><p>HTTP Error 400. The request is badly formed.</p>
</BODY></HTML>
* Closing connection #0

I am not sure what's going wrong. How do I solve the issue and pull the data with CURL ?

UPDATE:

I tried putting the -L in the command in case there is an issue with redirect as pointed out by @Nicholas in his answer, but it's still the same error :( . I don't think it is really a redirect issue because I am getting the 400 bad request response from the API right?.

My URL is like this: .../_api/web/lists/GetByTitle('new list')/items. I guessed the error has something to do with GetByTitle('new list')..so I tried giving it a different URL: .../_api/web/lists and it didn't give me the 400 bad request Error! but instead I got HTTP/1.1 401 Unauthorized. SO my intial error had something to do with encoding the URL ?

I wanted to see why Wget is working and used the --debug option to print out the whole log and I found something interesting: 

[~ /data/home/user/pir]$wget --user user --debug --password password --header='Accept: application/json;odata=verbose' "URL"
Setting --password (password) to password
Setting --header (header) to Accept: application/json;odata=verbose
DEBUG output created by Wget 1.12 on linux-gnu.

--2018-02-13 07:13:08--  URL
Host “sharepoint2.domain.com” has not issued a general basic challenge.
Resolving sharepoint2.domain.com... xx.xx.xx.xx
Caching sharepoint2.domain.com => xx.xx.xx.xx
Connecting to sharepoint2.domain.com|xx.xx.xx.xx|:80... connected.
Created socket 3.
Releasing 0x0000000002229e70 (new refcount 1).

---request begin---
GET path HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: application/json;odata=verbose
Host: sharepoint2.domain.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Unauthorized
Content-Type: text/plain; charset=utf-8
Server: Microsoft-IIS/8.5
SPRequestGuid: 90174a9e-d921-1018-c709-d64cd2403a12
request-id: 90174a9e-d921-1018-c709-d64cd2403a12
SPRequestDuration: 29
SPIisLatency: 1
WWW-Authenticate: NTLM
WWW-Authenticate: Negotiate
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4859
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Tue, 13 Feb 2018 13:13:08 GMT
Connection: keep-alive
Content-Length: 16

---response end---
401 Unauthorized
Registered socket 3 for persistent reuse.
Skipping 16 bytes of body: [401 UNAUTHORIZED] done.
Empty NTLM message, starting transaction.
Creating a type-1 NTLM message.
Reusing existing connection to sharepoint2.domain.com:80.
Reusing fd 3.

---request begin---
GET <path> HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: application/json;odata=verbose
Host: sharepoint2.domain.com
Connection: Keep-Alive
Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/8.5
WWW-Authenticate: NTLM TlRMTVNTUAACAAAABAAEADgAAAAGAoEC33/kD9suGK8AAAAAAAAAAOAA4AA8AAAABgOAJQAAAA9DT1JQAgAIAEMATwBSAFAAAQAUAFcAUABOAE8AQQBTAFAAQwAwADYABAAsAGMAbwByAHAALgBiAGEAbgBrAG8AZgBhAG0AZQByAGkAYwBhAC4AYwBvAG0AAwBSAHcAcABuAG8AYQBzAHAAYwAwADYALgB1AHMAcABuAG8AYwB0AC4AYQBtAHIAcwAuAGIAYQBuAGsAbwBmAGEAbQBlAHIAaQBjAGEALgBjAG8AbQAFACIAYgBhAG4AawBvAGYAYQBtAGUAcgBpAGMAYQAuAGMAbwBtAAcACADOsXRkzKTTAQAAAAA=
SPRequestGuid: 90174a9e-8925-1018-c709-d7cd9dd3150d
request-id: 90174a9e-8925-1018-c709-d7cd9dd3150d
SPRequestDuration: 1
SPIisLatency: 0
WWW-Authenticate: Negotiate
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4859
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Tue, 13 Feb 2018 13:13:08 GMT
Connection: keep-alive
Content-Length: 0

---response end---
401 Unauthorized
Skipping 0 bytes of body: [] done.
Received a type-2 NTLM message.
Creating a type-3 NTLM message.
Reusing existing connection to sharepoint2.domain.com:80.
Reusing fd 3.

---request begin---
GET <path> HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: application/json;odata=verbose
Host: sharepoint2.domain.com
Connection: Keep-Alive
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAEcAAAAYABgAXwAAAAAAAABAAAAABwAHAEAAAAAAAAAARwAAAAAAAAB3AAAAAYIAAHpraDl4Y2w02/8GDGRzkuXXcgWtte6c6h6ggw8yerZmu6ewFZpvd9hNAYfP/XDSQWvE5Eukuy0=

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: application/json;odata=verbose;charset=utf-8
Expires: Mon, 29 Jan 2018 13:13:08 GMT
Last-Modified: Tue, 13 Feb 2018 13:13:08 GMT
Server: Microsoft-IIS/8.5
X-SharePointHealthScore: 0
SPClientServiceRequestDuration: 4936
X-AspNet-Version: 4.0.30319
SPRequestGuid: 90174a9e-c925-1018-c709-d9b4ad681772
request-id: 90174a9e-c925-1018-c709-d9b4ad681772
Persistent-Auth: true
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4859
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Tue, 13 Feb 2018 13:13:13 GMT
Connection: close

---response end---
200 OK
Length: unspecified [application/json]
Saving to: “items.5”

The Wget request fails with HTTP/1.1 401 Unauthorized but it sill tries executing the request a total of 3 times and it gets connected 3rd time. and it does this all the time.

Could it be that I have to pass some argument to CURL to keep trying even after it failed with HTTP/1.1 401 Unauthorized ?

Any help would be appreciated. Thank you.

Improve this question

1 Answer 1

Reset to default
1

your issue has to do with the difference between wget and curl with respect to 302 responses.

see https://askubuntu.com/questions/938391/url-works-in-wget-but-not-in-curl-related-to-apt

Note the HTTP 302 status code. This is a redirect, meaning that the URL provided maps to another URL. Wget by default follows redirects, curl does not. Browsers also usually follow redirects by default. In order to make curl follow redirects, use the option -L or --location.

Quoting from the man page,

-L, --location (HTTP) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place.

So use the -L command line argument to curl and you will be right as rain.

UPDATE:

After you sent the wget debug output, we found wget is cycling between the attempts to authenticate and landing on ntlm. so by removing --negotiate from the curl while keeping -L helped.

Improve this answer
14
  • Thanks for the answer! I tried it hoping it would work, but it still gives me the same error! :(
    – Hemanth
    Commented Feb 13, 2018 at 0:56
  • hmm.... can you try just getting to the Home.aspx page without using the header to see if that works? let's try just downloading some aspx page and get that working, then go from there. what web service are you trying to call?
    – Nicholas DiPiazza
    Commented Feb 13, 2018 at 4:01
  • Here's the command I used: curl -v --ntlm --negotiate --user user:password -H 'Accept: application/json;odata=verbose' "URL"
    – Hemanth
    Commented Feb 13, 2018 at 4:11
  • ok try: curl -v --ntlm --negotiate --user 'user:password' http://yoursharepoint.domain/SitePages/Home.aspx and see if that works
    – Nicholas DiPiazza
    Commented Feb 13, 2018 at 4:12
  • Nope, doesn't seem to work. What baffles me is that wget gives me no trouble where as CURL doesn't seem to work at all.
    – Hemanth
    Commented Feb 13, 2018 at 4:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.