diff --git a/docs/curl.rst b/docs/curl.rst new file mode 100644 index 0000000..b4faace --- /dev/null +++ b/docs/curl.rst @@ -0,0 +1,260 @@ +Curl for all your web +====================== + +In this chapter we will learn about a very special command, `curl`. It is used +to trasfer data over network, it is written by `Daniel Stenberg +`_. It is most probably one of the highest used +software in the world, you can find it in your servers, cars and also in +television sets. + +In this chapter we will learn a few example use cases. + +Viewing a file +-------------- + +:: + + $ curl https://kushaldas.in/test.html + + + Test page + + + This is a test page. You can view it via curl. + + + + +Here we are reading the content of the file located at URL +`https://kushaldas.in/test.html`, by default curl shows the output on STDOUT. + +Downloading the file +--------------------- + +You can use `-o` flag to download a file and save it with the given filename. + +:: + + $ curl https://kushaldas.in/test.html -o /tmp/download.html + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 125 100 125 0 0 295 0 --:--:-- --:--:-- --:--:-- 296 + + +Download with the same name +---------------------------- + +Use `-O` flag to download the save the file with the same basename from the given URL. + +:: + + $ curl -O https://kushaldas.in/test.html + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 125 100 125 0 0 295 0 --:--:-- --:--:-- --:--:-- 296 + $ ls -l test.html + .rw-r--r--@ 125 kdas 14 Apr 20:45 test.htm + +Here the file is saved in the current directory as `test.html`. + +Inspecting HTTP headers +----------------------- + +You can use `-v` flag to inspect the HTTP headers in a request/response. + +:: + + $ curl -v http://httpbin.org/get + * Trying 54.91.120.77:80... + * Connected to httpbin.org (54.91.120.77) port 80 (#0) + > GET /get HTTP/1.1 + > Host: httpbin.org + > User-Agent: curl/7.79.1 + > Accept: */* + > + * Mark bundle as not supporting multiuse + < HTTP/1.1 200 OK + < Date: Fri, 15 Apr 2022 10:03:05 GMT + < Content-Type: application/json + < Content-Length: 256 + < Connection: keep-alive + < Server: gunicorn/19.9.0 + < Access-Control-Allow-Origin: * + < Access-Control-Allow-Credentials: true + < + { + "args": {}, + "headers": { + "Accept": "*/*", + "Host": "httpbin.org", + "User-Agent": "curl/7.79.1", + "X-Amzn-Trace-Id": "Root=1-625942d9-163a40480c9aea0470fd9c2e" + }, + "origin": "185.195.233.166", + "url": "http://httpbin.org/get" + } + * Connection #0 to host httpbin.org left intact + + +Here the lines with `>` at starting showing the headers in the request, and `<` +shows the headers in the response. + +For the rest of the chapter we will keep using `httpbin.org `_, +which is a service run by `Kenneth Reitz `_. +The service returns JSON as output. + +Say you want to only view the headers, and don't want to see the actual +file/URL content, you can use `-s` and `-o /dev/null` as flags. + +:: + + + $ curl -s -v http://httpbin.org/get -o /dev/null + * Trying 52.7.224.181:80... + * Connected to httpbin.org (52.7.224.181) port 80 (#0) + > GET /get HTTP/1.1 + > Host: httpbin.org + > User-Agent: curl/7.79.1 + > Accept: */* + > + * Mark bundle as not supporting multiuse + < HTTP/1.1 200 OK + < Date: Sat, 16 Apr 2022 09:18:46 GMT + < Content-Type: application/json + < Content-Length: 256 + < Connection: keep-alive + < Server: gunicorn/19.9.0 + < Access-Control-Allow-Origin: * + < Access-Control-Allow-Credentials: true + < + { [256 bytes data] + * Connection #0 to host httpbin.org left intact + + +Doing POST request using curl +----------------------------- + +We can do `HTTP POST `_ requests +using curl in two different ways. Using `-d` flag for simple form submission +style using `application/x-www-form-urlencoded` where each form names & values +are marked with `=` and separate by `&`. + +You can also use `--form/-F` for `multipart/form-data` where we can upload +files or send in large amount of binary data. + +:: + + $ curl -d "name=kushal&lang=Python" https://httpbin.org/post + { + "args": {}, + "data": "", + "files": {}, + "form": { + "lang": "Python", + "name": "kushal" + }, + "headers": { + "Accept": "*/*", + "Content-Length": "23", + "Content-Type": "application/x-www-form-urlencoded", + "Host": "httpbin.org", + "User-Agent": "curl/7.79.1", + "X-Amzn-Trace-Id": "Root=1-625a7542-3994f1a24d276db65e59c88f" + }, + "json": null, + "origin": "193.138.218.212", + "url": "https://httpbin.org/post" + } + + $ curl --form name=kushal --form lang=Python https://httpbin.org/post + { + "args": {}, + "data": "", + "files": {}, + "form": { + "lang": "Python", + "name": "kushal" + }, + "headers": { + "Accept": "*/*", + "Content-Length": "244", + "Content-Type": "multipart/form-data; boundary=------------------------870c3eede45c997d", + "Host": "httpbin.org", + "User-Agent": "curl/7.79.1", + "X-Amzn-Trace-Id": "Root=1-625a755e-2b91ece7042683285bd91332" + }, + "json": null, + "origin": "193.138.218.212", + "url": "https://httpbin.org/post" + } + +Above we had to pass both the form fields using `--form` twice. + +.. note:: You can read the `SPEC + `_ to learn + about the difference. + + +Following redirection +---------------------- + +One can use `-L` option to tell curl to follow any **3xx** redirect form the +server. To see this, first we will call with `-I` to `http://kushaldas.in`, +this will return a *302* redirection to the `https://kushaldas.in` site. In the +second run, we will also provide `-L`, so that curl will follow the +redirection. `-I` allows curl to do a `HEAD` request to the server. + +:: + + $ curl -I http://kushaldas.in + HTTP/1.1 302 Moved Temporarily + Server: nginx/1.18.0 + Date: Sat, 16 Apr 2022 15:03:02 GMT + Content-Type: text/html + Content-Length: 145 + Connection: keep-alive + Location: https://kushaldas.in/ + + + $ curl -LI http://kushaldas.in + HTTP/1.1 302 Moved Temporarily + Server: nginx/1.18.0 + Date: Sat, 16 Apr 2022 15:03:06 GMT + Content-Type: text/html + Content-Length: 145 + Connection: keep-alive + Location: https://kushaldas.in/ + + HTTP/2 200 + server: nginx/1.18.0 + date: Sat, 16 Apr 2022 15:03:06 GMT + content-type: text/html; charset=utf-8 + content-length: 27890 + last-modified: Fri, 01 Apr 2022 13:35:38 GMT + etag: "6246ffaa-6cf2" + strict-transport-security: max-age=31536000 + onion-location: https://kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion + permissions-policy: interest-cohort=() + x-frame-options: DENY + x-content-type-options: nosniff + referrer-policy: strict-origin + accept-ranges: bytes + + +Viewing more details about the transfer +--------------------------------------- + +We can use `--write-out` flag to get more details about the transfer. It prints +them after the main output, based on the variable we pass. For example we can +check the `HTTP status code` in both the calls. + +:: + + $ curl -s --write-out '%{http_code}' http://kushaldas.in -o /dev/null + 302 + $ curl -s --write-out '%{http_code}' https://kushaldas.in -o /dev/null + 200 + +You can pass `--write-out '%{json}'` to see the all the different details as +JSON. Read the man page of curl for more details. + diff --git a/docs/index.rst b/docs/index.rst index 1b3db06..04a9e3b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -14,6 +14,7 @@ Welcome to Linux command line for you and me! startingcommands fhsandaccess useful + curl users files processes