Erlang Thursday – httpc:request/1 and httpc:request/4

Today’s Erlang Thursday is on httpc:request/1 and httpc:request/4. The httpc module is Erlang’s HTTP 1.1 client, and the request function is a powerful way to make web requests using Erlang.

To start using the httpc module, you first have to make sure inets has been started.

inets:start().
% ok

httpc:request/1 takes one argument, and that is the URL, as a Erlang string, you want to make the request against.

httpc:request("http://www.example.com").
% {ok,{{"HTTP/1.1",200,"OK"},
%      [{"cache-control","max-age=604800"},
%       {"date","Thu, 22 Jan 2015 02:57:06 GMT"},
%       {"accept-ranges","bytes"},
%       {"etag",""359670651""},
%       {"server","ECS (ftw/FBE4)"},
%       {"content-length","1270"},
%       {"content-type","text/html"},
%       {"expires","Thu, 29 Jan 2015 02:57:06 GMT"},
%       {"last-modified","Fri, 09 Aug 2013 23:54:35 GMT"},
%       {"x-cache","HIT"},
%       {"x-ec-custom-error","1"}],
%      "<!doctype html>n<html>n<head>n    <title>Example Domain</title>nn    <meta ..."}}

httpc:request/1 is the equivalent of the httpc:request/4 function called as httpc:request(get, {Url, []}, [], []).

httpc:request(get, {"http://www.example.com", []}, [], []).
% {ok,{{"HTTP/1.1",200,"OK"},
%      [{"cache-control","max-age=604800"},
%       {"date","Thu, 22 Jan 2015 03:04:31 GMT"},
%       {"accept-ranges","bytes"},
%       {"etag",""359670651""},
%       {"server","ECS (ftw/FBE4)"},
%       {"content-length","1270"},
%       {"content-type","text/html"},
%       {"expires","Thu, 29 Jan 2015 03:04:31 GMT"},
%       {"last-modified","Fri, 09 Aug 2013 23:54:35 GMT"},
%       {"x-cache","HIT"},
%       {"x-ec-custom-error","1"}],
%      "<!doctype html>n<html>n<head>n    <title>Example Domain</title>nn    <meta ..."}}

You can specify headers as part of your request. For example, say we want to get DuckDuckGo’s page in Swedish in honor of Erlang being created by Ericsson. To do that, we add a tuple of {"Accept-Language", "sv"} to the headers list as part of the request.

httpc:request(get, {"http://duckduckgo.com/", [{"Accept-Language", "sv"}]}, [], []).
% {ok,{{"HTTP/1.1",200,"OK"},
%      [{"cache-control","max-age=1"},
%       {"connection","keep-alive"},
%       {"date","Thu, 22 Jan 2015 03:19:29 GMT"},
%       {"accept-ranges","bytes"},
%       {"etag",""54bfe2a8-1488""},
%       {"server","nginx"},
%       {"content-length","5256"},
%       {"content-type","text/html; charset=UTF-8"},
%       {"expires","Thu, 22 Jan 2015 03:19:30 GMT"}],
%      "<!DOCTYPE html>n<!--[if IEMobile 7 ]> <html lang="sv_SE" class="no-js iem7"> ..."}}

The third argument of httpc:request/4 is a list of HTTP option tuples. For example, you need to set timeouts on the response in order to avoid waiting on a response from an irresponsive or slow website because if it doesn’t respond in time, the requesting code needs to back off and try again later to avoid triggering the equivalent of a Denial of Service attack. In this case, I am specifying a timeout of 0, expressed in milliseconds, to ensure a timeout happens for illustrative purposes.

httpc:request(get, {"http://erlang.org/", []}, [{timeout, 0}], []).
{error,{failed_connect,[{to_address,{"erlang.org",80}},
                        {inet,[inet],timeout}]}}

As it’s final argument, httpc:request/4 takes a list of other options, these options are for how the Erlang side of things should work. Maybe you want to make a request asynchronously, and want to receive a message when it is complete. To do that you can specify an option tuple of {sync, false}.

{ok, RequestId} = httpc:request(get, {"http://www.example.com", []}, [], [{sync, false}]).
% {ok,#Ref<0.0.0.179>}
receive {http, {RequestId, Result}} -> Result after 500 -> error end.
% {{"HTTP/1.1",200,"OK"},
%  [{"cache-control","max-age=604800"},
%   {"date","Thu, 22 Jan 2015 03:08:41 GMT"},
%   {"accept-ranges","bytes"},
%   {"etag",""359670651""},
%   {"server","ECS (ftw/FBE4)"},
%   {"content-length","1270"},
%   {"content-type","text/html"},
%   {"expires","Thu, 29 Jan 2015 03:08:41 GMT"},
%   {"last-modified","Fri, 09 Aug 2013 23:54:35 GMT"},
%   {"x-cache","HIT"},
%   {"x-ec-custom-error","1"}],
%  <<"<!doctype html>n<html>n<head>n    <title>Example Domain</title>nn    <meta "...>>}

Or maybe you want to get the response body back as an Erlang binary instead of a string.

httpc:request(get, {"http://www.example.com", []}, [], [{body_format, binary}]).
% {ok,{{"HTTP/1.1",200,"OK"},
%      [{"cache-control","max-age=604800"},
%       {"date","Thu, 22 Jan 2015 03:58:55 GMT"},
%       {"accept-ranges","bytes"},
%       {"etag",""359670651""},
%       {"server","ECS (ftw/FBE4)"},
%       {"content-length","1270"},
%       {"content-type","text/html"},
%       {"expires","Thu, 29 Jan 2015 03:58:55 GMT"},
%       {"last-modified","Fri, 09 Aug 2013 23:54:35 GMT"},
%       {"x-cache","HIT"},
%       {"x-ec-custom-error","1"}],
%      <<"<!doctype html>n<html>n<head>n    <title>Example Domain</title>nn    <meta "...>>}}

This post just scratches the surface of what you can do with httpc:request/4, and I highly recommend checking out the Erlang documentation for the httpc module. For more examples and information, also check out the Erlang inets User Guide, and the chapter “HTTP Client“.

–Proctor

Share this: