Erlang Thursday – httpc:request/1 and httpc:request/4

Today’s Erlang Thursday is on httpc:request/1 and httpc:request/4. The httpc module is Erlang’s HTTP 1.1 client, and the request function is a powerful way to make web requests using Erlang.

To start using the httpc module, you first have to make sure inets has been started.

inets:start().
% ok

httpc:request/1 takes one argument, and that is the URL, as a Erlang string, you want to make the request against.

httpc:request("http://www.example.com").
% {ok,{{"HTTP/1.1",200,"OK"},
%      [{"cache-control","max-age=604800"},
%       {"date","Thu, 22 Jan 2015 02:57:06 GMT"},
%       {"accept-ranges","bytes"},
%       {"etag","\"359670651\""},
%       {"server","ECS (ftw/FBE4)"},
%       {"content-length","1270"},
%       {"content-type","text/html"},
%       {"expires","Thu, 29 Jan 2015 02:57:06 GMT"},
%       {"last-modified","Fri, 09 Aug 2013 23:54:35 GMT"},
%       {"x-cache","HIT"},
%       {"x-ec-custom-error","1"}],
%      "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta ..."}}

httpc:request/1 is the equivalent of the httpc:request/4 function called as httpc:request(get, {Url, []}, [], []).

httpc:request(get, {"http://www.example.com", []}, [], []).
% {ok,{{"HTTP/1.1",200,"OK"},
%      [{"cache-control","max-age=604800"},
%       {"date","Thu, 22 Jan 2015 03:04:31 GMT"},
%       {"accept-ranges","bytes"},
%       {"etag","\"359670651\""},
%       {"server","ECS (ftw/FBE4)"},
%       {"content-length","1270"},
%       {"content-type","text/html"},
%       {"expires","Thu, 29 Jan 2015 03:04:31 GMT"},
%       {"last-modified","Fri, 09 Aug 2013 23:54:35 GMT"},
%       {"x-cache","HIT"},
%       {"x-ec-custom-error","1"}],
%      "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta ..."}}

You can specify headers as part of your request. For example, say we want to get DuckDuckGo’s page in Swedish in honor of Erlang being created by Ericsson. To do that, we add a tuple of {"Accept-Language", "sv"} to the headers list as part of the request.

httpc:request(get, {"http://duckduckgo.com/", [{"Accept-Language", "sv"}]}, [], []).
% {ok,{{"HTTP/1.1",200,"OK"},
%      [{"cache-control","max-age=1"},
%       {"connection","keep-alive"},
%       {"date","Thu, 22 Jan 2015 03:19:29 GMT"},
%       {"accept-ranges","bytes"},
%       {"etag","\"54bfe2a8-1488\""},
%       {"server","nginx"},
%       {"content-length","5256"},
%       {"content-type","text/html; charset=UTF-8"},
%       {"expires","Thu, 22 Jan 2015 03:19:30 GMT"}],
%      "<!DOCTYPE html>\n<!--[if IEMobile 7 ]> <html lang=\"sv_SE\" class=\"no-js iem7\"> ..."}}

The third argument of httpc:request/4 is a list of HTTP option tuples. For example, you need to set timeouts on the response in order to avoid waiting on a response from an irresponsive or slow website because if it doesn’t respond in time, the requesting code needs to back off and try again later to avoid triggering the equivalent of a Denial of Service attack. In this case, I am specifying a timeout of 0, expressed in milliseconds, to ensure a timeout happens for illustrative purposes.

httpc:request(get, {"http://erlang.org/", []}, [{timeout, 0}], []).
{error,{failed_connect,[{to_address,{"erlang.org",80}},
                        {inet,[inet],timeout}]}}

As it’s final argument, httpc:request/4 takes a list of other options, these options are for how the Erlang side of things should work. Maybe you want to make a request asynchronously, and want to receive a message when it is complete. To do that you can specify an option tuple of {sync, false}.

{ok, RequestId} = httpc:request(get, {"http://www.example.com", []}, [], [{sync, false}]).
% {ok,#Ref<0.0.0.179>}
receive {http, {RequestId, Result}} -> Result after 500 -> error end.
% {{"HTTP/1.1",200,"OK"},
%  [{"cache-control","max-age=604800"},
%   {"date","Thu, 22 Jan 2015 03:08:41 GMT"},
%   {"accept-ranges","bytes"},
%   {"etag","\"359670651\""},
%   {"server","ECS (ftw/FBE4)"},
%   {"content-length","1270"},
%   {"content-type","text/html"},
%   {"expires","Thu, 29 Jan 2015 03:08:41 GMT"},
%   {"last-modified","Fri, 09 Aug 2013 23:54:35 GMT"},
%   {"x-cache","HIT"},
%   {"x-ec-custom-error","1"}],
%  <<"<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta "...>>}

Or maybe you want to get the response body back as an Erlang binary instead of a string.

httpc:request(get, {"http://www.example.com", []}, [], [{body_format, binary}]).
% {ok,{{"HTTP/1.1",200,"OK"},
%      [{"cache-control","max-age=604800"},
%       {"date","Thu, 22 Jan 2015 03:58:55 GMT"},
%       {"accept-ranges","bytes"},
%       {"etag","\"359670651\""},
%       {"server","ECS (ftw/FBE4)"},
%       {"content-length","1270"},
%       {"content-type","text/html"},
%       {"expires","Thu, 29 Jan 2015 03:58:55 GMT"},
%       {"last-modified","Fri, 09 Aug 2013 23:54:35 GMT"},
%       {"x-cache","HIT"},
%       {"x-ec-custom-error","1"}],
%      <<"<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta "...>>}}

This post just scratches the surface of what you can do with httpc:request/4, and I highly recommend checking out the Erlang documentation for the httpc module. For more examples and information, also check out the Erlang inets User Guide, and the chapter “HTTP Client“.

–Proctor

,

Leave a Comment

Ruby Tuesday – Net::HTTP::get

Today’s Ruby Tuesday is a short one, it is on Net::HTTP::get.

There are times when you need a very basic ability to make a HTTP call, and Net::HTTP::get can help you with that.

To first be able to use Net::HTTP::get you need to require net/http.

require 'net/http'

Net::HTTP::get takes either a URI object, or a hostname, path and optional port, which defaults to port 80 if not specified. Net::HTTP::get returns a string that is the body of the response.

Net::HTTP.get(URI('http://www.example.com'))
# => "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 50px;\n        background-color: #fff;\n        border-radius: 1em;\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        body {\n            background-color: #fff;\n        }\n        div {\n            width: auto;\n            margin: 0 auto;\n            border-radius: 0;\n            padding: 1em;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is established to be used for illustrative examples in documents. You may use this\n    domain in examples without prior coordination or asking for permission.</p>\n    <p><a href=\"http://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
Net::HTTP.get('www.example.com', '/')
# => "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 50px;\n        background-color: #fff;\n        border-radius: 1em;\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        body {\n            background-color: #fff;\n        }\n        div {\n            width: auto;\n            margin: 0 auto;\n            border-radius: 0;\n            padding: 1em;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is established to be used for illustrative examples in documents. You may use this\n    domain in examples without prior coordination or asking for permission.</p>\n    <p><a href=\"http://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
[11] pry(main)> res = Net::HTTP.get('www.example.com', '/')
=> "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 50px;\n        background-color: #fff;\n        border-radius: 1em;\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        body {\n            background-color: #fff;\n        }\n        div {\n            width: auto;\n            margin: 0 auto;\n            border-radius: 0;\n            padding: 1em;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is established to be used for illustrative examples in documents. You may use this\n    domain in examples without prior coordination or asking for permission.</p>\n    <p><a href=\"http://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
res.class
# => String

Be warned, that this is for very simple cases, and the documentation notes that this is not recommended if you are doing performing many HTTP requests, as it does not persist the connection.

If you need more features or a higher level of abstraction for making HTTP requests I would suggest checking out Faraday or Typhoeus, but I do want to call out that there is Net::HTTP::get if you need it.

–Proctor

,

Leave a Comment

Erlang Thursday – erlang:apply/3

Today’s Erlang Thursday is on erlang:apply/3.

With functional languages we love to pass functions around as the first class citizens that they are. But sometimes we don’t know which function it is that we will need to invoke, causing us to be unsure of the arguments the function takes up front. If we knew, we could just invoke it as Fun(Arg1, Arg2, ..., ArgN), but that doesn’t work if we could get different functions of varying arities. Enter erlang:apply/3.

erlang:apply/3, takes the module name, the function name, and a list of the arguments to be passed to the function. The function passed to erlang:apply/3 must also have been exported, otherwise an error will be raised.

erlang:apply(lists, max, [[7, 3, 5, 11, 1]]).
% 11
erlang:apply(lists, merge, [[1, 2, 3], [a, b, c]]).        
% [1,2,3,a,b,c]
erlang:apply(lists, merge, [[1, 2, 3], [a, b, c]]).           
% [1,2,3,a,b,c]

The Erlang documentation points out that this should be used only when the number of arguments is not known at compile time. Otherwise we could just do the a normal function invocation, even if passed a anonymous function.

fun lists:max/1([1, 2, 3, 4]).
4

The erlang module also includes a version erlang:apply/2 that takes a function as it’s first argument, and a list of the arguments to be passed to the function as it’s second argument.

erlang:apply(fun lists:merge/2, [[1, 2, 3], [a, b, c]]).
% [1,2,3,a,b,c]

While erlang:apply/2 and erlang:apply/3 will not be part of your common usage, there are cases where it is needed, like last weeks timer:tc. And though your usage of it will likely be rare, it is still good to know that you have it handy.

–Proctor

,

Leave a Comment

Ruby Tuesday – Proc#call

Today’s Ruby Tuesday is Proc#call.

Procs are blocks of code that are bound to local variables, and as such, we need a way to be able to invoke them at a later time. Enter Proc#call.

Proc#call takes a list of arguments which are then used as the arguments to the Proc that call is being invoked on.

Proc.new {|x| x * x }.call 9
# => 81
Kernel.proc{|x| x * x }.call 11
# => 121
lambda {|x| x * x }.call 7
# => 49
->(x){x * x}.call 5
# => 25
->(x){x * x}.call 5
# => 25

If Proc#call is invoked with extra arguments, it will either discard the extra arguments, or raise an error if it was created as a lambda.

Proc.new {|x| x * x }.call 9, 3
# => 81
Kernel.proc{|x| x * x }.call 11, 15
# => 121
lambda {|x| x * x }.call 7, 13
# ArgumentError: wrong number of arguments (2 for 1)
# from (pry):94:in `block in __pry__'
->(x){x * x}.call 5, 3
# ArgumentError: wrong number of arguments (2 for 1)
# from (pry):93:in `block in __pry__'

Proc#call also has some other syntactic sugar providing alias for calling the method. You can use [], .(), or even the method Proc#yield as variations of Proc#call.

Proc.new {|x| x * x }[9]
=> 81
Proc.new {|x| x * x }.(9)
=> 81
Proc.new {|x| x * x }.yield 9
=> 81

As an extra bonus, there is also the method Method#call that can be used when you have an named method and you have a Method object for it. This call method behaves in the manner of lambda in that it expects the correct number of arguments to be passed to it.

def square(x)
  x * x
end

method(:square).call 2
# => 4
method(:square).call 2, 4
# ArgumentError: wrong number of arguments (2 for 1)
# from (pry):78:in `square'

–Proctor

,

Leave a Comment

Erlang Thursday – timer:tc/3

Today’s Erlang Thursday is on timer:tc/3.

I am sure we have all written some timing code where we capture the current time, do something, capture the current time again and then find the difference to find out how long something took to execute. In Erlang, that generally looks something like the following:

Time1 = now().
% {1420,519186,111375}
timer:sleep(4000).  % Do something
% ok
Time2 = now().
% {1420,519190,118280}
timer:now_diff(Time2, Time1).
% 4006905

Note that we have to use timer:now_diff/2, since the now() function returns the timestamp as a tuple, and we can’t just do normal subtraction on that tuple like we might be able to in other languages.

Of course as good “engineers”, we know that since we need to do timings in various places of the app we can just create our own function to do that, and have that live in just one place.

The downside is: the wise people on the Erlang language team have done that for us already and provided it in the form of timer:tc/3.

timer:tc/3 takes the module name, function name, and a list of the arguments to be passed to the function. And since we usually want the result of the function we are calling, in addition to the timing, the return value is a tuple of the time in microseconds, and the result of applying the function passed to timer:tc/3.

timer:tc(timer, sleep, [4000]). 
% {4003097,ok}
timer:tc(lists, foldl, [fun(X, Accum) -> X + Accum end, 0, lists:seq(1, 2000000)]). 
% {5099481,2000001000000}

There is also timer:tc/1 which takes just a function and applies it, and timer:tc/2 which takes a function and applies it with the given arguments.

timer:tc(fun() -> lists:foldl(fun(X, Accum) -> X + Accum end, 0, lists:seq(1, 2000000)) end).
% {5709293,2000001000000}
timer:tc(fun lists:foldl/3, [fun(X, Accum) -> X + Accum end, 0, lists:seq(1, 2000000)]).
% {5766480,2000001000000}

–Proctor

,

1 Comment

Ruby Tuesday – Benchmark::realtime

Today’s Ruby Tuesday is on Benchmark::realtime.

How many times have you written this Ruby method?

def time_it
  start = Time.now  
  yield
  Time.now - start
end

And truthfully, how many different times have you seen this method defined in the same application even?

We all know this method, and have likely written it, even if it was in other programming languages. It is the standard, “How long is this part of the code taking?” wrapper that we decide we should generalize into a method.

time_it { (1..20000).map{|x| x * x * x} }
# => 0.004334
time_it { (1..200000).map{|x| x * x * x} }
# => 0.029896
time_it { (1..2000000).map{|x| x * x * x} }
# => 0.414218
time_it { (1..20000000).map{|x| x * x * x} }
# => 8.827975

There is good news about this though. The Ruby language team has written this for you and all you have to do is require 'benchmark' to get access to it. It is Benchmark::realtime.

require 'benchmark'
# => true
Benchmark.realtime { (1..20000000).map{|x| x * x * x} }
# => 6.681451

And make sure to check out the Benchmark module for other methods such as measure, which gives you a breakdown of the user CPU time, the system CPU time, total CPU time, total elapsed time and time of children processes.

Benchmark.measure { (1..20000000).map{|x| x * x * x} }
# => #<Benchmark::Tms:0x007f89ce401cf8
#  @cstime=0.0,
#  @cutime=0.0,
#  @label="",
#  @real=13.406651,
#  @stime=3.1399999999999997,
#  @total=12.549999999999997,
#  @utime=9.409999999999997>

–Proctor

,

Leave a Comment

exit status 1 on zeus start

We were in the middle of a Rails upgrade at work along with updating various gems to help ease that upgrade.

Somewhere down the line after doing various updates to the different gems and switching back and forth between git branches, zeus would hang on start with just the output of

exit status 1

If I tried to run any of the zeus subcommands, they would hang and be completely unresponsive until I shut down zeus.

It turned out the problem was all of the bundle updates and bundle installs as part of upgrading the project was the culprit. It would seem that all of the different versions of the gems lying around were confusing zeus.

A simple run of bundle clean --force and voilà, zeus start was able to go about its business as normal.

And a tip of the hat to the full answer found here on stackoverflow.com, which includes additional steps you can try if just running the bundle clean --force doesn’t fix your issue.

Leave a Comment

Erlang Thursday – lists:any/2

Today’s Erlang Thursday function of the week is lists:any/2.

lists:any/2 takes a predicate function as the first argument, and a list to iterate over as its second argument. lists:any/2 returns true if the predicate function returns true for any of the elements in the given list, otherwise, lists:any/2 returns false.

lists:any(fun erlang:is_atom/1, [1, 2, 3, 4, 5, 6, 7]).
% false
lists:any(fun erlang:is_atom/1, [1, 2, 3, 4, a, 6, 7]).
% true
lists:any(fun erlang:is_atom/1, [{1, 2}, 3, 4, a, 6, 7]). 
% true
lists:any(fun(X) -> X rem 2 == 1 end, [1, 2, 4]).
% true
lists:any(fun(X) -> X rem 2 == 1 end, [0, 2, 4]).    
% false

lists:any/2 is eager, and will return with a result of true as soon as it is found, and will ignore processing the rest of the list.

timer:tc(lists, any, [fun(X) -> X rem 2 == 1 end, lists:seq(2, 200000, 2)]).
% {248410,false}
timer:tc(lists, any, [fun(X) -> X rem 2 == 0 end, lists:seq(2, 200000, 2)]).
% {13,true}

The lists module also contains a function lists:all/2, similar to lists:any/2, but checks if the predicate function returns true for every element in the supplied list.

lists:all(fun erlang:is_number/1, [1, 2, 3, 4, a, 6, 7]).
% false
lists:all(fun erlang:is_number/1, [1, 2, 3, 4, 5, 6, 7]).
% true

lists:all/2 is also eager, and will return with a result of false as soon as it is found, and will ignore processing the rest of the list.

timer:tc(lists, all, [fun(X) -> X rem 2 == 0 end, lists:seq(2, 200000, 2)]).
% {235276,true}
timer:tc(lists, all, [fun(X) -> X rem 2 == 1 end, lists:seq(2, 200000, 2)]).
% {14,false}

–Proctor

,

Leave a Comment

Ruby Tuesday – Enumerable#any?

This weeks Ruby Tuesday entry covers Enumerable#any?.

Enumerable#any? takes a block and returns true if the block returns a truthy value for any of the elements in the enumeration, otherwise it returns false.

[1, 2, 3, 4, 5, 6].any? {|x| x.even?}
# => true
["foo", "bar", "snafu", "Snuffleupagus"].any? {|x| x.size > 10}
# => true
["foo", "bar", "snafu", "Snuffleupagus"].any? {|x| x.size < 2}
#=> false

The block is optional for Enumerable#any?, and if not specified, any uses the block {|obj| obj} by default.

[nil, nil, nil].any?
# => false
[false, nil].any?
# => false
[1, 2, 3].any?
# => true

Enumerable#any? is eager, so if it finds an element where the block evaluates to a truthy value it stops processing the rest of the enumerable.

Benchmark.realtime{  (1..20000000).lazy.select(&:even?).any?{|x| x.even? } }
# => 3.6e-05
Benchmark.realtime{  (1..20000000).lazy.select(&:even?).any?{|x| x.odd? } }
# => 5.709414

Ruby also has counterparts for any? which are Enumerable#all? and Enumerable#none?. These behave the similiar as Enumerable#any?, except that Enumerable#all? checks to that the return value of the block is truthy for all elements.

[1, 3, 5, 8].all?{|x| x.odd?}
# => false
[1, 3, 5, 81].all?{|x| x.odd?}
# => true
["foo", "bar", "snafu", "Snuffleupagus"].all?{|x| x.size > 10}
# => false
["foo", "bar", "snafu", "Snuffleupagus"].all?{|x| x.size > 2}
# => true
[true, true, true].all?
# => true
["false", true, true].all?
# => true
[true, true, nil].all?
# => false
[false, nil].all?
# => false

While Enumerable#none checks the that return value of the block is a falsey value for all elements.

[1, 3, 5, 7, 9].none?{|x| x.even?}
# => true
[1, 3, 5, 7, 8].none?{|x| x.even?}
# => false
[true, true, true].none?
# => false
[nil, nil, false, nil].none?
# => true

Both Enumerable#all? and Enumerable#none? are both eager as well.

Benchmark.realtime{  (1..20000000).lazy.select(&:even?).all?{|x| x.even? } }
# => 5.250464
Benchmark.realtime{  (1..20000000).lazy.select(&:odd?).all?{|x| x.even? } }
# => 3.3e-05
Benchmark.realtime{  (1..20000000).lazy.select(&:odd?).none?{|x| x.even? } }
# => 5.365924
Benchmark.realtime{  (1..20000000).lazy.select(&:even?).none?{|x| x.even? } }
# => 3.7e-05

–Proctor

,

Leave a Comment

Erlang Thursday – lists:partition/2

Today’s Erlang Thursday is lists:partition/2.

lists:partition/2 takes two arguments, a predicate function that will be called for every entry in the list, and returns a boolean value. The second argument to lists:partition/2 is the list to be partitioned.

lists:partition/2 returns a two-tuple, with the first item in the tuple being the list of those items for which the predicate function returns true. The second item in the tuple is a list of those items for which the predicate function returned false.

lists:partition(fun(X) -> X rem 2 == 1 end, [1, 2, 3, 4, 5, 6, 7]).
% {[1,3,5,7],[2,4,6]}
lists:partition(fun(X) -> X rem 3 == 0 end, [1, 2, 3, 4, 5, 6, 7]).
% {[3,6],[1,2,4,5,7]}
lists:partition(fun erlang:is_atom/1, [a, 1, [b, c], 'B', fun lists:sum/1]).
% {[a,'B'],[1,[b,c],#Fun<lists.sum.1>]}
lists:partition(fun erlang:is_atom/1, [a, 1, {b, [z]}, 'B', fun lists:sum/1]).
% {[a,'B'],[1,{b,[z]},#Fun<lists.sum.1>]}
lists:partition(fun erlang:is_atom/1, []).                                    
% {[],[]}

–Proctor

,

Leave a Comment