Monthly Archives: December 2014

Ruby Tuesday – Enumerable#any?

This weeks Ruby Tuesday entry covers Enumerable#any?.

Enumerable#any? takes a block and returns true if the block returns a truthy value for any of the elements in the enumeration, otherwise it returns false.

[1, 2, 3, 4, 5, 6].any? {|x| x.even?}
# => true
["foo", "bar", "snafu", "Snuffleupagus"].any? {|x| x.size > 10}
# => true
["foo", "bar", "snafu", "Snuffleupagus"].any? {|x| x.size < 2}
#=> false

The block is optional for Enumerable#any?, and if not specified, any uses the block {|obj| obj} by default.

[nil, nil, nil].any?
# => false
[false, nil].any?
# => false
[1, 2, 3].any?
# => true

Enumerable#any? is eager, so if it finds an element where the block evaluates to a truthy value it stops processing the rest of the enumerable.

Benchmark.realtime{  (1..20000000).lazy.select(&:even?).any?{|x| x.even? } }
# => 3.6e-05
Benchmark.realtime{  (1..20000000).lazy.select(&:even?).any?{|x| x.odd? } }
# => 5.709414

Ruby also has counterparts for any? which are Enumerable#all? and Enumerable#none?. These behave the similiar as Enumerable#any?, except that Enumerable#all? checks to that the return value of the block is truthy for all elements.

[1, 3, 5, 8].all?{|x| x.odd?}
# => false
[1, 3, 5, 81].all?{|x| x.odd?}
# => true
["foo", "bar", "snafu", "Snuffleupagus"].all?{|x| x.size > 10}
# => false
["foo", "bar", "snafu", "Snuffleupagus"].all?{|x| x.size > 2}
# => true
[true, true, true].all?
# => true
["false", true, true].all?
# => true
[true, true, nil].all?
# => false
[false, nil].all?
# => false

While Enumerable#none checks the that return value of the block is a falsey value for all elements.

[1, 3, 5, 7, 9].none?{|x| x.even?}
# => true
[1, 3, 5, 7, 8].none?{|x| x.even?}
# => false
[true, true, true].none?
# => false
[nil, nil, false, nil].none?
# => true

Both Enumerable#all? and Enumerable#none? are both eager as well.

Benchmark.realtime{  (1..20000000).lazy.select(&:even?).all?{|x| x.even? } }
# => 5.250464
Benchmark.realtime{  (1..20000000).lazy.select(&:odd?).all?{|x| x.even? } }
# => 3.3e-05
Benchmark.realtime{  (1..20000000).lazy.select(&:odd?).none?{|x| x.even? } }
# => 5.365924
Benchmark.realtime{  (1..20000000).lazy.select(&:even?).none?{|x| x.even? } }
# => 3.7e-05

–Proctor

Erlang Thursday – lists:partition/2

Today’s Erlang Thursday is lists:partition/2.

lists:partition/2 takes two arguments, a predicate function that will be called for every entry in the list, and returns a boolean value. The second argument to lists:partition/2 is the list to be partitioned.

lists:partition/2 returns a two-tuple, with the first item in the tuple being the list of those items for which the predicate function returns true. The second item in the tuple is a list of those items for which the predicate function returned false.

lists:partition(fun(X) -> X rem 2 == 1 end, [1, 2, 3, 4, 5, 6, 7]).
% {[1,3,5,7],[2,4,6]}
lists:partition(fun(X) -> X rem 3 == 0 end, [1, 2, 3, 4, 5, 6, 7]).
% {[3,6],[1,2,4,5,7]}
lists:partition(fun erlang:is_atom/1, [a, 1, [b, c], 'B', fun lists:sum/1]).
% {[a,'B'],[1,[b,c],#Fun<lists.sum.1>]}
lists:partition(fun erlang:is_atom/1, [a, 1, {b, [z]}, 'B', fun lists:sum/1]).
% {[a,'B'],[1,{b,[z]},#Fun<lists.sum.1>]}
lists:partition(fun erlang:is_atom/1, []).                                    
% {[],[]}

–Proctor

Ruby Tuesday – Enumerable#partition

Today’s Ruby Tuesday method is Enumerable#partition.

Enumerable#partition takes a block and returns an array of two arrays. The first item in the returned array is an array of items for which the block returns a truthy value. The second item in the returned array are the items for which the block returns a falsey value.

[:a, 1, "b", :c, [1, 2, 3,]].partition{|x| x.is_a? Symbol}
# => [[:a, :c], [1, "b", [1, 2, 3]]]
[1, 2, 3, 4, 5, 6, 7].partition{|x| x.even?}
# => [[2, 4, 6], [1, 3, 5, 7]]
["foo", "snafu", "bar", "hello world"].partition{|x| x.size < 5}
# => [["foo", "bar"], ["snafu", "hello world"]]
["foo", "snafu", "bar", "hello world"].partition{|x| x.size < 15}
# => [["foo", "snafu", "bar", "hello world"], []]
[1, 2, 3, 4].partition{|x| x}
# => [[1, 2, 3, 4], []]
[1, 2, 3, 4].partition{|x| nil}
# => [[], [1, 2, 3, 4]]

–Proctor

Erlang Thursday – lists:zip/2

Today’s Erlang Thursday is lists:zip/2.

lists:zip/2 returns a new list of two-tuples, from the corresponding elements of the two lists passed as arguments.

lists:zip([a, b, c, d], [1, 2, 3, 4]).
% [{a,1},{b,2},{c,3},{d,4}]
lists:zip([a, b, c, d], [{1, 3}, {2, 5}, {3, 7}, {4, 11}]).
% [{a,{1,3}},{b,{2,5}},{c,{3,7}},{d,{4,11}}]
lists:zip([a, b], [fun lists:map/3, fun lists:foldl/3]).
% [{a,#Fun<lists.map.3>},{b,#Fun<lists.foldl.3>}]

If the lists are not of the same length a function clause match exception is thrown.

lists:zip([a, b, c, d], [1, 2, 3]).                        
% ** exception error: no function clause matching lists:zip([d],[]) (lists.erl, line 385)
%      in function  lists:zip/2 (lists.erl, line 385)
%      in call from lists:zip/2 (lists.erl, line 385)
lists:zip([a, b, c], [1, 2, 3, 4]).                        
% ** exception error: no function clause matching lists:zip([],[4]) (lists.erl, line 385)
%      in function  lists:zip/2 (lists.erl, line 385)
%      in call from lists:zip/2 (lists.erl, line 385)

There is also a 3-arity version of the zip function lists:zip3/3, which behaves like lists:zip/2 but takes three lists as arguments instead of two.

lists:zip3([a, b, c, d], [1, 2, 3, 4], ["alpha", "bravo", "charlie", "delta"]).         
% [{a,1,"alpha"},{b,2,"bravo"},{c,3,"charlie"},{d,4,"delta"}]

If you need to combine the arguments in a different manner, you can use lists:zipwith/3, or lists:zipwith3/4, each of which takes as the first argument a 2-arity combining function.

lists:zipwith(fun(X, Y) -> X * Y end, [1, 2, 3, 4], [2, 3, 4, 5]).
% [2,6,12,20]
lists:zipwith(fun(X, Y) -> X + Y end, [1, 2, 3, 4], [2, 3, 4, 5]).                                                
% [3,5,7,9]
lists:zipwith3(fun(X, Y, Z) -> X * Y * Z end, [1, 2, 3, 4], [2, 3, 4, 5], [4, 3, 2, 1]).
% [8,18,24,20]
lists:zipwith3(fun(X, Y, Z) -> {{X, Y}, Z} end, [a, b, c, d], [1, 2, 3, 4], ["alpha", "bravo", "charlie", "delta"]).
% [{{a,1},"alpha"},
%  {{b,2},"bravo"},
%  {{c,3},"charlie"},
%  {{d,4},"delta"}]

–Proctor

Ruby Tuesday – Enumerable#zip

Today’s Ruby Tuesday covers Enumerable#zip.

Enumerable#zip weaves together elements of multiple Enumerables. It iterates through each item in the enum and combines it with the corresponding element from each of the arguments passed to zip, resulting in an array of arrays.

[:a, :b, :c].zip([1, 2, 3])
# => [[:a, 1], [:b, 2], [:c, 3]]
[:a, :b, :c].zip([1, 2, 3], [:foo, :bar, :baz])
# => [[:a, 1, :foo], [:b, 2, :bar], [:c, 3, :baz]]

If an item in the argument list is shorter then the enum that zip is being called on, nil is used where the item would have been added.

[:a, :b, :c].zip([1], [:foo, :bar, :baz, :snafu])
# => [[:a, 1, :foo], [:b, nil, :bar], [:c, nil, :baz]]

The length of the returned array is the length of the enum that zip is being called on. Any extra elements in the enumerable passed to zip are ignored, and if are in a lazy enumerble not computed.

[:a, :b, :c].zip([1, 2, 3, 4], [:foo, :bar, :baz, :snafu])
# => [[:a, 1, :foo], [:b, 2, :bar], [:c, 3, :baz]]

### Only takes what is needed ###
require 'benchmark'
# => true
Benchmark.measure do
  [:a, :b].zip( (1..1_000_000_000_000).lazy.map{|x| x*x*x} ) 
end
# => #<Benchmark::Tms:0x007fd5749d9288
#  @cstime=0.0,
#  @cutime=0.0,
#  @label="",
#  @real=4.3e-05,
#  @stime=0.0,
#  @total=0.0,
#  @utime=0.0>

#### For much much less, took 8+ seconds ####
Benchmark.measure do
  [:a, :b].zip( (1..10_000_000).map{|x| x*x*x} )   
end
# => #<Benchmark::Tms:0x007faed4779e00
#  @cstime=0.0,
#  @cutime=0.0,
#  @label="",
#  @real=8.616259,
#  @stime=0.7200000000000002,
#  @total=8.53,
#  @utime=7.809999999999999>

Enumerable#zip can also be given a block. If a block is provided, the combined array of items is passed to the block, and not returned as a result. This behavior is distinct from Enumerable#each in that each still returns the Enumerable it was invoked on, where as zip returns nil as it’s result when a block is given.

[:a, :b, :c].zip([1, 2, 3, 4], [:foo, :bar, :baz, :snafu]) { |items| puts items.to_s }
# [:a, 1, :foo]
# [:b, 2, :bar]
# [:c, 3, :baz]
# => nil
[:a, :b, :c].zip([1, 2, 3, 4], [:foo, :bar, :baz, :snafu]).each { |items| puts items.to_s }
# [:a, 1, :foo]
# [:b, 2, :bar]
# [:c, 3, :baz]
# => [[:a, 1, :foo], [:b, 2, :bar], [:c, 3, :baz]]

–Proctor

Erlang Thursday – lists:foldl/3 and lists:foldr/3

Today’s Erlang Thursday is lists:foldl/3 and lists:foldr/3.

lists:foldl/3 is Erlang’s version of the reduce function. lists:foldl/3 takes a function, an initial accumulator value, and a list, and returns a single value.

The first argument given to foldl is a function that takes two arguments, the item currently iterating over, and the accumulated value. The result of applying this function is used as new new value for the accumulator for the following item, or if no items are left it becomes the result of foldl.

The next argument to lists:foldl/3 is an initial value of the accumulator. This is different than some other languages, as those languages will take the initial value for the accumulator as an optional value and use the first item to be traversed as the default value for the accumulator. But in Erlang an initial value for the accumulator is a required argument to both lists:foldl/3 and lists:foldr/3.

The third, and last, argument to foldl is the list to iterate over.

lists:foldl(fun(X, Sum) -> Sum + X end, 0, [1, 2, 3, 4, 5]).
% 15
lists:foldl(fun(X, Product) -> Product * X end, 1, [1, 2, 3, 4, 5]).
% 120
lists:foldl(fun(X, Accum) -> io:format("~p ", [X]) end, void, [1, 2, 3, 4, 5]).
% 1 2 3 4 5 ok
lists:foldl(fun(X, Accum) -> io:format("~p ", [X]), Accum end, void, [1, 2, 3, 4, 5]).
% 1 2 3 4 5 void
lists:foldl(fun(X, Accum) -> Accum + X end, 1, []).               
% 1
lists:foldl(fun(X, Result) -> lists:umerge(Result, X) end, [], [[1, 2, 3], [3, 5, 8], [11, 13, 17]]).
% [1,2,3,5,8,11,13,17]

The Erlang lists module also contains the function foldr/3 which traverses the list from left to right, or from the last item in the list to the first.

lists:foldr(fun(X, Accum) -> io:format("~p ", [X]), Accum end, void, [1, 2, 3, 4, 5]).
% 5 4 3 2 1 void
lists:foldr(fun(X, Accum) -> Accum + X end, 1, []).
% 1

The Erlang documentation points out that lists:foldl/3 is generally preferable over lists:foldr/3 because lists:foldl/3 is tail recursive, where lists:foldr/3 is not.

cURL and Squid

Yesterday I was attempting to use curl to import data into our development servers to load up lots of data for a demo, and my curl command kept spitting back an error.

HTTP/1.0 417 Expectation Failed
Server: squid/3.1.12
Mime-Version: 1.0
Date: Wed, 10 Dec 2014 13:50:35 GMT
Content-Type: text/html
Content-Length: 3688
X-Squid-Error: ERR_INVALID_REQ 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from localhost
Via: 1.0 localhost (squid/3.1.12)
Connection: close
....
<p>Some possible problems are:</p>
<ul>
<li><p>Missing or unknown request method.</p></li>
<li><p>Missing URL.</p></li>
<li><p>Missing HTTP Identifier (HTTP/1.0).</p></li>
<li><p>Request is too large.</p></li>
<li><p>Content-Length missing for POST or PUT requests.</p></li>
<li><p>Illegal character in hostname; underscores are not allowed.</p></li>
<li><p>HTTP/1.1 <q>Expect:</q> feature is being asked from an HTTP/1.0 software.</p></li>
</ul>
....
<div id="footer">
<p>Generated Wed, 10 Dec 2014 13:50:35 GMT by localhost (squid/3.1.12)</p>
<!-- ERR_INVALID_REQ -->
</div>
</body></html>

Sadly, I wasted time by not thoroughly reading the error message, as I was attempting to go too fast with the goal to get extra data loaded for the demo. Finally, after taking a look at it this morning with fresh eyes, I saw the error was coming from Squid, and was not a response from our development servers complaining about some misconfigured settings or bad data format.

Finding it was had to do with Squid on my local machine, I was able to find that Squid (v3.1) was returning a 417 status code when using cURL’s default header settings.

The fix when using v3.1 of Squid is to add the option -H “Expect:” to the curl command to take off the Expect header.

The better option, if you can manage it, is to upgrade past v3.1 of Squid to at least v3.2, which claims HTTP/1.1 support. More information about the HTTP/1.1 support of Squid can be found here: http://wiki.squid-cache.org/Features/HTTP11

—Proctor

Ruby Tuesday – Enumerable#reduce aka inject

Today’s Ruby Tuesday is Enumerable#reduce, also known as Enumerable#inject.

Enumerable#reduce works against an enum and takes a initial value for the accumulator and two parameter block. The first parameter to the block is the current accumulated value, and second parameter is the current element in the iteration. If no accumulator value is specified, the first item in the enum is taken to be used as the accumulator value.

["baz","Supercalifragilisticexpialidocious", "qwerty"].reduce("") do |shortest, item|
  item.length < shortest.length ? item : shortest
end
# => ""
["baz","Supercalifragilisticexpialidocious", "qwerty"].reduce do |longest, item|
  item.length > longest.length ? item : longest
end
# => "Supercalifragilisticexpialidocious"
["baz", "Supercalifragilisticexpialidocious", "qwerty"].reduce do |shortest, item|
  item.length < shortest.length ? item : shortest
end
# => "baz"

Calling reduce on an empty enum just returns the accumulator, or if no initial value for the accumulator given returns the result of calling first on the enum, which in the case of Array is nil.

[].reduce(:accum){|accum, item| accum * item}
# => :accum
[].reduce{|accum, item| accum * item}
# => nil

Enumerable#reduce can also take a symbol instead of a block, which is the name of the method to invoke on the accumulator value. This form of reduce also takes an initial accumulator value, or uses the first item as the accumulator.

[1, 2, 3, 4, 5].reduce(:+)
# => 15
[1, 2, 3, 4, 5].reduce(:*)
# => 120
[1, 2, 3, 4, 5].reduce(10, :*)
# => 1200
[100, 10, 2].reduce(:/)
# => 5
[100, 10, 2].reduce(1_000_000, :/)
# => 500

–Proctor

Erlang Thursday – lists:map/2

Today’s Erlang Thursday features lists:map/2.

lists:map/2 takes two arguments, a “mapping” function which takes a single argument, and a list of Erlang terms. The result is a new list of items that is the result of mapping the function applied to each value in the original list.

lists:map(fun(X) -> X + 1 end, [1, 2, 3, 4]).
% [2,3,4,5]
lists:map(fun(X) -> X * X end, [1, 2, 3, 4]).
% [1,4,9,16]

Because strings in Erlang are just lists of integers, you can map against strings as well.

lists:map(fun(X) -> X - 1 end, "IBM").
% "HAL"

On Functions in Erlang

If you look at the first example above, you see that the first argument we are passing is fun(X) -> X + 1 end. This is Erlang’s syntax for an anonymous function. An anonymous function takes the form:

fun(Args1) OptionalGuardClause1 ->
        Expression1, Expression2;
   (Args2) OptionalGuardClause2 ->
        Expression3, Expression4;
   (Args3) OptionalGuardClause3 ->
        Expression5, Expression6;
end

Because we have the power of normal functions, except for being able to recursively call an anonymous function before version 17.0, we can use anonymous functions with multiple clauses when passing to map.

lists:map(fun(X) when is_atom(X) -> atom; (X) -> nil end, [1, x, {x}, [], 'B']).
[nil,atom,nil,nil,atom]

Passing Existing Named Functions to lists:map/1

While in some cases an inline anonymous function may work, many times we would want to have a named function for clarity. We can pass named functions to map by using the qualified name of the function – module:function_name/arity – prefaced with fun. The examples below use math:log10/1 and erlang:is_atom/1 to demonstrate.

lists:map(fun math:log10/1, [1, 10, 100, 1000, 10000]).
[0.0,1.0,2.0,3.0,4.0]
lists:map(fun erlang:is_atom/1, [1, x, {x}, [], 'B']).
[false,true,false,false,true]

–Proctor

Ruby Tuesday – Enumerable#map aka collect

Today’s Ruby Tuesday method is Enumerable#map, also known as Enumerable#collect.

Enumerable#map is a function with a long lineage. Enumerable#map takes a set of items, and maps the inputs onto a new set of values via a transformation function, or in the case of Ruby, a block.

[1, 2, 3, 4].map{ |x| x + 1 }
# => [2, 3, 4, 5]
[1, 2, 3, 4].map{ |x| x * x }
# => [1, 4, 9, 16]

In Ruby, the map method is an alias of Enumerable#collect.

[1, 2, 3, 4].map{ |x| x * x }
# => [1, 4, 9, 16]
[1, 2, 3, 4].collect{ |x| x * x }
# => [1, 4, 9, 16]

Ruby also has some ways of taking existing methods, and passing them as the block to be invoked by the map function. If we want to do a transform by calling a method on the object passed to the block, we can prepend the name of the method with a &: and pass that to the map function as an argument.

['i', 'b', 'm'].map{ |x| x.upcase }
=> ["I", "B", "M"]
['i', 'b', 'm'].map(&:upcase)
=> ["I", "B", "M"]

You want to be careful with this though, as not all functions on an object are transformation functions but may modify the underlying object, which may cause confusion to readers of your code if they are expecting the original enumeration to be unchanged.

items = ['i', 'b', 'm']
# => ["i", "b", "m"]
items.map(&:upcase)
# => ["I", "B", "M"]
items
# => ["i", "b", "m"]
items = ['i', 'b', 'm']
# => ["i", "b", "m"]
items.map(&:upcase!)
#=> ["I", "B", "M"]
items
# => ["I", "B", "M"]