Tag Archives: Erlang

Erlang Thursday – erl_tar:extract/1

Today’s Erlang Thursday cover’s erl_tar:extract/1.

erl_tar:extract/1 takes a file, either as a binary tuple, file descriptor tuple, or filename, and extracts the contents of the tar out to the current directory.

Since we will need to have a tar file to extract, let’s create some files and add them to a new tar file.

$ echo "woof" > dog.txt
$ echo "meow" > cat.txt
$ echo "sparkle" > pony.txt
$ echo 'Wocka Wocka Wocka!' > bear.txt
$ tar -cvf animal_sounds.tar dog.txt cat.txt pony.txt bear.txt
a dog.txt
a cat.txt
a pony.txt
a bear.txt

And while we are at it, lets create a compressed version as well.

$ tar -cvzf animal_sounds.tar.gz dog.txt cat.txt pony.txt bear.txt
a dog.txt
a cat.txt
a pony.txt
a bear.txt

Since we are going to test out extracting the tar, we will go ahead and clean up the files that we put in the tar.

$ rm dog.txt cat.txt pony.txt bear.txt

With all the ceremony of making sure we have a tar file to experiment with out of the way, it is time to fire up our Erlang shell, and call erl_tar:extract/1.

erl_tar:extract("animal_sounds.tar").
% ok

That seemed straight forward enough, so let’s see if we have our files extracted back out at the command prompt.

$ ls dog.txt cat.txt pony.txt bear.txt
bear.txt cat.txt  dog.txt  pony.txt
$ rm dog.txt cat.txt pony.txt bear.txt

And since we saw them, we will go ahead and remove them to get back to a clean state.

erl_tar:extract/2

Erlang also has a erl_tar:extract/2, which allows us to give options to the extraction process, by passing a list as its second argument.

We can have erl_tar:extract/2 extract the files and tell it to be verbose, and then follow that up with another extraction, where we specify that we not only want it to be verbose, but don’t overwrite any files that are already there.

erl_tar:extract("animal_sounds.tar", [verbose]).
% x /Users/proctor/tmp/dog.txt
%
% x /Users/proctor/tmp/cat.txt
%
% x /Users/proctor/tmp/pony.txt
%
% x /Users/proctor/tmp/bear.txt
%
% ok
erl_tar:extract("animal_sounds.tar", [verbose, keep_old_files]).
% x /Users/proctor/tmp/dog.txt - exists, not created
%
% x /Users/proctor/tmp/cat.txt - exists, not created
%
% x /Users/proctor/tmp/pony.txt - exists, not created
%
% x /Users/proctor/tmp/bear.txt - exists, not created
%
% ok

And yet again, we swing back to the command prompt to remove the extracted files.

$ rm dog.txt cat.txt pony.txt bear.txt

Next we extract animal_sounds.tar.gz by passing the atom compressed in the list of options.

erl_tar:extract("animal_sounds.tar.gz", [verbose, compressed, keep_old_files]).
% x /Users/proctor/tmp/dog.txt
%
% x /Users/proctor/tmp/cat.txt
%
% x /Users/proctor/tmp/pony.txt
%
% x /Users/proctor/tmp/bear.txt
%
% ok

And sometimes when working with a tar file in your program, you don’t want to have to do all the management of the files on the filesystem just to read the contents of a tar file, so there is even an option to keep it all in memory.

erl_tar:extract("animal_sounds.tar.gz", [verbose, compressed, keep_old_files, memory]).
% {ok,[{"dog.txt",<<"woofn">>},
%      {"cat.txt",<<"meown">>},
%      {"pony.txt",<<"sparklen">>},
%      {"bear.txt",<<"Wocka Wocka Wocka!n">>}]}

When passing the memory option, the return value of erl_tar:extract/2 becomes an tuple of the status, and a list of tuples composed of the filename, and the contents of the file as a Binary for each file in the tar that was extracted.

If an error occurs on extraction to memory, for example we forget to pass the compressed option to a compressed tar file, it returns an error tuple.

erl_tar:extract("animal_sounds.tar.gz", [verbose, memory]).
% {error,eof}

There are quite a bit more options that erl_tar:extract/2 can take as well, so I highly recommend checking out the documentation for the full list of options.

–Proctor

Erlang Thursday – erl_tar:create/2

Today’s Erlang Thursday is on erl_tar:create/2.

erl_tar:create/2 creates a tar file with a given filename and adds the given list of filenames to the archive.

erl_tar:create/2 takes two arguments, the first is a filename to write to, and the second argument is a list of filenames to add to the tar file.

First, we will open up a new OS shell session and create some files to add to a new tar file.

$ echo "foo" > foo.txt
$ echo "bar" > bar.txt
$ echo "baz" > baz.txt
$ ls
bar.txt  baz.txt  foo.txt  test.tar

Now that we have some files to archive, we can open up a new erl session, and create a new tar file named test.tar.

erl_tar:create("test.tar", ["foo.txt", "bar.txt", "baz.txt"]).
% ok

That looks like it worked; so let’s go to a OS shell, and inspect the resulting file for the filename we gave to erl_tar:create/3.

$ tar -tf test.tar
foo.txt
bar.txt
baz.txt

And yes, tar can read that file and tells us that the three files we added are indeed part of the tar file.

erl_tar:create/3

Erlang also provides erl_tar:create/3 that takes a options list as it’s last argument.

We will create a new file, with the same contents, and pass in that we want this tar file to be compressed, and to be verbose with what it is doing as well.

erl_tar:create("options.tar.gz", 
               ["foo.txt", "bar.txt", "baz.txt"],
               [compressed, verbose]).
% a foo.txt
% a bar.txt
% a baz.txt
% ok

Again, let’s switch back to our OS shell, and inspect the resulting file.

$ tar -tf options.tar.gz
foo.txt
bar.txt
baz.txt

And let’s test it to see if it was considered compressed by gzip.

$ gzip --test options.tar.gz
$

And there we go, gzip considers this a compressed file with integrity. So let’s take a look at the size difference between the two tar files we created.

$ ls -l test.tar options.tar.gz
-rw-r--r--  1 -------  -----    154 Sep XX HH:MM options.tar.gz
-rw-r--r--  1 -------  -----  10240 Sep XX HH:MM test.tar

And looking at the filesize we can see that it is definately compressed, as options.tar.gz is two orders of magnitude smaller than test.tar.

Creating a file that already exists

As we just created test.tar and saw it had the contents, let’s see what happens when we call create on a file that already exists, by passing the same filename with a empty list of files.

erl_tar:create("test.tar", []).
% ok

And we take a look at the contents, we can see the original tar has been replaced.

$ tar -tf test.tar
$

This tells us that erl_tar:create/2 will create a tar file and overwrite the existing file, and doesn’t error out if the file already exists (assuming the user the shell is running has access to write to that file/directory).

Creating a tar for a path that doesn’t exist

If we give a bad path for a file, we can see that erl_tar:create/2 will return a error tuple, with the filename and reason for the failure.

erl_tar:create('/path/does/not/exist.tar', []).
% {error,{'/path/does/not/exist.tar',enoent}}

Other Potential Gotchas

First, the documentation states that it takes filename()s as arguments, but the documentation page for erl_tar does not specify on that page what a filename data type is.

If you use atom()s for the filename, you are going to get an error like the one below that I was getting at first, before using string()s for the filenames.

erl_tar:create('test.tar', ['foo.txt', 'bar.txt', 'baz.txt']).
** exception error: no function clause matching filename:join([]) (filename.erl, line 392)
     in function  erl_tar:split_filename/4 (erl_tar.erl, line 423)
     in call from erl_tar:create_header/3 (erl_tar.erl, line 352)
     in call from erl_tar:add1/4 (erl_tar.erl, line 305)
     in call from erl_tar:foreach_while_ok/2 (erl_tar.erl, line 940)
     in call from erl_tar:create/3 (erl_tar.erl, line 114)

Second, according to the Limitations section of the erl_tar documentation page, filenames should be less than 100 characters for maximum compatability across different systems and version of the tar program.

Lastly, it is on us the user to include the file extension when specifing the filename, as erl_tar:compress/2 does not manage the extension for us.

–Proctor

Erlang Thursday – erl_tidy:file/1

Today’s Erlang Thursday takes a look at the erl_tidy module in Erlang, and we start with erl_tidy/1.

erl_tidy:file/1 takes a filename, and “tidies” up and pretty prints the sourcecode in the file specified.

As I have a quick FizzBuzz solution hanging around in a tmp directory, I will just start with that file, and see what erl_tidy:file/1 does to that file, as I am pretty sure it is not completely pretty.

-module(fizzbuzz).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Translations = lists:map(fun translate/1, lists:seq(1, N)),
    lists:foreach(fun(Item) -> io:format("~s~n", [Item]) end, Translations).

translate(N) when N rem 3 =:= 0 andalso N rem 5 =:= 0 ->
   'FizzBuzz';
translate(N) when N rem 3 =:= 0 ->
   'Fizz';
translate(N) when N rem 5 =:= 0 ->
   'Buzz';
translate(N) ->
   integer_to_list(N).

We open up a new Erlang shell and call erl_tidy:file/1 with the fizzbuzz.erl.

erl_tidy:file("fizzbuzz.erl").
% fizzbuzz.erl:6: replacing call to `lists:map/2' with a list comprehension.
% fizzbuzz.erl:6: changing application of implicit fun to direct local call.
% ok

Looks like it didn’t like the way I was using map in the original file, and it changed the way I was calling translate/1.

Let’s close out the shell, and see what the directory looks like now.

ls -l
total 16
-rw-r--r--  1 proctor  staff  402 Sep  9 22:06 fizzbuzz.erl
-rw-r--r--  1 proctor  staff  405 Sep  9 22:05 fizzbuzz.erl.bak

Let’s take a look at fizzbuzz.erl.bak and see what is in that file, mainly to confirm that it is the content that was originally in fizzbuzz.erl.

-module(fizzbuzz).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Translations = lists:map(fun translate/1, lists:seq(1, N)),
    lists:foreach(fun(Item) -> io:format("~s~n", [Item]) end, Translations).

translate(N) when N rem 3 =:= 0 andalso N rem 5 =:= 0 ->
   'FizzBuzz';
translate(N) when N rem 3 =:= 0 ->
   'Fizz';
translate(N) when N rem 5 =:= 0 ->
   'Buzz';
translate(N) ->
   integer_to_list(N).

That does indeed look like the original contents of fizzbuzz.erl.

Now let’s take a look at the updated fizzbuzz.erl file to see what the result of erl_tidy:file/1 is.

-module(fizzbuzz).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Translations = [translate(V1) || V1 <- lists:seq(1, N)],
    lists:foreach(fun (Item) -> io:format("~s~n", [Item])
		  end,
		  Translations).

translate(N) when N rem 3 =:= 0 andalso N rem 5 =:= 0 ->
    'FizzBuzz';
translate(N) when N rem 3 =:= 0 -> 'Fizz';
translate(N) when N rem 5 =:= 0 -> 'Buzz';
translate(N) -> integer_to_list(N).

As the message said, we have a list comprehension instead of map, and it is calling the translate/1 function directly, as mentioned in the two output messages.

It also put the end of the fun passed to lists:foreach/2 on a new line, and the list Translations on a new line as well.

And finally, the last three function clauses of translate/1 were combined into single line functions, and the trailing newline at the end of the file was removed.

According to the documentation page, had the fizzbuzz.eerl file had any unused functions, those would have also have been tidied up, along with “updating obsolete constructs and function calls, etc.”

All in all, a nice little tool of an Erlang application that can help your code keep a certain style and cleanliness to it.

–Proctor

Erlang Thursday – c:pid/3

Today’s Erlang Thursday is a short one on c:pid/3.

c:pid/3 takes the three different parts of a pid as its arguments, and returns a constructed Pid type corresponding to those values.

We’ll call self to get a pid that we know is good, and we can use that to compare the result of calling c:pid/3.

self().
% <0.42.0>
c:pid(0, 42, 0).
% <0.42.0>
self() =:= c:pid(0, 42, 0).
% true

Why is this useful? Sometimes when inspecting what is going on in a live system there are certain calls in Erlang that expect a pid() type, and not just the pid numbers.

c:regs().
%
% ** Registered procs on node nonode@nohost **
% Name                  Pid          Initial Call                      Reds Msgs
% application_controlle <0.7.0>      erlang:apply/2                     463    0
% code_server           <0.19.0>     erlang:apply/2                  128774    0
% erl_prim_loader       <0.3.0>      erlang:apply/2                  163760    0
% error_logger          <0.6.0>      gen_event:init_it/6                220    0
% file_server_2         <0.18.0>     file_server:init/1                 448    0
% global_group          <0.17.0>     global_group:init/1                 59    0
% global_name_server    <0.13.0>     global:init/1                       51    0
% inet_db               <0.16.0>     inet_db:init/1                     206    0
% init                  <0.0.0>      otp_ring0:start/2                 3398    0
% kernel_safe_sup       <0.27.0>     supervisor:kernel/1                 58    0
% kernel_sup            <0.11.0>     supervisor:kernel/1              49109    0
% rex                   <0.12.0>     rpc:init/1                          35    0
% standard_error        <0.21.0>     erlang:apply/2                       9    0
% standard_error_sup    <0.20.0>     supervisor_bridge:standar           41    0
% user                  <0.24.0>     group:server/3                      36    0
% user_drv              <0.23.0>     user_drv:server/2                17940    0
%
% ** Registered ports on node nonode@nohost **
% Name                  Id              Command
% ok
erlang:is_process_alive(c:pid(0, 6, 0)).
% true

So let’s see what happens when we try to feed it something to break it, but in a meaningful way.

c:pid(0, 0, 0).
% <0.0.0>
c:pid(111110, 0, 1111110).
% ** exception error: bad argument
%      in function  list_to_pid/1
%         called as list_to_pid("<111110.0.1111110>")
%      in call from c:pid/3 (c.erl, line 424)

So it looks like the pid <0.0.0> is a valid pid, but when we throw it something else, we see it is trying to call list_to_pid.

So let’s take a quick look at list_to_pid.

erlang:list_to_pid("<0.42.0>").
% <0.42.0>
c:pid(0, 42, 0) =:= erlang:list_to_pid("<0.42.0>").
% true

So it looks like c:pid/3 is a wrapper function around list_to_pid that builds the 3 parts of a pid into a string, and then gets a pid() type from that call.

–Proctor

Erlang Thursday – user_default

Today’s Erlang Thursday takes a look at user_default.

I stumbled across this when trying to verify what version of c/1 and l/1 was used when called in the shell, and came across shell_default documentation.

It mentioned that if there are functions that we would like to have available in the shell, we can have a module user_default and specify its location in our .erlang file in our home directory.

Since I have a fizzbuzz example in Erlang handy, lets start out with that for our user_default module as proof that we can get it working. And we’ll put it in tmp under our home directory.

-module(user_default).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Translations = lists:map(fun translate/1, lists:seq(1, N)),
    lists:foreach(fun(Item) -> io:format("~s~n", [Item]) end, Translations).

translate(N) when N rem 3 =:= 0 andalso N rem 5 =:= 0 ->
   'FizzBuzz';
translate(N) when N rem 3 =:= 0 ->
   'Fizz';
translate(N) when N rem 5 =:= 0 ->
   'Buzz';
translate(N) ->
   integer_to_list(N).

Let’s open up a new .erlang file in our home directory, and add the following line, pointing to the location of the user_default file we just created.

code:load_abs("tmp/user_default").

If you already have a .erlang file, then the call to code:load_abs/1 for the user_default module must be the first line in the file.

Time to ensure both files are saved, and open up the erlang shell, and try calling fizzbuzz/1.

$ erl
Erlang/OTP 17 [erts-6.2.1]  [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V6.2.1  (abort with ^G)
1> fizzbuzz(20).
** exception error: undefined shell command fizzbuzz/1
2> q().
ok

Doesn’t seem to be working as advertised. Realizing that this is load_abs, and user_default is a new module, the error is probably because there is no file to load. Let’s go compile the module using erlc, and try again.

$ cd tmp/
$ erlc user_default.erl
$ ls user_default.*
user_default.beam user_default.erl
$ cd ..

We now have a BEAM file there, so let’s startup the Erlang shell again and try to call fizzbuzz/1.

$ erl
Erlang/OTP 17 [erts-6.2.1]  [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V6.2.1  (abort with ^G)
1> fizzbuzz(20).
1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16
17
Fizz
19
Buzz
ok
2>

And it works! We now have fizzbuzz/1 available for use in the shell without needing to specify a module.

For more information on the .erlang configuration file, check out the Configuration section of the Erlang Getting Started documentation.

–Proctor

Erlang Thursday – c:i/0

Today’s Erlang Thursday continues with another function in the c module, c:i/0.

c:i/0 reports the system information, displaying information about all the processes on a given node.

c:i().
% Pid                   Initial Call                          Heap     Reds Msgs
% Registered            Current Function                     Stack
% <0.0.0>               otp_ring0:start/2                      987     4987    0
% init                  init:loop/1                              2
% <0.3.0>               erlang:apply/2                        6772   823443    0
% erl_prim_loader       erl_prim_loader:loop/3                   6
% <0.6.0>               gen_event:init_it/6                    376      220    0
% error_logger          gen_event:fetch_msg/5                    8
% <0.7.0>               erlang:apply/2                        1598      463    0
% application_controlle gen_server:loop/6                        7
% <0.9.0>               application_master:init/4              376       44    0
%                       application_master:main_loop/2           6
% <0.10.0>              application_master:start_it/4          233       69    0
%                       application_master:loop_it/4             5
% <0.11.0>              supervisor:kernel/1                   4185    49109    0
% kernel_sup            gen_server:loop/6                        9
% <0.12.0>              rpc:init/1                             233       35    0
% rex                   gen_server:loop/6                        9
% <0.13.0>              global:init/1                          233       51    0
% global_name_server    gen_server:loop/6                        9
% <0.14.0>              erlang:apply/2                         233       19    0
%                       global:loop_the_locker/1                 5
% <0.15.0>              erlang:apply/2                         233        3    0
%                       global:loop_the_registrar/0              2
% <0.16.0>              inet_db:init/1                         233      206    0
% inet_db               gen_server:loop/6                        9
% <0.17.0>              global_group:init/1                    233       59    0
% global_group          gen_server:loop/6                        9
% <0.18.0>              file_server:init/1                    2586     2562    0
% file_server_2         gen_server:loop/6                        9
% <0.19.0>              erlang:apply/2                        2586   155919    0
% code_server           code_server:loop/1                       3
% <0.20.0>              supervisor_bridge:standard_error/      233       41    0
% standard_error_sup    gen_server:loop/6                        9
% <0.21.0>              erlang:apply/2                         233        9    0
% standard_error        standard_error:server_loop/1             2
% <0.22.0>              supervisor_bridge:user_sup/1           610       87    0
%                       gen_server:loop/6                        9
% <0.23.0>              erlang:apply/2                         233       24    0
% user                  user:server_loop/2                       5
% <0.24.0>              kernel_config:init/1                   233      286    0
%                       gen_server:loop/6                        9
% <0.25.0>              supervisor:kernel/1                    233       58    0
% kernel_safe_sup       gen_server:loop/6                        9
% <0.29.0>              kjell_profile:init/1                   987    27100    0
% kjell_profile         gen_server:loop/6                        9
% <0.30.0>              kjell_extension:init/1                2586     3903    0
% kjell_extension       gen_server:loop/6                        9
% <0.45.0>              k_user_drv:server/2                    987     2218    0
% user_drv              k_user_drv:server_loop/5                 8
% <0.46.0>              k_group:server/3                       987    14541    0
%                       k_group:server_loop/3                    4
% <0.47.0>              erlang:apply/2                       28690     4406    0
%                       kjell:shell_rep/4                       17
% <0.48.0>              erlang:apply/2                        1598    20585    0
%                       c:pinfo/1                               49
% Total                                                      58707  1110447    0
%                                                              237
% ok

We can see that it returns the process id (pid), the initial function that started the process, the size of the heap, the number of reductions, the number of messages in the message queue, the registered name, the current function the process is in, and the stack size.

c:i/0 also includes a total of the total heap size, reductions, message queue size and stack size.

There c module also includes a c:ni/0, that reports the system information above across all nodes.

Looking at the process info, we can find a couple of processes related to kjell, which I used instead of erl for ease of finding their process information.

Looking at one of the kjell related processes, we can grab that pid, and do a deeper inspection of that process by calling c:i/3, which displays information about a process, but can reference the pid by the 3 integer values that make up the process’s pid.

c:i(0, 47, 0).
% [{current_function,{kjell,shell_rep,4}},
%  {initial_call,{erlang,apply,2}},
%  {status,waiting},
%  {message_queue_len,0},
%  {messages,[]},
%  {links,[<0.48.0>,<0.46.0>]},
%  {dictionary,[{{result,1},ok},
%               {{command,1},[{call,1,{remote,1,{atom,1,c},{atom,1,i}},[]}]},
%               {evaluator,<0.48.0>}]},
%  {trap_exit,true},
%  {error_handler,error_handler},
%  {priority,normal},
%  {group_leader,<0.46.0>},
%  {total_heap_size,46421},
%  {heap_size,28690},
%  {stack_size,17},
%  {reductions,4479},
%  {garbage_collection,[{min_bin_vheap_size,46422},
%                       {min_heap_size,233},
%                       {fullsweep_after,65535},
%                       {minor_gcs,2}]},
%  {suspending,[]}]

When looking at the information related to a specific process, we can see it shows the linked process, messages and message queue length, heap and stack information, ad various other settings that could be of use.

Again, this is not as pretty as the observer application, but they come in handy when you don’t have access past a terminal, as in working from a jump box, but it gives you a good idea of what is going on in your running Erlang node.

–Proctor

Erlang Thursday – c:regs/0

Today’s Erlang Thursday continues looking at the c module, and looks at c:regs/0.

c:regs/0 displays information about the registered processes on the current node, such as process name, process id, the number of reductions performed, and more.

c:regs().
# 
# ** Registered procs on node nonode@nohost **
# Name                  Pid          Initial Call                      Reds Msgs
# application_controlle <0.7.0>      erlang:apply/2                     463    0
# code_server           <0.19.0>     erlang:apply/2                  121202    0
# erl_prim_loader       <0.3.0>      erlang:apply/2                  156994    0
# error_logger          <0.6.0>      gen_event:init_it/6                220    0
# file_server_2         <0.18.0>     file_server:init/1                  92    0
# global_group          <0.17.0>     global_group:init/1                 59    0
# global_name_server    <0.13.0>     global:init/1                       51    0
# inet_db               <0.16.0>     inet_db:init/1                     206    0
# init                  <0.0.0>      otp_ring0:start/2                 3398    0
# kernel_safe_sup       <0.28.0>     supervisor:kernel/1                 58    0
# kernel_sup            <0.11.0>     supervisor:kernel/1              49109    0
# rex                   <0.12.0>     rpc:init/1                          35    0
# standard_error        <0.21.0>     erlang:apply/2                       9    0
# standard_error_sup    <0.20.0>     supervisor_bridge:standar           41    0
# user                  <0.24.0>     group:server/3                      36    0
# user_drv              <0.23.0>     user_drv:server/2                 1219    0
# 
# ** Registered ports on node nonode@nohost **
# Name                  Id              Command
# ok

While this is not quite as nice as what is provided by the observer GUI, this is a useful tool to be able to get an idea what what the processes are, and what they are doing when you are not able to have the observer GUI running.

The c module also contains a function c:nregs/0 which displays information about all processes for all of the nodes that the node it is run from knows about.

node().
# 'foo@127.0.0.1'
nodes().
# ['bar@127.0.0.1']
c:nregs().
# 
# ** Registered procs on node 'foo@127.0.0.1' **
# Name                  Pid          Initial Call                      Reds Msgs
# application_controlle <0.7.0>      erlang:apply/2                     463    0
# auth                  <0.19.0>     auth:init/1                        880    0
# code_server           <0.25.0>     erlang:apply/2                  122302    0
# erl_epmd              <0.18.0>     erl_epmd:init/1                    268    0
# erl_prim_loader       <0.3.0>      erlang:apply/2                  163458    0
# error_logger          <0.6.0>      gen_event:init_it/6                264    0
# file_server_2         <0.24.0>     file_server:init/1                  92    0
# global_group          <0.23.0>     global_group:init/1                 67    0
# global_name_server    <0.13.0>     global:init/1                      339    0
# inet_db               <0.16.0>     inet_db:init/1                     255    0
# init                  <0.0.0>      otp_ring0:start/2                 5405    0
# kernel_safe_sup       <0.34.0>     supervisor:kernel/1                 58    0
# kernel_sup            <0.11.0>     supervisor:kernel/1              57226    0
# net_kernel            <0.20.0>     net_kernel:init/1                  792    0
# net_sup               <0.17.0>     supervisor:erl_distributi          285    0
# rex                   <0.12.0>     rpc:init/1                          35    0
# standard_error        <0.27.0>     erlang:apply/2                       9    0
# standard_error_sup    <0.26.0>     supervisor_bridge:standar           41    0
# user                  <0.30.0>     group:server/3                      36    0
# user_drv              <0.29.0>     user_drv:server/2                 1661    0
# 
# ** Registered ports on node 'foo@127.0.0.1' **
# Name                  Id              Command
# 
# ** Registered procs on node 'bar@127.0.0.1' **
# Name                  Pid          Initial Call                      Reds Msgs
# application_controlle <6108.7.0>   erlang:apply/2                     463    0
# auth                  <6108.19.0>  auth:init/1                        880    0
# code_server           <6108.25.0>  erlang:apply/2                  124588    0
# erl_epmd              <6108.18.0>  erl_epmd:init/1                    268    0
# erl_prim_loader       <6108.3.0>   erlang:apply/2                  164400    0
# error_logger          <6108.6.0>   gen_event:init_it/6                301    0
# file_server_2         <6108.24.0>  file_server:init/1                  92    0
# global_group          <6108.23.0>  global_group:init/1                 67    0
# global_name_server    <6108.13.0>  global:init/1                      341    0
# inet_db               <6108.16.0>  inet_db:init/1                     255    0
# inet_gethost_native   <6108.42.0>  inet_gethost_native:serve           83    0
# inet_gethost_native_s <6108.41.0>  supervisor_bridge:inet_ge           41    0
# init                  <6108.0.0>   otp_ring0:start/2                 5515    0
# kernel_safe_sup       <6108.34.0>  supervisor:kernel/1                127    0
# kernel_sup            <6108.11.0>  supervisor:kernel/1              57226    0
# net_kernel            <6108.20.0>  net_kernel:init/1                  796    0
# net_sup               <6108.17.0>  supervisor:erl_distributi          285    0
# rex                   <6108.12.0>  rpc:init/1                        1302    0
# standard_error        <6108.27.0>  erlang:apply/2                       9    0
# standard_error_sup    <6108.26.0>  supervisor_bridge:standar           41    0
# user                  <6108.30.0>  group:server/3                      36    0
# user_drv              <6108.29.0>  user_drv:server/2                 2801    0
# 
# ** Registered ports on node 'bar@127.0.0.1' **
# Name                  Id              Command
# ok

–Proctor

A List of Questions to Address Before Creating a Microservice

Last Friday, I got a meeting invite at work to discuss creating a new microservice as part of our application.

Whether it is Unix style programs, Domain Driven Design bounded contexts, classes and objects that adhere to the Single Responsibility Principle, I have been a supporter of small focused applications that have a single job to do, and do it well. This is one of the things that has appealed to me about Erlang as I have been digging ever deeper into it.

I will be first to step forward and promote the idea of microservices, but I will also be the first to come across like I don’t support them.

These questions are to make sure that proper thought is given to the implications of creating a microservice architecture, so that we don’t shoot ourselves in the foot and become the case study of why microservices are just a bunch of hot air, instead of being a case study for why and how it can work.

These questions are likely applicable to any new application, and not just microservices, and are inspired by the 8 Fallacies of Distributed Computing, the book Enterprise Integration Patterns, Domain Driven Design, my learning path with Erlang, and too many more to be named.

In no particular order at this point, but the general order at which they came into my head, here are the questions we should ask ourselves to help determine if a microservice is a good idea.

  • What other information do this service need from other parts of the system? Is this truly a vertical slice of a domain?
  • What other outside systems do we depend on for this service?
  • What happens if one of the services dependencies is unavailable?
  • How do we know if this service is running? Generating errors?
  • Which parts of the system will be consuming the service?
  • How to we abort without taking down the consumers of this service?
  • What does the size of the request look like?
  • What does the size of the response look like?
  • What is the latency of this service?
    • What is the latency of just returning a 200 OK with a hardcoded return value.
    • What is the expected latency of processing a full request?
  • What is the expected SLA of the service?
    • How do we expect to meet that SLA?
    • What is the SLA for uptime?
    • What is the SLA for response time? Average response time? 95th percentile? 99th percentile? Worst case?
  • What is our default response to return if we are about to break the SLA?
  • Are we expecting this service to be exposed to the outside world? Live within an isolated network?
  • Do we need authentication?
  • Who would be authorized to consume this service?
  • How are we expecting to manage access to this service?
  • Do we need to encrypt the data exchange?
  • What internal storage/persistance mechanism(s) do we need as part of this service to keep it isolated?
  • How many Requests per Second are we expecting this service to need to serve?
  • How do we expect this service to be deployed? What deployment dependencies are we expecting to need?
  • How frequently do we expect this service to need to be updated after deployment?
  • How many instances of this service do we think we will need to have running?
  • How do we coordinate information exchange between multiple instances of the service?
  • What is the expected time between a change notification and a consistent view of the system?
  • If any one instance of the service in a cluster fails, do the rest fail?
    • How do we keep the other instances from failing?
    • How does an instance of the service catch back up to the latest state once it has recovered?
  • If part of the service cluster fails, can we safely and automatically restart that part of the cluster?
  • How many failures in a time period do we allow before escalating a larger issue?
    • What is that time period?
    • How do we escalate issues?
  • How do we expect these larger issues to be addressed?
  • What does it take to start the service from an empty slate?
  • What does it take to stop the service?
  • Can we have multiple versions of the service deployed and serving requests at the same time?
  • How do we know what instance of the service served a request?
  • What is the strategy to resolve the service endpoint from a blank
  • What is the expected communication medium/protocol/payload we expect to be using to communicate with this service?
    • Message bus channel subscriptions? HTTP requests? REST “proper” with Hypermedia? “Dumb” REST? JSON payload? XML payload? Protobuff payloads?
  • How do we expect load to be distributed between any instances of this service?
  • When making a request to an outside service, what do we do when awaiting a response?
    • Block? Start processing another request? Do something that is not I/O based?
  • How are we expecting to manage versioning of the APIs that this service is expect to provide?
  • Does this service need to respond to incoming calls/notifications?
  • If this service does need to respond, it is expected to be synchronous, “appear” synchronous, or be completely asynchronous style of response?
  • If asynchronous responses are expected, how does the service get the information it needs to know to where to send the response to?
  • How do we expect to trace a flow between work and the requests and responses that triggered that work? Is there a way to trace causality?
  • What is the minimum infrastructure/frameworks that is needed to provide the service?
    • Is this a service? Microservice? Additional monolithic application?
  • What is the problem domain (bounded context) of this service?
    • How do we know when we are adding features that should belong in other services?
  • How many requests are we expecting are needed to complete a business use case?
    • Is there any way to shrink that number? Can requests be combined?

This is by no means a complete list of questions we should be asking ourselves, but the start of a conversation to understand the scope of what it takes for a new service to be created and deployed. These are my brain dump of questions that help a team know if they know how to swim, and how deep the water is, before diving head first into the sea of microservices.

Let me know what other questions you think are missing.

–Proctor

Erlang Thursday Bonus – Functional fizzbuzz

A bonus Erlang Thursday for everyone today.

This past weekend I came across the post Bro, Do You Even FizzBuzz?!? about solving FizzBuzz in Clojure without using the modulus operator.

After translating it to Ruby as a point for a co-worker and publishing that translation, I thought I would translate it to Erlang as well to see the difference.

-module(fizzbuzz).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Results = do_fizzbuzz(N),
    lists:foreach(fun(X) -> io:format("~p~n", [X]) end, Results).

do_fizzbuzz(N) ->
    Fizzes = cycle(["", "", "fizz"], N),
    Buzzes = cycle(["", "", "", "", "buzz"], N),
    FizzBuzzes = lists:zipwith(fun lists:append/2, Fizzes, Buzzes),
    Numbers = lists:seq(1, N),
    lists:zipwith(fun translate/2, Numbers, FizzBuzzes).

cycle(List, N) ->
    lists:sublist(lists:append(lists:duplicate(N, List)), N).

translate(Number, "") ->
    integer_to_list(Number);
translate(_, Translation) ->
    Translation.

A couple of points to note in the Erlang solution.

First, Erlang doesn’t have any direct concept of lazy lists/sequences, and there is no cycle function, so I had to improvise by calling lists:duplicate, lists:append, and lists:sublist a list that cycles over the source list to create a list that has N elements. While this is not exactly the most efficient way to do this, it shows that it can be done.

Second, the ability of using lists:zipwith helps with some of the ideas of pipelining, as we can process the items as they are zipped together, instead of having to process them as a different step.

Third, instead of using a case statement, we can use a function with a guard clause to determine if a translation exists or not, and use the translation if so, and the number if not.

I hope this gives you some food for thought, and would love to hear your feedback on how this could be improved even more, or other ways that FizzBuzz can be done besides the usual pattern matching with the guard clauses checking the remainders.

–Proctor

Erlang Thursday – c:m/1

Today’s Erlang Thursday continues to take a look at the c module with c:m/1.

c:m/1 takes an atom of a module name, and returns information about the module. It prints out information about the compliation date, time, and options; the object (BEAM) file that it was loaded from, and a list of functions exported by the module.

We’ll start with taking a look at the string module in Erlang.

c:m(string).
% Module string compiled: Date: November 28 2014, Time: 06.47
% Compiler options:  [{outdir,"/private/tmp/erlang-pY1Kv2/otp-OTP-17.3.4/lib/stdlib/src/../ebin"},
%                     {i,"/private/tmp/erlang-pY1Kv2/otp-OTP-17.3.4/lib/stdlib/src/../include"},
%                     {i,"/private/tmp/erlang-pY1Kv2/otp-OTP-17.3.4/lib/stdlib/src/../../kernel/include"},
%                     warnings_as_errors,debug_info]
% Object file: /usr/local/Cellar/erlang/17.3.4/lib/erlang/lib/stdlib-2.2/ebin/string.beam
% Exports:
% centre/2                      rstr/2
% centre/3                      span/2
% chars/3                       str/2
% chars/2                       strip/1
% chr/2                         strip/2
% concat/2                      strip/3
% copies/2                      sub_string/2
% cspan/2                       sub_string/3
% equal/2                       sub_word/2
% join/2                        sub_word/3
% left/2                        substr/2
% left/3                        substr/3
% len/1                         to_float/1
% module_info/0                 to_integer/1
% module_info/1                 to_lower/1
% rchr/2                        to_upper/1
% right/2                       tokens/2
% right/3                       words/1
%                               words/2
% ok

We can see that this was compiled on my machine on November 28th of 2014, and had the warnings_as_errors and debug_info turned on, as well as the location of the beam file, and all of the different functions the string module exports.

Next, we will look at a module compiled from inside the shell.

c(fizzbuzz).
% {ok,fizzbuzz}
c:m(fizzbuzz).
% Module fizzbuzz compiled: Date: August 5 2015, Time: 22.14
% Compiler options:  []
% Object file: /Users/proctor/tmp/fizzbuzz.beam
% Exports:
%          fizzbuzz/1
%          module_info/0
%          module_info/1
% ok

c:m(fizzbuzz) shows that it was compiled, and was loaded from my tmp directory, and exports fizzbuzz/1 along with the two versions of module_info that every module exports.

Again, this is one of those functions that you might not use everyday, but when it comes to debugging and inspecting your Erlang application becomes a very useful function to know about.

–Proctor