Tag Archives: Erlang

Erlang Thursday – erl_tar:create/2

Today’s Erlang Thursday is on erl_tar:create/2.

erl_tar:create/2 creates a tar file with a given filename and adds the given list of filenames to the archive.

erl_tar:create/2 takes two arguments, the first is a filename to write to, and the second argument is a list of filenames to add to the tar file.

First, we will open up a new OS shell session and create some files to add to a new tar file.

$ echo "foo" > foo.txt
$ echo "bar" > bar.txt
$ echo "baz" > baz.txt
$ ls
bar.txt  baz.txt  foo.txt  test.tar

Now that we have some files to archive, we can open up a new erl session, and create a new tar file named test.tar.

erl_tar:create("test.tar", ["foo.txt", "bar.txt", "baz.txt"]).
% ok

That looks like it worked; so let’s go to a OS shell, and inspect the resulting file for the filename we gave to erl_tar:create/3.

$ tar -tf test.tar
foo.txt
bar.txt
baz.txt

And yes, tar can read that file and tells us that the three files we added are indeed part of the tar file.

erl_tar:create/3

Erlang also provides erl_tar:create/3 that takes a options list as it’s last argument.

We will create a new file, with the same contents, and pass in that we want this tar file to be compressed, and to be verbose with what it is doing as well.

erl_tar:create("options.tar.gz", 
               ["foo.txt", "bar.txt", "baz.txt"],
               [compressed, verbose]).
% a foo.txt
% a bar.txt
% a baz.txt
% ok

Again, let’s switch back to our OS shell, and inspect the resulting file.

$ tar -tf options.tar.gz
foo.txt
bar.txt
baz.txt

And let’s test it to see if it was considered compressed by gzip.

$ gzip --test options.tar.gz
$

And there we go, gzip considers this a compressed file with integrity. So let’s take a look at the size difference between the two tar files we created.

$ ls -l test.tar options.tar.gz
-rw-r--r--  1 -------  -----    154 Sep XX HH:MM options.tar.gz
-rw-r--r--  1 -------  -----  10240 Sep XX HH:MM test.tar

And looking at the filesize we can see that it is definately compressed, as options.tar.gz is two orders of magnitude smaller than test.tar.

Creating a file that already exists

As we just created test.tar and saw it had the contents, let’s see what happens when we call create on a file that already exists, by passing the same filename with a empty list of files.

erl_tar:create("test.tar", []).
% ok

And we take a look at the contents, we can see the original tar has been replaced.

$ tar -tf test.tar
$

This tells us that erl_tar:create/2 will create a tar file and overwrite the existing file, and doesn’t error out if the file already exists (assuming the user the shell is running has access to write to that file/directory).

Creating a tar for a path that doesn’t exist

If we give a bad path for a file, we can see that erl_tar:create/2 will return a error tuple, with the filename and reason for the failure.

erl_tar:create('/path/does/not/exist.tar', []).
% {error,{'/path/does/not/exist.tar',enoent}}

Other Potential Gotchas

First, the documentation states that it takes filename()s as arguments, but the documentation page for erl_tar does not specify on that page what a filename data type is.

If you use atom()s for the filename, you are going to get an error like the one below that I was getting at first, before using string()s for the filenames.

erl_tar:create('test.tar', ['foo.txt', 'bar.txt', 'baz.txt']).
** exception error: no function clause matching filename:join([]) (filename.erl, line 392)
     in function  erl_tar:split_filename/4 (erl_tar.erl, line 423)
     in call from erl_tar:create_header/3 (erl_tar.erl, line 352)
     in call from erl_tar:add1/4 (erl_tar.erl, line 305)
     in call from erl_tar:foreach_while_ok/2 (erl_tar.erl, line 940)
     in call from erl_tar:create/3 (erl_tar.erl, line 114)

Second, according to the Limitations section of the erl_tar documentation page, filenames should be less than 100 characters for maximum compatability across different systems and version of the tar program.

Lastly, it is on us the user to include the file extension when specifing the filename, as erl_tar:compress/2 does not manage the extension for us.

–Proctor

Erlang Thursday – erl_tidy:file/1

Today’s Erlang Thursday takes a look at the erl_tidy module in Erlang, and we start with erl_tidy/1.

erl_tidy:file/1 takes a filename, and “tidies” up and pretty prints the sourcecode in the file specified.

As I have a quick FizzBuzz solution hanging around in a tmp directory, I will just start with that file, and see what erl_tidy:file/1 does to that file, as I am pretty sure it is not completely pretty.

-module(fizzbuzz).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Translations = lists:map(fun translate/1, lists:seq(1, N)),
    lists:foreach(fun(Item) -> io:format("~s~n", [Item]) end, Translations).

translate(N) when N rem 3 =:= 0 andalso N rem 5 =:= 0 ->
   'FizzBuzz';
translate(N) when N rem 3 =:= 0 ->
   'Fizz';
translate(N) when N rem 5 =:= 0 ->
   'Buzz';
translate(N) ->
   integer_to_list(N).

We open up a new Erlang shell and call erl_tidy:file/1 with the fizzbuzz.erl.

erl_tidy:file("fizzbuzz.erl").
% fizzbuzz.erl:6: replacing call to `lists:map/2' with a list comprehension.
% fizzbuzz.erl:6: changing application of implicit fun to direct local call.
% ok

Looks like it didn’t like the way I was using map in the original file, and it changed the way I was calling translate/1.

Let’s close out the shell, and see what the directory looks like now.

ls -l
total 16
-rw-r--r--  1 proctor  staff  402 Sep  9 22:06 fizzbuzz.erl
-rw-r--r--  1 proctor  staff  405 Sep  9 22:05 fizzbuzz.erl.bak

Let’s take a look at fizzbuzz.erl.bak and see what is in that file, mainly to confirm that it is the content that was originally in fizzbuzz.erl.

-module(fizzbuzz).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Translations = lists:map(fun translate/1, lists:seq(1, N)),
    lists:foreach(fun(Item) -> io:format("~s~n", [Item]) end, Translations).

translate(N) when N rem 3 =:= 0 andalso N rem 5 =:= 0 ->
   'FizzBuzz';
translate(N) when N rem 3 =:= 0 ->
   'Fizz';
translate(N) when N rem 5 =:= 0 ->
   'Buzz';
translate(N) ->
   integer_to_list(N).

That does indeed look like the original contents of fizzbuzz.erl.

Now let’s take a look at the updated fizzbuzz.erl file to see what the result of erl_tidy:file/1 is.

-module(fizzbuzz).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Translations = [translate(V1) || V1 <- lists:seq(1, N)],
    lists:foreach(fun (Item) -> io:format("~s~n", [Item])
		  end,
		  Translations).

translate(N) when N rem 3 =:= 0 andalso N rem 5 =:= 0 ->
    'FizzBuzz';
translate(N) when N rem 3 =:= 0 -> 'Fizz';
translate(N) when N rem 5 =:= 0 -> 'Buzz';
translate(N) -> integer_to_list(N).

As the message said, we have a list comprehension instead of map, and it is calling the translate/1 function directly, as mentioned in the two output messages.

It also put the end of the fun passed to lists:foreach/2 on a new line, and the list Translations on a new line as well.

And finally, the last three function clauses of translate/1 were combined into single line functions, and the trailing newline at the end of the file was removed.

According to the documentation page, had the fizzbuzz.eerl file had any unused functions, those would have also have been tidied up, along with “updating obsolete constructs and function calls, etc.”

All in all, a nice little tool of an Erlang application that can help your code keep a certain style and cleanliness to it.

–Proctor

Erlang Thursday – c:pid/3

Today’s Erlang Thursday is a short one on c:pid/3.

c:pid/3 takes the three different parts of a pid as its arguments, and returns a constructed Pid type corresponding to those values.

We’ll call self to get a pid that we know is good, and we can use that to compare the result of calling c:pid/3.

self().
% <0.42.0>
c:pid(0, 42, 0).
% <0.42.0>
self() =:= c:pid(0, 42, 0).
% true

Why is this useful? Sometimes when inspecting what is going on in a live system there are certain calls in Erlang that expect a pid() type, and not just the pid numbers.

c:regs().
%
% ** Registered procs on node nonode@nohost **
% Name                  Pid          Initial Call                      Reds Msgs
% application_controlle <0.7.0>      erlang:apply/2                     463    0
% code_server           <0.19.0>     erlang:apply/2                  128774    0
% erl_prim_loader       <0.3.0>      erlang:apply/2                  163760    0
% error_logger          <0.6.0>      gen_event:init_it/6                220    0
% file_server_2         <0.18.0>     file_server:init/1                 448    0
% global_group          <0.17.0>     global_group:init/1                 59    0
% global_name_server    <0.13.0>     global:init/1                       51    0
% inet_db               <0.16.0>     inet_db:init/1                     206    0
% init                  <0.0.0>      otp_ring0:start/2                 3398    0
% kernel_safe_sup       <0.27.0>     supervisor:kernel/1                 58    0
% kernel_sup            <0.11.0>     supervisor:kernel/1              49109    0
% rex                   <0.12.0>     rpc:init/1                          35    0
% standard_error        <0.21.0>     erlang:apply/2                       9    0
% standard_error_sup    <0.20.0>     supervisor_bridge:standar           41    0
% user                  <0.24.0>     group:server/3                      36    0
% user_drv              <0.23.0>     user_drv:server/2                17940    0
%
% ** Registered ports on node nonode@nohost **
% Name                  Id              Command
% ok
erlang:is_process_alive(c:pid(0, 6, 0)).
% true

So let’s see what happens when we try to feed it something to break it, but in a meaningful way.

c:pid(0, 0, 0).
% <0.0.0>
c:pid(111110, 0, 1111110).
% ** exception error: bad argument
%      in function  list_to_pid/1
%         called as list_to_pid("<111110.0.1111110>")
%      in call from c:pid/3 (c.erl, line 424)

So it looks like the pid <0.0.0> is a valid pid, but when we throw it something else, we see it is trying to call list_to_pid.

So let’s take a quick look at list_to_pid.

erlang:list_to_pid("<0.42.0>").
% <0.42.0>
c:pid(0, 42, 0) =:= erlang:list_to_pid("<0.42.0>").
% true

So it looks like c:pid/3 is a wrapper function around list_to_pid that builds the 3 parts of a pid into a string, and then gets a pid() type from that call.

–Proctor

Erlang Thursday – user_default

Today’s Erlang Thursday takes a look at user_default.

I stumbled across this when trying to verify what version of c/1 and l/1 was used when called in the shell, and came across shell_default documentation.

It mentioned that if there are functions that we would like to have available in the shell, we can have a module user_default and specify its location in our .erlang file in our home directory.

Since I have a fizzbuzz example in Erlang handy, lets start out with that for our user_default module as proof that we can get it working. And we’ll put it in tmp under our home directory.

-module(user_default).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Translations = lists:map(fun translate/1, lists:seq(1, N)),
    lists:foreach(fun(Item) -> io:format("~s~n", [Item]) end, Translations).

translate(N) when N rem 3 =:= 0 andalso N rem 5 =:= 0 ->
   'FizzBuzz';
translate(N) when N rem 3 =:= 0 ->
   'Fizz';
translate(N) when N rem 5 =:= 0 ->
   'Buzz';
translate(N) ->
   integer_to_list(N).

Let’s open up a new .erlang file in our home directory, and add the following line, pointing to the location of the user_default file we just created.

code:load_abs("tmp/user_default").

If you already have a .erlang file, then the call to code:load_abs/1 for the user_default module must be the first line in the file.

Time to ensure both files are saved, and open up the erlang shell, and try calling fizzbuzz/1.

$ erl
Erlang/OTP 17 [erts-6.2.1]  [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V6.2.1  (abort with ^G)
1> fizzbuzz(20).
** exception error: undefined shell command fizzbuzz/1
2> q().
ok

Doesn’t seem to be working as advertised. Realizing that this is load_abs, and user_default is a new module, the error is probably because there is no file to load. Let’s go compile the module using erlc, and try again.

$ cd tmp/
$ erlc user_default.erl
$ ls user_default.*
user_default.beam user_default.erl
$ cd ..

We now have a BEAM file there, so let’s startup the Erlang shell again and try to call fizzbuzz/1.

$ erl
Erlang/OTP 17 [erts-6.2.1]  [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V6.2.1  (abort with ^G)
1> fizzbuzz(20).
1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16
17
Fizz
19
Buzz
ok
2>

And it works! We now have fizzbuzz/1 available for use in the shell without needing to specify a module.

For more information on the .erlang configuration file, check out the Configuration section of the Erlang Getting Started documentation.

–Proctor

Erlang Thursday – c:i/0

Today’s Erlang Thursday continues with another function in the c module, c:i/0.

c:i/0 reports the system information, displaying information about all the processes on a given node.

c:i().
% Pid                   Initial Call                          Heap     Reds Msgs
% Registered            Current Function                     Stack
% <0.0.0>               otp_ring0:start/2                      987     4987    0
% init                  init:loop/1                              2
% <0.3.0>               erlang:apply/2                        6772   823443    0
% erl_prim_loader       erl_prim_loader:loop/3                   6
% <0.6.0>               gen_event:init_it/6                    376      220    0
% error_logger          gen_event:fetch_msg/5                    8
% <0.7.0>               erlang:apply/2                        1598      463    0
% application_controlle gen_server:loop/6                        7
% <0.9.0>               application_master:init/4              376       44    0
%                       application_master:main_loop/2           6
% <0.10.0>              application_master:start_it/4          233       69    0
%                       application_master:loop_it/4             5
% <0.11.0>              supervisor:kernel/1                   4185    49109    0
% kernel_sup            gen_server:loop/6                        9
% <0.12.0>              rpc:init/1                             233       35    0
% rex                   gen_server:loop/6                        9
% <0.13.0>              global:init/1                          233       51    0
% global_name_server    gen_server:loop/6                        9
% <0.14.0>              erlang:apply/2                         233       19    0
%                       global:loop_the_locker/1                 5
% <0.15.0>              erlang:apply/2                         233        3    0
%                       global:loop_the_registrar/0              2
% <0.16.0>              inet_db:init/1                         233      206    0
% inet_db               gen_server:loop/6                        9
% <0.17.0>              global_group:init/1                    233       59    0
% global_group          gen_server:loop/6                        9
% <0.18.0>              file_server:init/1                    2586     2562    0
% file_server_2         gen_server:loop/6                        9
% <0.19.0>              erlang:apply/2                        2586   155919    0
% code_server           code_server:loop/1                       3
% <0.20.0>              supervisor_bridge:standard_error/      233       41    0
% standard_error_sup    gen_server:loop/6                        9
% <0.21.0>              erlang:apply/2                         233        9    0
% standard_error        standard_error:server_loop/1             2
% <0.22.0>              supervisor_bridge:user_sup/1           610       87    0
%                       gen_server:loop/6                        9
% <0.23.0>              erlang:apply/2                         233       24    0
% user                  user:server_loop/2                       5
% <0.24.0>              kernel_config:init/1                   233      286    0
%                       gen_server:loop/6                        9
% <0.25.0>              supervisor:kernel/1                    233       58    0
% kernel_safe_sup       gen_server:loop/6                        9
% <0.29.0>              kjell_profile:init/1                   987    27100    0
% kjell_profile         gen_server:loop/6                        9
% <0.30.0>              kjell_extension:init/1                2586     3903    0
% kjell_extension       gen_server:loop/6                        9
% <0.45.0>              k_user_drv:server/2                    987     2218    0
% user_drv              k_user_drv:server_loop/5                 8
% <0.46.0>              k_group:server/3                       987    14541    0
%                       k_group:server_loop/3                    4
% <0.47.0>              erlang:apply/2                       28690     4406    0
%                       kjell:shell_rep/4                       17
% <0.48.0>              erlang:apply/2                        1598    20585    0
%                       c:pinfo/1                               49
% Total                                                      58707  1110447    0
%                                                              237
% ok

We can see that it returns the process id (pid), the initial function that started the process, the size of the heap, the number of reductions, the number of messages in the message queue, the registered name, the current function the process is in, and the stack size.

c:i/0 also includes a total of the total heap size, reductions, message queue size and stack size.

There c module also includes a c:ni/0, that reports the system information above across all nodes.

Looking at the process info, we can find a couple of processes related to kjell, which I used instead of erl for ease of finding their process information.

Looking at one of the kjell related processes, we can grab that pid, and do a deeper inspection of that process by calling c:i/3, which displays information about a process, but can reference the pid by the 3 integer values that make up the process’s pid.

c:i(0, 47, 0).
% [{current_function,{kjell,shell_rep,4}},
%  {initial_call,{erlang,apply,2}},
%  {status,waiting},
%  {message_queue_len,0},
%  {messages,[]},
%  {links,[<0.48.0>,<0.46.0>]},
%  {dictionary,[{{result,1},ok},
%               {{command,1},[{call,1,{remote,1,{atom,1,c},{atom,1,i}},[]}]},
%               {evaluator,<0.48.0>}]},
%  {trap_exit,true},
%  {error_handler,error_handler},
%  {priority,normal},
%  {group_leader,<0.46.0>},
%  {total_heap_size,46421},
%  {heap_size,28690},
%  {stack_size,17},
%  {reductions,4479},
%  {garbage_collection,[{min_bin_vheap_size,46422},
%                       {min_heap_size,233},
%                       {fullsweep_after,65535},
%                       {minor_gcs,2}]},
%  {suspending,[]}]

When looking at the information related to a specific process, we can see it shows the linked process, messages and message queue length, heap and stack information, ad various other settings that could be of use.

Again, this is not as pretty as the observer application, but they come in handy when you don’t have access past a terminal, as in working from a jump box, but it gives you a good idea of what is going on in your running Erlang node.

–Proctor

Erlang Thursday – c:regs/0

Today’s Erlang Thursday continues looking at the c module, and looks at c:regs/0.

c:regs/0 displays information about the registered processes on the current node, such as process name, process id, the number of reductions performed, and more.

c:regs().
# 
# ** Registered procs on node nonode@nohost **
# Name                  Pid          Initial Call                      Reds Msgs
# application_controlle <0.7.0>      erlang:apply/2                     463    0
# code_server           <0.19.0>     erlang:apply/2                  121202    0
# erl_prim_loader       <0.3.0>      erlang:apply/2                  156994    0
# error_logger          <0.6.0>      gen_event:init_it/6                220    0
# file_server_2         <0.18.0>     file_server:init/1                  92    0
# global_group          <0.17.0>     global_group:init/1                 59    0
# global_name_server    <0.13.0>     global:init/1                       51    0
# inet_db               <0.16.0>     inet_db:init/1                     206    0
# init                  <0.0.0>      otp_ring0:start/2                 3398    0
# kernel_safe_sup       <0.28.0>     supervisor:kernel/1                 58    0
# kernel_sup            <0.11.0>     supervisor:kernel/1              49109    0
# rex                   <0.12.0>     rpc:init/1                          35    0
# standard_error        <0.21.0>     erlang:apply/2                       9    0
# standard_error_sup    <0.20.0>     supervisor_bridge:standar           41    0
# user                  <0.24.0>     group:server/3                      36    0
# user_drv              <0.23.0>     user_drv:server/2                 1219    0
# 
# ** Registered ports on node nonode@nohost **
# Name                  Id              Command
# ok

While this is not quite as nice as what is provided by the observer GUI, this is a useful tool to be able to get an idea what what the processes are, and what they are doing when you are not able to have the observer GUI running.

The c module also contains a function c:nregs/0 which displays information about all processes for all of the nodes that the node it is run from knows about.

node().
# 'foo@127.0.0.1'
nodes().
# ['bar@127.0.0.1']
c:nregs().
# 
# ** Registered procs on node 'foo@127.0.0.1' **
# Name                  Pid          Initial Call                      Reds Msgs
# application_controlle <0.7.0>      erlang:apply/2                     463    0
# auth                  <0.19.0>     auth:init/1                        880    0
# code_server           <0.25.0>     erlang:apply/2                  122302    0
# erl_epmd              <0.18.0>     erl_epmd:init/1                    268    0
# erl_prim_loader       <0.3.0>      erlang:apply/2                  163458    0
# error_logger          <0.6.0>      gen_event:init_it/6                264    0
# file_server_2         <0.24.0>     file_server:init/1                  92    0
# global_group          <0.23.0>     global_group:init/1                 67    0
# global_name_server    <0.13.0>     global:init/1                      339    0
# inet_db               <0.16.0>     inet_db:init/1                     255    0
# init                  <0.0.0>      otp_ring0:start/2                 5405    0
# kernel_safe_sup       <0.34.0>     supervisor:kernel/1                 58    0
# kernel_sup            <0.11.0>     supervisor:kernel/1              57226    0
# net_kernel            <0.20.0>     net_kernel:init/1                  792    0
# net_sup               <0.17.0>     supervisor:erl_distributi          285    0
# rex                   <0.12.0>     rpc:init/1                          35    0
# standard_error        <0.27.0>     erlang:apply/2                       9    0
# standard_error_sup    <0.26.0>     supervisor_bridge:standar           41    0
# user                  <0.30.0>     group:server/3                      36    0
# user_drv              <0.29.0>     user_drv:server/2                 1661    0
# 
# ** Registered ports on node 'foo@127.0.0.1' **
# Name                  Id              Command
# 
# ** Registered procs on node 'bar@127.0.0.1' **
# Name                  Pid          Initial Call                      Reds Msgs
# application_controlle <6108.7.0>   erlang:apply/2                     463    0
# auth                  <6108.19.0>  auth:init/1                        880    0
# code_server           <6108.25.0>  erlang:apply/2                  124588    0
# erl_epmd              <6108.18.0>  erl_epmd:init/1                    268    0
# erl_prim_loader       <6108.3.0>   erlang:apply/2                  164400    0
# error_logger          <6108.6.0>   gen_event:init_it/6                301    0
# file_server_2         <6108.24.0>  file_server:init/1                  92    0
# global_group          <6108.23.0>  global_group:init/1                 67    0
# global_name_server    <6108.13.0>  global:init/1                      341    0
# inet_db               <6108.16.0>  inet_db:init/1                     255    0
# inet_gethost_native   <6108.42.0>  inet_gethost_native:serve           83    0
# inet_gethost_native_s <6108.41.0>  supervisor_bridge:inet_ge           41    0
# init                  <6108.0.0>   otp_ring0:start/2                 5515    0
# kernel_safe_sup       <6108.34.0>  supervisor:kernel/1                127    0
# kernel_sup            <6108.11.0>  supervisor:kernel/1              57226    0
# net_kernel            <6108.20.0>  net_kernel:init/1                  796    0
# net_sup               <6108.17.0>  supervisor:erl_distributi          285    0
# rex                   <6108.12.0>  rpc:init/1                        1302    0
# standard_error        <6108.27.0>  erlang:apply/2                       9    0
# standard_error_sup    <6108.26.0>  supervisor_bridge:standar           41    0
# user                  <6108.30.0>  group:server/3                      36    0
# user_drv              <6108.29.0>  user_drv:server/2                 2801    0
# 
# ** Registered ports on node 'bar@127.0.0.1' **
# Name                  Id              Command
# ok

–Proctor

A List of Questions to Address Before Creating a Microservice

Last Friday, I got a meeting invite at work to discuss creating a new microservice as part of our application.

Whether it is Unix style programs, Domain Driven Design bounded contexts, classes and objects that adhere to the Single Responsibility Principle, I have been a supporter of small focused applications that have a single job to do, and do it well. This is one of the things that has appealed to me about Erlang as I have been digging ever deeper into it.

I will be first to step forward and promote the idea of microservices, but I will also be the first to come across like I don’t support them.

These questions are to make sure that proper thought is given to the implications of creating a microservice architecture, so that we don’t shoot ourselves in the foot and become the case study of why microservices are just a bunch of hot air, instead of being a case study for why and how it can work.

These questions are likely applicable to any new application, and not just microservices, and are inspired by the 8 Fallacies of Distributed Computing, the book Enterprise Integration Patterns, Domain Driven Design, my learning path with Erlang, and too many more to be named.

In no particular order at this point, but the general order at which they came into my head, here are the questions we should ask ourselves to help determine if a microservice is a good idea.

  • What other information do this service need from other parts of the system? Is this truly a vertical slice of a domain?
  • What other outside systems do we depend on for this service?
  • What happens if one of the services dependencies is unavailable?
  • How do we know if this service is running? Generating errors?
  • Which parts of the system will be consuming the service?
  • How to we abort without taking down the consumers of this service?
  • What does the size of the request look like?
  • What does the size of the response look like?
  • What is the latency of this service?
    • What is the latency of just returning a 200 OK with a hardcoded return value.
    • What is the expected latency of processing a full request?
  • What is the expected SLA of the service?
    • How do we expect to meet that SLA?
    • What is the SLA for uptime?
    • What is the SLA for response time? Average response time? 95th percentile? 99th percentile? Worst case?
  • What is our default response to return if we are about to break the SLA?
  • Are we expecting this service to be exposed to the outside world? Live within an isolated network?
  • Do we need authentication?
  • Who would be authorized to consume this service?
  • How are we expecting to manage access to this service?
  • Do we need to encrypt the data exchange?
  • What internal storage/persistance mechanism(s) do we need as part of this service to keep it isolated?
  • How many Requests per Second are we expecting this service to need to serve?
  • How do we expect this service to be deployed? What deployment dependencies are we expecting to need?
  • How frequently do we expect this service to need to be updated after deployment?
  • How many instances of this service do we think we will need to have running?
  • How do we coordinate information exchange between multiple instances of the service?
  • What is the expected time between a change notification and a consistent view of the system?
  • If any one instance of the service in a cluster fails, do the rest fail?
    • How do we keep the other instances from failing?
    • How does an instance of the service catch back up to the latest state once it has recovered?
  • If part of the service cluster fails, can we safely and automatically restart that part of the cluster?
  • How many failures in a time period do we allow before escalating a larger issue?
    • What is that time period?
    • How do we escalate issues?
  • How do we expect these larger issues to be addressed?
  • What does it take to start the service from an empty slate?
  • What does it take to stop the service?
  • Can we have multiple versions of the service deployed and serving requests at the same time?
  • How do we know what instance of the service served a request?
  • What is the strategy to resolve the service endpoint from a blank
  • What is the expected communication medium/protocol/payload we expect to be using to communicate with this service?
    • Message bus channel subscriptions? HTTP requests? REST “proper” with Hypermedia? “Dumb” REST? JSON payload? XML payload? Protobuff payloads?
  • How do we expect load to be distributed between any instances of this service?
  • When making a request to an outside service, what do we do when awaiting a response?
    • Block? Start processing another request? Do something that is not I/O based?
  • How are we expecting to manage versioning of the APIs that this service is expect to provide?
  • Does this service need to respond to incoming calls/notifications?
  • If this service does need to respond, it is expected to be synchronous, “appear” synchronous, or be completely asynchronous style of response?
  • If asynchronous responses are expected, how does the service get the information it needs to know to where to send the response to?
  • How do we expect to trace a flow between work and the requests and responses that triggered that work? Is there a way to trace causality?
  • What is the minimum infrastructure/frameworks that is needed to provide the service?
    • Is this a service? Microservice? Additional monolithic application?
  • What is the problem domain (bounded context) of this service?
    • How do we know when we are adding features that should belong in other services?
  • How many requests are we expecting are needed to complete a business use case?
    • Is there any way to shrink that number? Can requests be combined?

This is by no means a complete list of questions we should be asking ourselves, but the start of a conversation to understand the scope of what it takes for a new service to be created and deployed. These are my brain dump of questions that help a team know if they know how to swim, and how deep the water is, before diving head first into the sea of microservices.

Let me know what other questions you think are missing.

–Proctor

Erlang Thursday Bonus – Functional fizzbuzz

A bonus Erlang Thursday for everyone today.

This past weekend I came across the post Bro, Do You Even FizzBuzz?!? about solving FizzBuzz in Clojure without using the modulus operator.

After translating it to Ruby as a point for a co-worker and publishing that translation, I thought I would translate it to Erlang as well to see the difference.

-module(fizzbuzz).

-export([fizzbuzz/1]).

fizzbuzz(N) ->
    Results = do_fizzbuzz(N),
    lists:foreach(fun(X) -> io:format("~p~n", [X]) end, Results).

do_fizzbuzz(N) ->
    Fizzes = cycle(["", "", "fizz"], N),
    Buzzes = cycle(["", "", "", "", "buzz"], N),
    FizzBuzzes = lists:zipwith(fun lists:append/2, Fizzes, Buzzes),
    Numbers = lists:seq(1, N),
    lists:zipwith(fun translate/2, Numbers, FizzBuzzes).

cycle(List, N) ->
    lists:sublist(lists:append(lists:duplicate(N, List)), N).

translate(Number, "") ->
    integer_to_list(Number);
translate(_, Translation) ->
    Translation.

A couple of points to note in the Erlang solution.

First, Erlang doesn’t have any direct concept of lazy lists/sequences, and there is no cycle function, so I had to improvise by calling lists:duplicate, lists:append, and lists:sublist a list that cycles over the source list to create a list that has N elements. While this is not exactly the most efficient way to do this, it shows that it can be done.

Second, the ability of using lists:zipwith helps with some of the ideas of pipelining, as we can process the items as they are zipped together, instead of having to process them as a different step.

Third, instead of using a case statement, we can use a function with a guard clause to determine if a translation exists or not, and use the translation if so, and the number if not.

I hope this gives you some food for thought, and would love to hear your feedback on how this could be improved even more, or other ways that FizzBuzz can be done besides the usual pattern matching with the guard clauses checking the remainders.

–Proctor

Erlang Thursday – c:m/1

Today’s Erlang Thursday continues to take a look at the c module with c:m/1.

c:m/1 takes an atom of a module name, and returns information about the module. It prints out information about the compliation date, time, and options; the object (BEAM) file that it was loaded from, and a list of functions exported by the module.

We’ll start with taking a look at the string module in Erlang.

c:m(string).
% Module string compiled: Date: November 28 2014, Time: 06.47
% Compiler options:  [{outdir,"/private/tmp/erlang-pY1Kv2/otp-OTP-17.3.4/lib/stdlib/src/../ebin"},
%                     {i,"/private/tmp/erlang-pY1Kv2/otp-OTP-17.3.4/lib/stdlib/src/../include"},
%                     {i,"/private/tmp/erlang-pY1Kv2/otp-OTP-17.3.4/lib/stdlib/src/../../kernel/include"},
%                     warnings_as_errors,debug_info]
% Object file: /usr/local/Cellar/erlang/17.3.4/lib/erlang/lib/stdlib-2.2/ebin/string.beam
% Exports:
% centre/2                      rstr/2
% centre/3                      span/2
% chars/3                       str/2
% chars/2                       strip/1
% chr/2                         strip/2
% concat/2                      strip/3
% copies/2                      sub_string/2
% cspan/2                       sub_string/3
% equal/2                       sub_word/2
% join/2                        sub_word/3
% left/2                        substr/2
% left/3                        substr/3
% len/1                         to_float/1
% module_info/0                 to_integer/1
% module_info/1                 to_lower/1
% rchr/2                        to_upper/1
% right/2                       tokens/2
% right/3                       words/1
%                               words/2
% ok

We can see that this was compiled on my machine on November 28th of 2014, and had the warnings_as_errors and debug_info turned on, as well as the location of the beam file, and all of the different functions the string module exports.

Next, we will look at a module compiled from inside the shell.

c(fizzbuzz).
% {ok,fizzbuzz}
c:m(fizzbuzz).
% Module fizzbuzz compiled: Date: August 5 2015, Time: 22.14
% Compiler options:  []
% Object file: /Users/proctor/tmp/fizzbuzz.beam
% Exports:
%          fizzbuzz/1
%          module_info/0
%          module_info/1
% ok

c:m(fizzbuzz) shows that it was compiled, and was loaded from my tmp directory, and exports fizzbuzz/1 along with the two versions of module_info that every module exports.

Again, this is one of those functions that you might not use everyday, but when it comes to debugging and inspecting your Erlang application becomes a very useful function to know about.

–Proctor

Erlang Thursday – c:xm/1

Today’s Erlang Thursday takes a turn down a slightly different route and takes a look in the c module at c:xm/1.

c:xm/1 takes either a atom, representing the name of a module, or a string, representing a filename, and inspects that module for unused and undefined functions, as well as calls to deprecated functions.

First let’s take a look at the erlang module, and see if there is anything in there that is deprecated.

c:xm(erlang).
% [{deprecated,[]},{undefined,[]},{unused,[]}]

Looks like there are no calls to deprecated functions, no calls to undefined functions, and no unused functions floating around in the erlang module. Note: This is running under Erlang 17.3.4, and you may get a different result depending on the version of Erlang you are running, because erlang:now/0 has been deprecated as of v18.0.

Trying to come up with an example of an existing module that might have some of these criteria, I took a look at the README for Erlang 17.0, and did a search for deprecated. Doing that there was a note:

The module pg has been deprecated and will be removed in Erlang/OTP 18.

So let’s pass that module to c:xm/1 and see what we get.

c:xm(pg).
% [{deprecated,[{{pg,create,1},{pg,master,1}},
%               {{pg,create,2},{pg,master,1}}]},
%  {undefined,[]},
%  {unused,[]}]

And we can see that we do get information back about deprecated functions in the pg module.

While the odds are low that you will need to use this function in your normal day to day work, as the tooling around Erlang generally seems to take care of this for you, this was intriguing enough that it seemed worthy of calling it out, especially if for those time when the compilation of Erlang code is done from inside the Erlang shell.

–Proctor