Generate and publish edoc in Jenkins using Javadoc plugin

edoc is Erlang’s answer to javadoc and can be easily added to an existing Jenkins  Erlang build job leveraging the Jenkins javadoc plugin to generate and display documentation.

Generate edoc

Generate edoc for a rebar enabled project by invoking the following command
./rebar doc skip_deps=true

This automatically creates creates a doc dir which will be populated with generated html files

Configure jenkins

Edit your existing Jenkins Erlang build job, configure as follows

  • Add in the additional rebar doc command (./rebar doc skip_deps=true)
  • Add post-build action of type ‘Publish Javadoc’
    Adding javadoc as a post build step
  • Configure the plugin to point at the contents of the generated doc dir
    Configure javadoc plugin to point at doc dir
  • Save settings and kick off job
  • After job finishes click on the new document link
    Click on newly created document link

webmachine – wmtrace exception

webmachine – wmtrace exception – Problem discription

I was getting the following exception when i thought i had wmtrace successfully set up.


The server encountered an error while processing this request:
[{erlang,iolist_to_binary,[error],[]},
{webmachine_decision_core,encode_body,1,
[{file,"src/webmachine_decision_core.erl"},
{line,668}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},
{line,567}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},
{line,33}]},
{webmachine_mochiweb,loop,2,[{file,"src/webmachine_mochiweb.erl"},{line,74}]},
{mochiweb_http,parse_headers,5,[{file,"src/mochiweb_http.erl"},{line,180}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]

I had following setup complete and i was seeing the .trace files successfully created after hitting requests to my resource.

init(Config) ->
%% enable tracing the decision core for debugging
{{trace, "traces"}, Config}.

And adding the vmtrace dispatch rule

{["wmtrace",'*'], %% exposed at /wmtrace/*
wmtrace_resource, %% defined in wmtrace_resource.erl
%% (included with webmachine)
[{trace_dir, "traces"}]}. %% trace files are in "traces" directory

Solution

What i had skipped over in the documentation was that the visualization to be enabled.

After hitting ./start.sh hit return and enter the following in the erlang shell (absolute path).

wmtrace_resource:add_dispatch_rule("wmtrace", "/tmp").

…or relative path

wmtrace_resource:add_dispatch_rule("wmtrace", "traces").

This should yield a more inviting message like Traces in /tmp when you hit hostname:port/wmtrace

Resources

 

 

 

REST using Webmachine, Good resource

Great deck by KevSmith on using Webmachine for a clean RESTful service –

escript: bad_central_directory

escript: bad_central_directory

escript is packaged as part of the erlang otp and essentially executes uncompiled erlang code. Rebar delegates alot of work to it. Recently, I was getting the following error when trying to run the webmachine project files from the excellent book, ‘7 webservers in 7 weeks‘.

I’m running R16B02 on OS X
$ erl --version
Erlang R16B02 (erts-5.10.3) [source-b44b726] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

$ ./rebar clean get-deps compile
escript: bad_central_directory

Even getting the version of rebar failed
$ ./rebar version
escript: bad_central_directory

Solution

Download the latest version of rebar from github,
invoke their ./bootsrap command.
$ git clone git://github.com/rebar/rebar.git
Cloning into 'rebar'...
remote: Counting objects: 6053, done.
remote: Compressing objects: 100% (2630/2630), done.
remote: Total 6053 (delta 3623), reused 5596 (delta 3205)
Receiving objects: 100% (6053/6053), 2.13 MiB | 554 KiB/s, done.
Resolving deltas: 100% (3623/3623), done.
$ cd rebar
$ ./bootstrap

This should build successfully. Verify by getting version

$ ./rebar version
rebar 2.1.0 R16B02 20131203_115508 git 2.1.0-13-g30531b6

Copy the rebar executable into the project which was giving the ‘escript’ error.
Invoke ‘make’ and should be able to build project successfully.

Creating a RESTful Service in Chicago Boss

Here’s an end-to-end REST implementation in Chicago Boss. Not sure has anyone a nice concise guide out there so hope this is useful.

Overview

A very simple model (Book[title, author, insertTimestamp, updateTimestamp])

CRUD implementation where

  •             Create is a POST request
  •             Read is a GET request
  •             Update is a PUT request
  •             Delete is a DELETE request

Focus on the service approach, use a WS client (eg. SoapUI) to test

Use local mysql for simplicity

Assumptions

Erlang is installed (I’m using R16B01)

Access to a mysql server where you can create a schema and user. I have one running locally

DDL

As mentioned, we have a very simple model. A book has two attributes, which are user driven, and two meta attributes which the service will manage.


CREATE DATABASE IF NOT EXISTS `bookdb` /*!40100 DEFAULT CHARACTER SET utf8 */;
USE `bookdb`;
DROP TABLE IF EXISTS `books`;
--
-- Author: jason o riordan Database: bookdb
-- Date: 10/15/13
-- File: books_ddl.sql
CREATE TABLE `books` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`book_name` varchar(128) DEFAULT NULL,
`author_name` varchar(128) DEFAULT NULL,
`insert_timestamp` timestamp DEFAULT '0000-00-00 00:00:00',
`update_timestamp` timestamp DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8;

Project Creation

Create a new ChicagoBoss project as follows

git clone https://github.com/evanmiller/ChicagoBoss.git
cd ChicagoBoss

I added the following line to rebar.config dependency section in order to include the cb_admin project

{cb_admin, “.*”, {git, “git://github.com/evanmiller/cb_admin.git”, “HEAD”}}

Now, pull down all ChicagoBoss’s dependencies and create out new project

./rebar get-deps
make app PROJECT=bookRestCatalogue

In the bookRestCatalogue dir I edited the boss.config file – boss.config

Note: port  is set to 8004, mysql connection details are set in db section

Model

Very simple model as outlined in the ddl

%% %%
%% @title book.erl
%% @author jasonoriordan
%% Created on 07. Oct 2013 9:48 AM
%% %%
%% @doc book.erl is a module that represents book model
%% @end
-module(book,[Id, BookName, AuthorName, InsertTimestamp::datetime(), UpdateTimestamp::datetime()]).
-author("jasonoriordan").
-compile(export_all).
%%
%% @doc Validate input, no empty fields are allowed in system
%% @end
validation_tests() ->
[
%% BookName Required
{fun() ->
BookName =/= undefined andalso
length(BookName) =/= 0
end, {book_name, "Book Name required"}},
%% AuthorName Required
{fun() ->
AuthorName =/= undefined andalso
length(AuthorName) =/= 0
end, {author_name, "Author Name required"}}
].

The model and ddl can be verified by hitting cb_admin and going to the model section

Cb admin model section showing Book

REST Contract/Controller

As mentioned we are going to have do CRUD actions on the book entity by have four services exposed. The logic is contained in a controller class, relevant extracts are explained below and full file is available on github

Create – HTTP POST

Pass the bookname and author name as URI arguments


%% Book CREATE
%%
%% Create a new carrier based on the passed params eg. book/save?authorname=jk&bookname=hp
save('POST', []) ->
AuthorName = Req:post_param("authorname"),
BookName = Req:post_param("bookname"),
NewBook = carrier:new(id,AuthorName, BookName, calendar:universal_time(),calendar:universal_time()),
% have record and return the newly created if ok or errors otherwise, in json format
case NewBook:save() of
{ok, SavedBook} ->
{json, SavedBook};
{error, Errors} ->
{json, Errors}
end.

Read – HTTP GET

Read the book names by passing the id

%% BOOK Read
%%
%% Read the bookId from the url eg. book/view/book-7
view('GET', [BookId]) ->
error_logger:info_msg("GET-book/view/~p request recieved~n",[BookId]),
Book = boss_db:find(BookId),
case Book of
[] ->
{output,<<"[]">>,[{"Content-Type","application/json"}]};
_Else ->
{json, Book}
end.

Read all carriers by hitting view_all

%% BOOK Read ALL
%%
%% Return all eg. book/view_all
view_all('GET', []) ->
error_logger:info_msg("GET-book/view_all request revieved~n"),
Books = boss_db:find(book, []),
case Books of
[] ->
{output,<<"[]">>,[{"Content-Type","application/json"}]};
_Else ->
{json, Books}
end.

Update – HTTP PUT

Update the book by passing the  bookname and authorname as URI arguments. Also updates the update_timestamp


%% Book Update
%%
%% Note: can also update by populating id with an actual value and calling 'new'
%% UpdatedBook = book:new(BookName,AuthorName, calendar:universal_time(),calendar:universal_time()),
%% Updates the book based on the passed params eg. book/update?bookid=myid&bookname=cuckoo&authorname=jk
update('PUT', []) ->
BookId = Req:post_param("bookid"),
BookName = Req:post_param("bookname"),
AuthorName = Req:post_param("authorname"),
Book = boss_db:find(BookId),
% update via attribute setting (as we don't want to update create_timestamp)
BookWithUpdatedName = Book:set(book_name,BookName),
BookWithUpdatedAuthor = BookWithUpdatedName:set(author_name,AuthorName),
BookWithUpdateTS = BookWithUpdatedAuthor:set(update_timestamp,calendar:universal_time()),
case BookWithUpdateTS:save() of
{ok, UpdatedBook} ->
{json, UpdatedBook};
{error, Errors} ->
{json, Errors}
end.

Delete – HTTP DELETE

Delete the book entity by passing

%% Book DELETE
%%
%% Delete the specified book from the system eg. book/delete/book-7
delete('DELETE', [BookId]) ->
boss_db:delete(BookId),
case boss_db:counter(BookId) of
0 ->
{output,<<"{ \"Deleted Status\": \"Book Deleted\" }">>,[{"Content-Type","application/json"}]};
_Else ->
{output,<<"error">>,[{"Content-Type","application/json"}]}
end.

Testing

Focus of this post is on service level so have added a SoapUI testsuite in order to test functionality

SoapUI test file is available from Git

Note: be sure to set attributes to Method level for create and update as per screenshot

SoapUI request showing configuration for Method level attributes

 

Resources

 

ChicagoBoss – The requested template (“src/view/blaindex.{dtl,html,txt,js,jade,eex}”) was not found. Additionally, no handler was found for processing 404 errors. You probably want to modify bla.routes to prevent errors like this one.

I wasted a lot of time with the above exception. Editing the routes file to no avail.

The problem here is most likely with the controller.

The problem in my case was that IntelliJ added below by default

%% API
-export([]).

where it should have been

%% API
-compile(export_all).

Simple one, but wasted a lot of time:(

Vagrantfile for an Erlang/MySql project using Chef

Vagrant is a great tool for getting a virtualized server up and running

The following is a cookbook and Vagrantfile for setting up a 64bit ubuntu box. I had a lot of workarounds to get an environment successfully set up, hopefully this will ease the pain

  • Install Vagrant and VirtualBox
  • Create cookbook folder for chef
  • Clone relevant recipes into mycookbook/myrecipes

git clone git://github.com/opscode-cookbooks/apt.git apt
git clone git://github.com/opscode-cookbooks/openssl.git openssl
git clone git://github.com/opscode-cookbooks/yum.git yum
git clone git://github.com/opscode-cookbooks/build-essential.git build-essential
git clone git://github.com/cookbooks/rubygems.git rubygems
git clone git://github.com/opscode-cookbooks/gems.git gems
git clone git://github.com/opscode-cookbooks/erlang.git erlang
git clone git://github.com/opscode-cookbooks/mysql.git mysql

cd mysql

it checkout 3.0.0

cd ..
git clone git://github.com/opscode-cookbooks/database.git database

  • Edit mycookbooks/erlang/recipes/default.rb to specify version/type/checksum of erlang install
  • Edit mycookbooks/database/recipes/default.rb  to specify databases/users/data to seed
  • Add Vagrantfile to repo

# -*- mode: ruby -*-
# vi: set ft=ruby :

# We’ll mount the Chef::Config[:file_cache_path] so it persists between
# Vagrant VMs
host_cache_path = File.expand_path(“../.cache”, __FILE__)
guest_cache_path = “/tmp/vagrant-cache”

# ensure the cache path exists
FileUtils.mkdir(host_cache_path) unless File.exist?(host_cache_path)

# Vagrantfile API/syntax version. Don’t touch unless you know what you’re doing!
VAGRANTFILE_API_VERSION = “2”

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

config.vm.box = “precise64”

config.vm.box_url = “http://files.vagrantup.com/precise64.box”

config.vm.synced_folder “…/myrepo/”, “/vagrant_data”

config.vm.provider :virtualbox do |vb|
vb.customize [“modifyvm”, :id, “–memory”, “1024”]
end

# http://gettingstartedwithdjango.com/questions/1/error-installing-chef/
config.vm.provision :shell, :inline => “sudo aptitude -y install build-essential”

config.vm.provision :chef_solo do |chef|
chef.cookbooks_path = “../../tm_recipes/cookbooks”
# stuff that should be in base box
chef.add_recipe “build-essential”
chef.add_recipe “openssl”
chef.add_recipe “yum”
chef.add_recipe “apt”
#http://stackoverflow.com/questions/15328369/error-executing-action-install-on-resource-chef-gemmysql-installing-ruby
chef.add_recipe “rubygems”
chef.add_recipe “gems”
# note: mysql version 3.0.0, due to bug
chef.add_recipe “mysql::ruby”
chef.add_recipe “mysql::server”
chef.add_recipe “database”

#leave erlang until last, it’s a big file

chef.add_recipe “erlang”
chef.log_level = :debug

chef.json = {
:mysql => {
:server_root_password => ‘root’,
:bind_address => ‘127.0.0.1’,
:server_root_password => ‘bla’,
:server_repl_password => ‘bla’,
:server_debian_password => ‘bla’

}
}
end
end

  • Create your virtual server by hitting the following commands

vagrant box add base http://files.vagrantup.com/precise64.box

vagrant up

vagrant ssh

 

Some issues encountered (Vagrant file contains links to solutions)

  • Downgraded mysql to version 3.0.0
  • Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.5/libmysqlclient18_5.5.24-0ubuntu0.12.04.1_amd64.deb  404  Not Found [IP: 91.189.92.190 80]
  • Erlang E: Unable to fetch some archives, maybe run apt-get update or try with –fix-missing?
  • (mysql::client line 57) had an error: Chef::Exceptions::Exec: apt-get
  • 404 Not Found [IP: 91.189.91.15 80]

 

bcrypt install trouble

Bcrypt is a useful library for erlang encryption. It is used by the Colosimo sample app

I had some trouble installing bcrypt.

When i typed the following in terminal:

erl
crypto:start().
bcrypt:start().

I got the following exception:
** exception error: undefined function bcrypt:start/0
or the following from ChicagoBoss:
{undef,
[{bcrypt,start,[],[]},

To get it working, start bcrypt using:
erl -pa ebin -boot start_sasl -s crypto -s bcrypt

From the top, Install in Erlang installation dir and start bcrypt as follows

sudo su
cd /erlanginstalldir/lib
git clone https://github.com/mrinalwadhwa/erlang-bcrypt.git
cd erlang-bcrypt
make
erl -pa ebin -boot start_sasl -s crypto -s bcrypt

Update
Good overview of bcrypt in ChicagoBoss
References:

https://github.com/mrinalwadhwa/erlang-bcrypt

https://github.com/imperialwicket/colosimo/issues/1

https://github.com/imperialwicket/colosimo/pull/2

 

ChicagoBoss Project Structure

ChicagoBoss is really neat.

One thing I’m swinging toward though in relation to project layout is to keep the ChicagoBoss-<Version#> at the dir structure. A lot of the tutorials talk about renaming to ‘ChicagoBoss’.

I’d see the benefits of keeping the version number as follows:

  • Easier to upgrade
  • Easier to test with different versions (including rollbacks)
  • Less repo activity on a ‘ChicagoBoss’ dir
  • Exceptions thrown for wrong CB version are very difficult to figure out

Remember the hook for ChicagoBoss dir is in your projects ‘boss.config’ file. Change to something like

{path, "../ChicagoBoss-0.8.6"},

Welcome

simple blog documenting issue solutions and gotcha’s from someone trying to get up to speed on erlang