As bad as anything else

2012 05 04

Rebar Releases and Being Wrong

This is an entry I wrote during the Bay Area Erlang Factory 2012, not long before R15B01 came out. Back at then, I found the time to discuss a few things with fellow Erlang programmers. One of them was related to a patch to Reltool that someone submitted while before. This was a fix made to make things simpler for Erlang programmers wanting to generate releases using rebar.

In your usual Erlang project using rebar, the structure is usually based under a clean OTP Application directory structure:

ebin/
src/
include/
priv/

There is also a bunch of other optional directories, including priv/, logs/, and deps/. The deps/ is specifically the default directory for dependencies downloaded at build-time.

This way of doing things works very, very well when it comes to writing OTP Applications. As long as anyone respects that structure, getting any app as a dependency should never hurt.

There is however, an insidious extension to this done when transforming an OTP Application into a release. When people generate release, their directory structure might now look a bit like this:

ebin/
src/
include/
priv/
deps/
rel/

The rel/ directory is where the release will be built. That one will contain a release configuration file, that references to a directory with libraries is usually found. The line of configuration is precisely {lib_dirs, ["../deps/"]}. You can already see where there might be a problem — the src/ directory is getting ignored.

How Can we fix this?

Many rebar users fix this issue in two common ways:

The problem with this (especially the second approach) is that it's badly using a standard that has existed in OTP for a long, long while. The idea is that Erlang works with the basic concept of a LIB_DIRS environment variable. That variable should point to one or more directory that contains a bunch of application directories:

./
 - some_app/
   - src/
   - ebin/
 - other_app/
   - src/
   - ebin/
   - priv/

Putting that directory in LIB_DIRS will make it visible to the Erlang VM. When you put that directory in the configuration for a release ({lib_dirs, [Dirs]}), Reltool (the OTP release-building tool that is called by rebar) gets these variables to be part of its places to search.

Setting the directory to ../../ is a bad idea, because it basically makes the release building take into account all the directories on the same level as the one you're currently developing. This can have unexpected effects when you put that one first and you might have conflicting versions of libraries in deps/ and the directory itself; one of the two versions will be picked, and you might not be sure which.

The patch I mentioned earlier basically solved the issue by allowing the following to be placed in application requirements: {app, foo, [{incl_cond, include}, {lib_dir, ".."}]}. This patch basically lets you keep the OTP Application structure and turn it into a release without needing the messy hacks. I might be out of touch, but I can't understand why the OTP team would accept this patch. It is basically officializing a workaround for people using OTP tools in a way it wasn't intended.

There are two actual solutions that work to build releases, and they involve using a proper directory structure:

That's it, really. It's not very hard to do, and it works fine with all the regular OTP stuff. You use a proper release directory structure for releases, and you can keep using a proper OTP Application structure for OTP Applications. I do not understand why the fix had to make it upstream to OTP when its main use is to transform an OTP Application structure in something compatible with releases, especially since this need is mostly a quirk of how default values in rebar and community standards that emerged from its use pushed a lot of people to do things wrong (arguably). I would have prefered for people in general to fix their repositories rather than forcing the OTP team to fix their (non-broken) tool to accomodate them there.

I guess in the end it's kind of irrelevant; the fix in no way keeps me from doing what I was doing before, after all.