NAME
goaljobs-reference - reference documentation for writing goaljobs
scripts
SUMMARY
open Unix
open Printf
open Goaljobs
let goal name args... =
target (condition);
require (goal1 ...);
require (goal2 ...);
(* code to implement the goal *)
let goal all () =
require (name args...)
every 30 minutes (
fun () ->
require (goal1 ...)
)
DESCRIPTION
Goaljobs is a flexible build system and business rules manager similar
to make and cron, but much more powerful. You can use it to automate
many complex tasks that have multiple steps (even with manual steps)
that have to be carried out in dependency order.
For a tutorial-like introduction to goaljobs, see
For examples, see the "examples/" directory in the source and
For reference documentation on how to write scripts, see below.
Note this man page does not cover the whole Goaljobs API. To read about
the Goaljobs API, look for the file "goaljobs.mli" (in the source code
or installed as part of the goaljobs package), or see the HTML
documentation installed as part of the goaljobs package.
THE SCRIPT FILE
The script file should usually start with opening these modules (none of
this are required, it's just useful to have them open):
open Unix
open Printf
open Goaljobs
This is followed by goals and/or functions and/or top-level OCaml
statements and/or "every" statements (a.k.a periodic jobs).
You can use multiple script files to make up a goaljobs program. You
have to list them in dependency order on the goaljobs command line
(earlier files required by later files), the same way that the OCaml
compiler works. So usually you end up writing:
goaljobs utils.ml another_library.ml script.ml
where "script.ml" requires the functions from the utils/library. Note
that circular dependencies are not possible.
GOALS
Each goal should have the following basic form:
let goal name args... =
target (condition);
require (goal1 ...);
require (goal2 ...);
(* code to implement the goal *)
There is no hard-and-fast rule about this. In particular you can put
arbitrary OCaml statements anywhere inside a goal (since a goal is just
a special form of OCaml function), but sticking to this overall plan is
a good idea.
There should be zero or one target. Multiple target statements should
not be used in a goal. The target should come as early as possible, and
the target condition should be as simple and fast to evaluate as is
practical (see "THE MEMORY" below).
There should be zero or any number of "require" statements. Each require
statement should name a single goal (with optional parameters for that
goal).
After that should come the code that implements the goal, which might
be, for example, a series of shell commands, but could even be
user-interactive.
As with ordinary OCaml functions, you can define goals recursively or
with mutual recursion using:
let rec goal1 args... =
...
and goal2 args... =
...
and goal3 args... =
...
A goal can also have no arguments:
let goal all () =
...
This defines the common goal called "all", which acts the same way as
"make all", ie. if you run the program without any arguments, it will
run the "all" goal if one exists.
PUBLISHING GOALS
If a goal is "published" it means it is available to be run directly
from the command line. All no-arg goals are published by default. You do
not need to do anything special for them. For example:
let goal clean () = sh "rm *~"
can be used on the command line:
./myscript clean
For goals which take any parameters, you have to define a small code
snippet that converts command line arguments to goal parameters (the
reason has to do with OCaml being strongly typed, and because goal
parameters might not all be strings).
let goal compile program sources =
target (more_recent [program] sources);
...
let () =
publish "compile" (
fun args ->
let program = List.hd args in
let sources = List.tl args in
require (compiled program sources)
)
Then you can use:
./myscript compile program main.c utils.c
TARGET AND REQUIRE
The target is promise or contract that you make that the given condition
*will* be true when the goal has finished running.
In the first example, the target is that the "o_file" (object) exists
and is newer than the "c_file" (source). The goal meets that target by
running the C compiler ("cc") which, if it succeeds, will ensure that
the object file exists and is newer than the source file.
let goal compiled c_file =
let o_file = change_file_extension "o" c_file in
target (more_recent [o_file] [c_file]);
sh "
cd $builddir
cc -c %s -o %s
" c_file o_file
In the second example, the goal requires that several files have been
compiled ("require (compiled ...)") before it can link the final
program:
let goal built program source =
target (more_recent [program] [source]);
require (compiled source);
let object = change_file_extension "o" source in
sh "
cd $builddir
cc %s -o %s
" object program
SPECIAL VALUES INSIDE GOALS
goalname
Inside goals, you can use "goalname" to get the name of the goal, ie:
let goal foo () =
printf "my name is %s\n" goalname
would print:
my name is foo
goalloc
Inside goals, you can use "goalloc" to get a printable source location
of the goal, ie:
let goal foo () =
printf "%s\n" goalloc
would print:
File "source.ml", line 2, characters 13-71 (end at line 3, character 23)
Note that the actual string format depends on the internal OCaml
function "Loc.to_string" so it might change in future.
onfail, onsuccess, onrun
Inside goals you can register function(s) which run if the goal
completes successfully ("onsuccess"), if the goal completes successfully
after running to the end ("onrun"), or if the goal fails ("onfail").
For example:
let goal built () =
onfail (fun _ -> eprintf "goal '%s' failed\n" goalname);
sh "
cc -o program main.o
"
If the shell command (or another part of the goal) fails, then this
would print out:
goal 'built' failed
The single parameter passed to "onfail" is the exception that was
thrown.
Note that the helper function "Goaljobs.mailto" is a useful function to
call from an "onfail" handler:
let from = "me@example.com"
let to_ = "you@example.com"
let logfile = log_program_output ()
let goal built () =
onfail (fun _ ->
let subject = sprintf "goal: %s: BUILD FAILED" goalname in
mailto ~from ~subject ~attach:[logfile] to_);
sh "
cc -o program main.o
"
"onsuccess" and "onrun" are slightly different from "onfail" and from
each other:
"onsuccess" functions can be called if a "target" condition is met and
the rest of the goal is short-circuited. "onrun" will only be called if
all the instructions in the goal actually run and succeed.
The single unit "()" parameter is passed to the "onsuccess" and "onrun"
functions.
You can register as many functions as you want for each handler. The
order in which the functions are called is not defined.
PERIODIC JOBS
If you want to have a goal that runs when some outside event happens you
have three choices: Manually run the script (this is basically what
"make" forces you to do). Have some sort of hook that runs the script
(eg. a git hook). Or use a periodic job to poll for an event or change.
Periodic jobs run regularly to poll for an outside event or change. If a
script has periodic jobs, then it runs continuously (or until you kill
it).
An example of a script that checks for new git commits and when it sees
one it will ensure it passes the tests:
let repo = Sys.getenv "HOME" // "repo"
let goal git_commit_tested commit =
let key = sprintf "repo-tested-%s" commit in
target (memory_exists key);
onrun (fun () -> memory_set key "1");
sh "
git clone %s test
cd test
./configure
make
make check
" repo_url
every 30 minutes (fun () ->
let commit = shout "cd %s && git rev-parse HEAD" repo in
(* Require that this commit has been tested. *)
require (git_commit_tested commit)
)
Some notes about the above example: Firstly only the current HEAD commit
is required to be tested. This is because older commits are irrelevant
and because if they failed the test before there is not point retesting
them (commits are immutable). Secondly we use the Memory to remember
that we have successfully tested a commit. This is what stops the
program from repeatedly testing the same commit.
SHELL
You can call out to the Unix shell using one of the functions
"Goaljobs.sh", "Goaljobs.shout" or "Goaljobs.shlines". (These functions
are documented in the "goaljobs.mli" file / API documentation.)
"sh" runs the command(s). "shout" collects the output of the command (to
stdout only) and returns it as a single string. "shlines" collects the
output and returns it as a list of lines.
"sh", "shout", "shlines" work like printf. ie. You can substitute
variables using %s, %d and so on. For example:
sh "rsync foo-%s.tar.gz example.com:/html/" version
Each shell runs in a new temporary directory. The temporary directory
and all its contents is deleted after the shell exits. If you want to
save any data, "cd" somewhere. If you don't want the temporary directory
creation, use "~tmpdir:false".
The environment variable $builddir is exported to the script. This is
the current directory when the goaljobs program was started.
Each invocation of "sh" (etc) is a single shell (this is slightly
different from how "make" works). For example:
sh "
package=foo-%s
tarball=$package.tar.gz
cp $HOME/$tarball .
tar zxf $tarball
cd $package
./configure
make
" version
The shell error mode is set such that if any single command returns an
error then the "sh" function as a whole exits with an error. Write:
command ||:
to ignore the result of a command.
"/bin/sh" is used unless you set "Goaljobs.shell" to some other value.
Note that the environment variable "SHELL" is *never* used.
THE MEMORY
"The Memory" is key/value storage which persists across goaljobs
sessions. It is stored in the file "$HOME/.goaljobs-memory" (which is a
binary file, but you can delete it if you want).
The Memory is locked during accesses, so it is safe to read or write it
from multiple parallel goaljobs sessions.
Keys and values are strings. The keys should be globally unique, so it
is suggested you use some application-specific prefix. eg: "myapp-key"
A common pattern is:
let goal tested version = let key = "myapp-tested-" ^ version in target
(memory_exists key); onrun (fun () -> memory_set key "1");
(* some code to test this version *)
Note in that example the value 1 is arbitrary. You just want to store
*any* value so that a later call to "memory_exists" will succeed.
For information about "Goaljobs.memory_*" APIs see the "goaljobs.mli"
file / API documentation.
FILES
"/bin/sh"
This is the default shell used by "sh*" APIs. You can change the
shell by setting the "Goaljobs.shell" reference.
"curl"
The curl program (on the path) is used to check for and download
URLs by APIs such as "Goaljobs.url_exists".
"~/.goaljobs-memory"
Persistent key/value store used when you use the "Goaljobs.memory_*"
APIs.
ENVIRONMENT VARIABLES
builddir
This environment variable is set to the current directory when the
goals program starts, and is available in goals, shell scripts, etc.
SEE ALSO
goaljobs(1)
AUTHORS
Richard W.M. Jones
COPYRIGHT
(C) Copyright 2013 Red Hat Inc.,
This program is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 2 of the License, or (at your
option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.