NAME goaljobs-reference - reference documentation for writing goaljobs scripts SUMMARY open Unix open Printf open Goaljobs let goal name args... = target (condition); require (goal1 ...); require (goal2 ...); (* code to implement the goal *) let goal all () = require (name args...) every 30 minutes ( fun () -> require (goal1 ...) ) DESCRIPTION Goaljobs is a flexible build system and business rules manager similar to make and cron, but much more powerful. You can use it to automate many complex tasks that have multiple steps (even with manual steps) that have to be carried out in dependency order. For a tutorial-like introduction to goaljobs, see For examples, see the "examples/" directory in the source and For reference documentation on how to write scripts, see below. Note this man page does not cover the whole Goaljobs API. To read about the Goaljobs API, look for the file "goaljobs.mli" (in the source code or installed as part of the goaljobs package), or see the HTML documentation installed as part of the goaljobs package. THE SCRIPT FILE The script file should usually start with opening these modules (none of this are required, it's just useful to have them open): open Unix open Printf open Goaljobs This is followed by goals and/or functions and/or top-level OCaml statements and/or "every" statements (a.k.a periodic jobs). You can use multiple script files to make up a goaljobs program. You have to list them in dependency order on the goaljobs command line (earlier files required by later files), the same way that the OCaml compiler works. So usually you end up writing: goaljobs utils.ml another_library.ml script.ml where "script.ml" requires the functions from the utils/library. Note that circular dependencies are not possible. GOALS Each goal should have the following basic form: let goal name args... = target (condition); require (goal1 ...); require (goal2 ...); (* code to implement the goal *) There is no hard-and-fast rule about this. In particular you can put arbitrary OCaml statements anywhere inside a goal (since a goal is just a special form of OCaml function), but sticking to this overall plan is a good idea. There should be zero or one target. Multiple target statements should not be used in a goal. The target should come as early as possible, and the target condition should be as simple and fast to evaluate as is practical (see "THE MEMORY" below). There should be zero or any number of "require" statements. Each require statement should name a single goal (with optional parameters for that goal). After that should come the code that implements the goal, which might be, for example, a series of shell commands, but could even be user-interactive. As with ordinary OCaml functions, you can define goals recursively or with mutual recursion using: let rec goal1 args... = ... and goal2 args... = ... and goal3 args... = ... A goal can also have no arguments: let goal all () = ... This defines the common goal called "all", which acts the same way as "make all", ie. if you run the program without any arguments, it will run the "all" goal if one exists. PUBLISHING GOALS If a goal is "published" it means it is available to be run directly from the command line. All no-arg goals are published by default. You do not need to do anything special for them. For example: let goal clean () = sh "rm *~" can be used on the command line: ./myscript clean For goals which take any parameters, you have to define a small code snippet that converts command line arguments to goal parameters (the reason has to do with OCaml being strongly typed, and because goal parameters might not all be strings). let goal compile program sources = target (more_recent [program] sources); ... let () = publish "compile" ( fun args -> let program = List.hd args in let sources = List.tl args in require (compiled program sources) ) Then you can use: ./myscript compile program main.c utils.c TARGET AND REQUIRE The target is promise or contract that you make that the given condition *will* be true when the goal has finished running. In the first example, the target is that the "o_file" (object) exists and is newer than the "c_file" (source). The goal meets that target by running the C compiler ("cc") which, if it succeeds, will ensure that the object file exists and is newer than the source file. let goal compiled c_file = let o_file = change_file_extension "o" c_file in target (more_recent [o_file] [c_file]); sh " cd $builddir cc -c %s -o %s " c_file o_file In the second example, the goal requires that several files have been compiled ("require (compiled ...)") before it can link the final program: let goal built program source = target (more_recent [program] [source]); require (compiled source); let object = change_file_extension "o" source in sh " cd $builddir cc %s -o %s " object program SPECIAL VALUES INSIDE GOALS goalname Inside goals, you can use "goalname" to get the name of the goal, ie: let goal foo () = printf "my name is %s\n" goalname would print: my name is foo goalloc Inside goals, you can use "goalloc" to get a printable source location of the goal, ie: let goal foo () = printf "%s\n" goalloc would print: File "source.ml", line 2, characters 13-71 (end at line 3, character 23) Note that the actual string format depends on the internal OCaml function "Loc.to_string" so it might change in future. onfail, onsuccess, onrun Inside goals you can register function(s) which run if the goal completes successfully ("onsuccess"), if the goal completes successfully after running to the end ("onrun"), or if the goal fails ("onfail"). For example: let goal built () = onfail (fun _ -> eprintf "goal '%s' failed\n" goalname); sh " cc -o program main.o " If the shell command (or another part of the goal) fails, then this would print out: goal 'built' failed The single parameter passed to "onfail" is the exception that was thrown. Note that the helper function "Goaljobs.mailto" is a useful function to call from an "onfail" handler: let from = "me@example.com" let to_ = "you@example.com" let logfile = log_program_output () let goal built () = onfail (fun _ -> let subject = sprintf "goal: %s: BUILD FAILED" goalname in mailto ~from ~subject ~attach:[logfile] to_); sh " cc -o program main.o " "onsuccess" and "onrun" are slightly different from "onfail" and from each other: "onsuccess" functions can be called if a "target" condition is met and the rest of the goal is short-circuited. "onrun" will only be called if all the instructions in the goal actually run and succeed. The single unit "()" parameter is passed to the "onsuccess" and "onrun" functions. You can register as many functions as you want for each handler. The order in which the functions are called is not defined. PERIODIC JOBS If you want to have a goal that runs when some outside event happens you have three choices: Manually run the script (this is basically what "make" forces you to do). Have some sort of hook that runs the script (eg. a git hook). Or use a periodic job to poll for an event or change. Periodic jobs run regularly to poll for an outside event or change. If a script has periodic jobs, then it runs continuously (or until you kill it). An example of a script that checks for new git commits and when it sees one it will ensure it passes the tests: let repo = Sys.getenv "HOME" // "repo" let goal git_commit_tested commit = let key = sprintf "repo-tested-%s" commit in target (memory_exists key); onrun (fun () -> memory_set key "1"); sh " git clone %s test cd test ./configure make make check " repo_url every 30 minutes (fun () -> let commit = shout "cd %s && git rev-parse HEAD" repo in (* Require that this commit has been tested. *) require (git_commit_tested commit) ) Some notes about the above example: Firstly only the current HEAD commit is required to be tested. This is because older commits are irrelevant and because if they failed the test before there is not point retesting them (commits are immutable). Secondly we use the Memory to remember that we have successfully tested a commit. This is what stops the program from repeatedly testing the same commit. SHELL You can call out to the Unix shell using one of the functions "Goaljobs.sh", "Goaljobs.shout" or "Goaljobs.shlines". (These functions are documented in the "goaljobs.mli" file / API documentation.) "sh" runs the command(s). "shout" collects the output of the command (to stdout only) and returns it as a single string. "shlines" collects the output and returns it as a list of lines. "sh", "shout", "shlines" work like printf. ie. You can substitute variables using %s, %d and so on. For example: sh "rsync foo-%s.tar.gz example.com:/html/" version Each shell runs in a new temporary directory. The temporary directory and all its contents is deleted after the shell exits. If you want to save any data, "cd" somewhere. If you don't want the temporary directory creation, use "~tmpdir:false". The environment variable $builddir is exported to the script. This is the current directory when the goaljobs program was started. Each invocation of "sh" (etc) is a single shell (this is slightly different from how "make" works). For example: sh " package=foo-%s tarball=$package.tar.gz cp $HOME/$tarball . tar zxf $tarball cd $package ./configure make " version The shell error mode is set such that if any single command returns an error then the "sh" function as a whole exits with an error. Write: command ||: to ignore the result of a command. "/bin/sh" is used unless you set "Goaljobs.shell" to some other value. Note that the environment variable "SHELL" is *never* used. THE MEMORY "The Memory" is key/value storage which persists across goaljobs sessions. It is stored in the file "$HOME/.goaljobs-memory" (which is a binary file, but you can delete it if you want). The Memory is locked during accesses, so it is safe to read or write it from multiple parallel goaljobs sessions. Keys and values are strings. The keys should be globally unique, so it is suggested you use some application-specific prefix. eg: "myapp-key" A common pattern is: let goal tested version = let key = "myapp-tested-" ^ version in target (memory_exists key); onrun (fun () -> memory_set key "1"); (* some code to test this version *) Note in that example the value 1 is arbitrary. You just want to store *any* value so that a later call to "memory_exists" will succeed. For information about "Goaljobs.memory_*" APIs see the "goaljobs.mli" file / API documentation. FILES "/bin/sh" This is the default shell used by "sh*" APIs. You can change the shell by setting the "Goaljobs.shell" reference. "curl" The curl program (on the path) is used to check for and download URLs by APIs such as "Goaljobs.url_exists". "~/.goaljobs-memory" Persistent key/value store used when you use the "Goaljobs.memory_*" APIs. ENVIRONMENT VARIABLES builddir This environment variable is set to the current directory when the goals program starts, and is available in goals, shell scripts, etc. SEE ALSO goaljobs(1) AUTHORS Richard W.M. Jones COPYRIGHT (C) Copyright 2013 Red Hat Inc., This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.