SIAL : Simple Image Access Language =================================== Sial is a C interpreter that permits easy access to the symbol and type information stored in a executable image like a coredump or live memory interfaces (e.g. /dev/kmem, /dev/mem). It support a FULL C syntax and the same variable and function scope and type. Symbols and type information in the image become standard variables and types in the sial scripts's context. This README focuses on the differences between sial and a C compiler so, for C syntax information, please refer to a C reference manual. I also explain the mechanisms of the API that allow sial to be inserted into any debugging tool that deals with objects and can furnish symbol and type information to sial through the API. The more specific lcrash sial implementation is described and a howto on creating commands is also given here. Preprocessor commands --------------------- All preprocessor commands I knew of are supported. #define, #undef, #ifdef, #if, #ifndef, #else, #elif, #endif and #include. This one is ignored: #ident #pragma Sial has a builtin secondary parser for preprocessor expression evaluation. Symbols ------- The symbols from the system image and their associated value are available from within sial. Since, most of the time, no type information is associated with the symbols, a reference to a symbol return the actual address of the symbol within the system image. So you might say that sial adds a level of reference for symbol information. Let's say there is a (int) symbol called "nproc" that contains the number of processes currently running on the system. To get the value of nproc from sial one would have to write something like this: void showprocs() { int i; int np; np=*(int*)nproc; for(i=0;ip_next) do something... } } Variable Initialization -------------------- Variable assignment at the time of declaration is supported. Also, the function __init() will be executed, if it is defined, right after the macro file as been compiled. Using an uinitialized variable will generate a run time error. Variable types -------------- All types made available from the API can be used. These are the types already defined in the executable image. Floating point types are not supported at this time. I have no plan to support it. Declaration of arrays is not supported. To access a array from the image, use a pointer. Unions and structures type declarations are supported and can be used to create additional types that become available within the same macro file. Typedef are supported. Function pointers are not supported (use 'string' type instead, see "Operators" below) Sial defines a 'string' types to handle ansi C strings within the interpreter. This string type also support some of the operator (+, ==, =, !=, etc... see below) Variable Scope -------------- All symbols available in the system image become global variable in the sial context. Variable declared within sial can be given one of 3 different scopes like in normal C. GLOBAL: A variable that is declared outside a function that is not marked as static. this variable if available to all macros that have been load into the interpreter. Ex: file1: int global; int func() { } file2: func2() { int i; i=global; } NB: since sial currently validates variable existence only at run time there is no need to declare a 'global' as an 'extern' variable. At run time, if none of the currently loaded macros define a global variable called 'global' then 'i=global' will fail with a 'unknown variable' error. FILE: A Variable that is declared outside any functions that is tagged with the static keyword. This variables is available to all functions defined in the same macro file. Ex: file1: static int maxproc=100; sraruc int array; __init() { int i; for(i=0;i<10;i++) array[i]=-1; } void func1() { int i; for(i=0;i, <=, >=. examples: s = "hello" + "World"; if("hello" == "world" ) { ... } The 'in' operator is supported on dynamic arrays. if(i in array) { ... } str=i in array ? "yes" : "no"; Function callbacks ------------------ Function calls through function pointers is not possible currently. Instead, use a 'string' type to achieve the same result. When sial is about to perform a call, it will look at the type of the variable used to name the function. If the type is 'string' it will use the value that string and call that function instead. func0(int i, int j) { printf("i=%d j=%d\n", i,j); } func1(string func) { func(1,2); } main() { func1("func0"); } In the above example, func1() ends up calling func0() not func. This can be used as a callback mechanism, specially useful for creating generating function that call s linked list of objects and calls a variable function for each object. Like a function that walks tasks or procs. The sizeof() operator is evaluated at execution time. So you can supply a type, a variable, or any valid expression and the appropriate size will be returned. The expression *will* be executed, so be careful. Statements ---------- All C statements except 'goto' are supported. The 'for(i in array)' is supported on dynamic arrays. Dynamic arrays ------------------------------- When indexing through a non pointer variable you end up creating a dynamic array. Example: int func() { char *cp, c; int array, i; cp=(char *)symbol; c=cp[10]; array[0]="one string"; array[12]="second string"; for(i in array) { printf("array[%d]=%s\n", i, array[i]); } } In the 'c=cp[10]' statement, sial goes to the system image to get one 'char' at the address symbol+10. In the second case, 'array' is not a pointer, it's a 'int'. So sial threats all indexing through it as dynamic. Additionally, sial supports multi levels of dynamic index, which makes possible to create random trees of indexed values: int func() { int array, i, j; array[10]="array10"; array[10][3]="array10,3"; array[20]="array20"; array[20][99]="array20,99"; for(i in array) { printf("array[%d]=%s\n", i, array[i]); for(j in array[i]) { printf("array[%d][%d]=%s\n", i, j, array[i][j]); } } } I think it is a good thing to have, since system image access and analysis require frequent lists search. So for example, with dynamic arrays, one can walk the proc list taking note of the proc*, then walking a user thread list taking note of the thread* and gathering some metrics for each of these threads. In order to get to these metrics at some later point in the execution, something like this could be used: func() { proc *p; for(p in procs) { thread *t; for(t in procs[p]) { int rss, size; /* we use index 0 for rss and 1 for size */ printf("proc %p, thread %p, rss:size = %d:%d\n" , p, t, procs[p][t][0], procs[p][t][1]); } } } Arrays are always passed by reference. On creation the reference count is set to one. So this array will exist untill the variable it's assigned to dies. Arrays can be created by dso's. See the DSo section below for more information and examples of this. Sial API -------- Sial can be integrated into any tool that needs to access symbol and type information from some object. Currently it is integrated in lcrash and icrash (tools that access Linux and Irix kernel images respectively), but it should be possible to use it, for example, in dbx or gdb. The API gives a simple interface through which the host application sends symbol and type (including member) information and gives access to the image itself so that sial can read random blocks of data from the image. >> sial_builtin(bt *bt) Install some set of builtin function. See below (builtin api). >> sial_chkfname(char *fname, int silent); Will check for the exsistance of a function in sial. Typically used to check xtra entry points before the application registers a new command (see sial_setcallback). >> sial_open(): The first function that should be called is sial_open(). sial_open() will return a value of 1 if everything is ok or 0 in case of some problem. This call initializes internal date for the sial package. >> sial_setapi(apiops* ops, int nbytes): This function will setup the callbacks that sial will use to get information from the application. See 'callback interface' below. >> sial_load(char *name); To have sial load and compile a macro or a set of macro use sial_load(). Parameter name gives the name of the file to compile. If name points to a directory instead, then all the files in this directory will be load. So an application would call sial_load() when it first starts up specifying some well known files or directories to load. For example $HOME/.xxx and /etc/xxx would be loaded, ~/.xxx containing user defined macros, and /etc/xxx containing system macros. >> sial_unload(char *funcname) To unload the a macro file use this function. "funcname" is the name of any global function in the file you want to unload. >> void sial_setcallback(void (*scb)(char *)); To be called prior to any load calls. After each loads, sial will call this function back with the name of each functions compiled. Typicly, the application will then perform checks and potencially install a new command for this function. ex: void reg_callback(char *name) { char fname[MAX_SYMNAMELEN+sizeof("_usage")+1]; _command_t cmds[2]; snprintf(fname, sizeof(fname), "%s_help", name); if(!sial_chkfname(fname, 0)) return; snprintf(fname, sizeof(fname), "%s_usage", name); if(!sial_chkfname(fname, 0)) return; cmds[0].cmd=strdup(name); cmds[0].real_cmd=0; cmds[0].cmdfunc=run_callback; cmds[0].cmdparse=parse_callback; cmds[0].cmdusage=usage_callback; cmds[0].cmdhelp=help_callback; cmds[1].cmd=0; unregister_cmd(cmds[0].cmd); (void)register_cmds(cmds); } >> sial_setipath(char *path) When sial processes a #include directive it will use the specified path as a search path. The usual PATH format is supported ex: "/etc/include:/usr/include". >> sial_setmpath(char *path) When sial_load() is called with a relative path name or just the name of a file, it will use a search PATH to locate it. The path parameter to sial_set,path() sets this path. The usual PATH format is supported ex: "/etc/xxx:/usr/lib/xxx". >> sial_setofile(FILE *ofile) All output of sial commands will be send to file ofile. >> sial_cmd(char *cmd, char **argv, int nargs) This is the way to execute a sial command that as been loaded. 'cmd' is the name of function to call. 'argv' are the argument to this function. 'nargs' is the number of argument in array 'argv'. Sial_cmd() will process argv and make the corresponding values available to the function by creating global variables that the function can test and use. >> sial_showallhelp() This command will send a complete list of the commands long with the usage and help for each one of them. This function should be called when the user request something like 'help all'. >> sial_showhelp(char *func) This will display the help information for a particular function loaded in sial. The callback interface ---------------------- Everytime sial needs some piece of information, it will call the application back for it. THe sial_apiset() function is used to install this callback interface into sial. Here is the list of callback functions: typedef unsigned long long ull; Sial_apiset() passes this structure to sial: typedef struct { int (*getmem)(ull, void *, int); int (*putmem)(ull, void *, int); int (*member)(char *, ull, type * , member *); int (*getctype)(int ctype, char * , type*); char* (*getrtype)(ull, type *); int (*alignment)(ull); int (*getval)(char *, ull *); enum_t* (*getenum)(char *name); def_t* (*getdefs)(); uint8_t (*get_uint8)(void*); uint16_t (*get_uint16)(void*); uint32_t (*get_uint32)(void*); uint64_t (*get_uint64)(void*); } apiops; The apiops* struct defines the following member and function pointers: -getmem(ull addr, void *buffer, int nbytes) Read nbytes from image at virtual address addr (32 or 64 bit) to buffer. -putmem(ull addr, void *buffer, int nbytes) Write nbytes from buffer to image at virtual address addr (32 or 64 bit). -member(char *name, ull pidx, type *tm, member *m); Get information on a structure member called name. Pidx is a unique type index for the parent structure. The getctype() function should fill in this index in it's type*. The dwarf model uses unique indexes (die offsets) that can be used here. 'tm' will hold information on the type of the member. 'm' will hold information on the member specific stuff (bit sizes, bit offset etc.). Use the sial_member_...() functions to setup m. Use the sial_type_...() functions to setup t. -getctype(int ctype, char *name, type *tout) Get type information for a complex type. Ctype specifies that name is a type of type struct/union or enum. tout contain the returned type information. -getrtype(ull idx, type *t) Gets the type string linked to a typedef. For example, the gettdeftype() would return "unsigned long long". This enables sial to drill down a typedef (a typedef can be build from a typedef itself) in order to perform proper type validation for assignment or function parameters or return values. -getval(char *sname, ull *value) Returns the value of symbol "sname" from the image. The value is returned in 'value'. On any image this is address of the symbol within the image itself, not the value of the symbol itself. See explanation of this above. -getenum(char *name); Return a list of enum values. Sial will make these available as symbol for the duration of the compile. -getdefs() Return a list of #defines to be active througout the sial session. -get_uint8/16/32/64() Return converted unsigned integers. As parameters are passed pointers to unsigned int values in dump representation. The return values are the corresponding unsigned int values in the representation of the host architecture, where sial is running. The builtin API --------------- Sometime it is necessary to create a C function that will handle some piece of the work, that a macro cannot do. Sial's builtin function are implemented this way. Generic function like 'printf' or 'getstr' can get some parameter input from the macros and do something (printf) or they get some information, map it to a sial value and return it to a macro (getstr). Sial can load new functiosn from DSOs. If the extension of a file name is ".so" then sial opens it and gets a list of function specs from it. Unload of that file will uninstall these functions. The API between the dso and sial is quite simple at this time. It has not been exercised as must as it would need to, so it might get more flexible and thus complex in the future. Here are two examples of simple extensions. This is an example of a simple extension. An equivalent os the "hello world" C program, but this one gets 2 parameters , one int and one string and returns the received int. #include "sial_api.h" value * helloworld(value *vi, value *vs) { int i=sial_getval(vi); char *s=(char*)sial_getval(vs); sial_msg("Hello to the world![%d] s=[%s]\n", i, s); return sial_makebtype(1); } BT_SPEC_TABLE = { { "int hello(int i, string s)", helloworld}, { 0, 0} }; static char *buf; BT_INIDSO_FUNC() { sial_msg("Hello world being initialized\n"); buf=sial_alloc(1000); return 1; } BT_ENDDSO_FUNC() { sial_msg("Hello world being shutdown\n"); sial_free(buf); } The BT_SPEC_TABLE is scanned. It's a simple table with 2 entries per functions and terminated with a NULL prototype. The DSO initializer function is called. If it returns 0 then installtion is terminates. If it returns 1 we proceed forward. The prototype is compiled and a syntax error will send the error message to the application output file (stdout usually). When the prototype as compiled with no errors the function is installed and ready to be used from sial macros. Type checking is performed by sial at execution time on both, the function parameters andthe function return. DSO's can also receive, create and manipulate dynamic arrays. Here is an example of this: #include "sial_api.h" #ifdef ARRAY_STATIC static value *v; #endif value * mkarray(value* vi) { int i=sial_getval(vi); #ifndef ARRAY_STATIC value *v=sial_makebtype(0); #endif sial_msg("Received value [%d]\n", i); /* build an array indexed w/ int w/ 2 string values */ sial_addvalarray(v, sial_makebtype(0) , sial_makestr("Value of index 0")); sial_addvalarray(v, sial_makebtype(2) , sial_makestr("Value of index 2")); #if ARRAY_STATIC /* For a static array use : Then the array will persist until you Free it. */ sial_refarray(v, 1); #endif return v; } value * showstrarray(value* va) { value *v1=sial_strindex(va, "foo"); value *v2=sial_strindex(va, "goo"); printf("array[1]=%d\n", sial_getval(v1)); printf("array[2]=%d\n", sial_getval(v2)); sial_addvalarray(va, sial_makestr("gaa"), sial_makebtype(3)); sial_addvalarray(va, sial_makestr("doo"), sial_makebtype(4)); sial_freeval(v1); sial_freeval(v2); return sial_makebtype(0); } value * showintarray(value* va) { value *v1=sial_intindex(va, 1); value *v2=sial_intindex(va, 2); printf("array[1]=%d\n", sial_getval(v1)); printf("array[2]=%d\n", sial_getval(v2)); sial_freeval(v1); sial_freeval(v2); return sial_makebtype(0); } BT_SPEC_TABLE = { { "int mkarray(int i)", mkarray}, { "void showintarray(int i)",showintarray}, { "void showstrarray(int i)",showstrarray}, { 0, 0} }; static char *buf; BT_INIDSO_FUNC() { sial_msg("mkarray initialized\n"); #ifdef ARRAY_STATIC /* we will need a static value to attach the array too */ v=sial_makebtype(0); #endif return 1; } BT_ENDDSO_FUNC() { sial_msg("mkarray being shutdown\n"); #ifdef ARRAY_STATIC sial_freeval(v); /* freing the value decrements the reference count by one. So, if none of the calling macros copied the value to a static sial variable, it will free the array */ #endif } Macro Construction ------------------ When sial as been integrated into an application and a basic set of builtin command as been created, it is time to start creating the macro themselves. Some basic rules and conventions apply to macro construction that make the coding and documenting steps of macro definition easy. I will use the function foo as an example. Function foo is defined in file /usr/tmp/sial/foo. Function foo is a user callable function, meaning that it can be executed by the sial_cmd() function. The command input section of the application can thus call sial_cmd("foo", char *argv, int nargs) to execute the foo macro. ------------ file foo ------------- foo_opt(){ return "ab:c"; } foo_usage(){ return "[-a] [-b barg] [-c] addr [addr [addr...]]"; } foo_help(){ return "This is an example function"; } static int doproc(proc_t *p) { printf("p=0x%p\n", p); } int foo() { int all, i; string barg; if(exists(aflag)) all=1; else all=0; if(exists("bflag")) bval=barg; for(i in argv) { proc_t *p; p=(proc_t*)atoi(argv[i], 16); doproc(p); } } ------------ end of file foo -------------- The application calls sial_load() to load foo. Sial calls back the application with the names of all fucntions declared in that file. The aplication can then register commands for the user to type according to this list of functions. In this case 'foo'. The application then uses sial_cmd() to run a specific command 'foo'. Before executing the command, sial checks if a foo_opt() function exists and if so, calls it. This function returns the proper getopt() argument specification string. If this function does not exists then all arguments are passed down to the foo() function directly. If the arguments supplied by the user do not follow the proper syntax then the function foo_usage() will be called, if it exists. If the foo_usage() function does not exists, a generic error message is generated by sial. If the command 'help foo' is issued, the application should be calling sial_exefunc("help_foo", 0) whish will return a VALUE_S for the help for foo. Or whatever foo_help() returns. Each option, their associated value and addition arguments are made available to the foo funtion by creating the following global variables before the actual call. Each option, if specified, will trigger the existence of flag variable. In the foo() case, this means that variables aflag, bflag and cflag can possibly exist. The function exists("variable name") can then be used to test for this option presence. If an option has an associated value (getopt's ':' is specified on the string foo_opt() returns) this value is made available as a string type variable called Xarg, where X is the option letter. In the case of foo() variable 'string barg' would exist if the -b option was supplied by the user. The rest of the arguments supplied by the user are made available in an array of 'string' called argv. argv[0] is set to the name of the function 'foo' and argc is a global that defines how many argv their are. Builtin functions ================= Here is a description of the current set of builtin functions. unsigned long long atoi(string value [, int base]) Convert a string value to a long long. Base is the base that should be used to process the string e.g. 8, 10 or 16. If not specified, then the standard numeric format will be scnanned for ex: 0x[0-9a-fA-F]+ : hexadecimal 0[0-7]+ : octal [1-9]+[0-9]* : decimal This function is used when converting command line arguments to pointers. Example: void mycommand() { int i; for(i=1;i> mycommand 0xa80000004ab14578 int exists(string name) Checks for the existance of a variable. Returns 1 if the variables does exist and 0 otherwise. This function is mostly used to test if some options were specified on when the macro was executed from command line. It can also be used to test for image variable. example: void mycommand() { if(exists("aflag")) { // user has specified -a option } } void exit() Terminate macro excution now. int getchar() Get a single character from tty. string gets() Get a line of input from tty. string getstr(void *) Gets a null terminated string from the image at the address specified. Sial will read a series of 16 byte values from the image untill it find the \0 character. Up to 4000 bytes will be read this way. string getnstr(void *, int n) Gets n characters from the image at the specified address and returns the corresponding string. string itoa(unsigned long long) Convert a unsigned long long to a decimal string. void printf(char *fmt, ...); Send a formatted message to the screen or output file. For proper allignment of output on 32 and 64 bit systems one can use the %> sequence along with the %p format. On a 32 bit system %p will print a 8 character hexadecimal value and on a 64 bit system it will print a 16 character value. So, to get proper alignment on both type of systems use the %> format which will print nothing on a 64 bit system but will print 8 times the following character on a 32 bit system. example: struct proc *p; printf("Proc %> uid pid\n"); printf("0x%p %8d %8d\n" , p, p->p_uid,p_p_pid); int sial_depend(string file) Loads a macro or directory of macros called 'file'. Contrary to sial_load() it will not give any error messages. Returns 1 on success 0 otherwise. int sial_load(string file) Loads and compiles a sial macro file. returns 1 if successful or 0 otherwise. void sial_unload(string file) Unload's a sial macro file string sprintf(string format, ...) Creates a string from the result of a sprintf. Example: void mycommand() { string msg; msg=sprintf("i=%d\n", i); } The result will be truncated to maxbytes if it would be longer. int strlen(string s) Return the length of string s. string substr(string s, int start, int len) Creates a string from the substring starting a charcater 'start' of 's' for 'len' characters. example: s=substr("this is the original", 6, 2); So 's' will become "is". -------------------------------------------------------- Questions/Comments Luc Chouinard, lucchouina@yahoo.com