Showing posts with label gEDA. Show all posts
Showing posts with label gEDA. Show all posts

Monday, December 03, 2012

Making schematics look good with "gaf export"

<CareBear\> peterbrett : hey. gaf export is f-ing awesome!

People who've been testing the gEDA "master" branch over the last few hours will have got a sneak preview of a cool new tool that will be arriving in gEDA/gaf 1.9.0. The new gaf export command-line utility lets you quickly and easily export your schematics and symbols to a variety of image formats.

I've been wanting to introduce a tool like this for a while, but it's only become possible thanks to finally finishing a couple of big features that have been cooking in my personal branches for a couple of years: a new Cairo-based rendering library for gEDA designed to be used for both rendering in gschem and for printing/exporting graphics, called "libgedacairo"; and a new gEDA configuration subsystem, which I'll write about in more detail another time.

To get started, suppose I want to create a PDF from a schematic called grey_counter_1.sch. It's very straightforward!

gaf export -o grey_counter_1.pdf grey_counter_1.sch

From the output filename that I passed to the "-o" option, gaf export will detect that I want a PDF. It'll detect the size of the drawing, centre it in the default paper (choosing some suitable margins) and generate a PDF file.

Batch generation of PostScript files

Many people previously used gschem along with the (relatively obscure) print.scm script for batch generation of PostScript files. Usually the command looked something like:

gschem -o grey_counter_1.ps -s /usr/share/gEDA/scheme/print.scm grey_counter_1.sch

Don't do this any more. It is slow (because it needs to load all of gschem's configuration), requires an graphical desktop to be running (because gschem can't start without trying to display its windows) and doesn't provide any way to directly customise formatting options without fiddling with Scheme scripts. Also, gaf export generates much nicer PDF output than PS, especially if you want to do anything with the output file other than printing. You could directly replaced the gschem command above with something like:

gaf export -o grey_counter_1.pdf grey_counter_1.sch

A Makefile rule for creating PDF files from schematic files might look like:

%.pdf: %.sch
	gaf export -o $@ -- $<

Of course, one advantage of the new tool is that it can do multi-page output. So rather than generating a whole bunch of separate PDF or PostScript files and stitching them together, you could directly generate a single PDF file with the whole of your design in it:

gaf export -o schematics.pdf grey_counter_1.sch filter_1.sch

Tweaking the output

gaf export also lets you tweak the output for different applications. Suppose I want to produce the PNG file displayed in this blog post. First, I don't care about paper sizes; I want the output file to be sized according to how large the drawing is. To do this, I can use -s auto. I can also set the margin on the output with -m 5px. I also want to print in colour (-c). So the overall command is:

gaf export -c -s auto -m 5px -o gaf_export__40160-1.png 40160-1.sym

It can also be useful to set the paper size (for example, to get suitable margins for larger paper sizes). By default, gaf export uses whatever GTK thinks the default paper size is on your system. For most people, this will be ISO A4. In addition to providing measurements directly via the -s option, the -p option lets you specify a PWG 5101.1-2002 paper name. For example, to use US "D" size paper:

gaf export -p na_d -o grey_counter_1.pdf grey_counter_1.sch

Changing default settings

The default settings for gaf export can be modified using the new gaf config command. For example, to set the default paper size for all your projects to US "Letter":

gaf config --user export paper na_letter

Or to make sure that all printing for a particular project is in colour:

gaf config -p /path/to/project/directory/ export monochrome false

Conclusion

gaf export is a fast, easy-to-use way of generating graphics files from your gEDA/gaf schematics and symbols. Along with several other new features, it will be available in the upcoming unstable gEDA/gaf 1.9.0 release.

Wednesday, December 28, 2011

Coming up to gEDA 1.7.2

Later this week, gEDA/gaf 1.7.2 will be released! Hopefully, this will be the final unstable release before gEDA/gaf 1.8.0, and it will contain a few neat new things that we've been working on over the last six months.

  • As I've discussed previously, this release will incorporate all of the basic Scheme API functionality, to make it easier to write extensions for gschem in pure Scheme. I've had quite a few e-mails and feature requests from people who have been doing interesting things with the new API, and I'm looking forward to hearing about more tools for enhancing gschem! All of the new functions are documented in the gEDA Scheme Reference Manual.
  • In terms of documentation, I've completely re-written the gschem User Guide, more fully describing many features in gschem that were previously poorly documented, and hopefully making it easier to find the information you need. Gareth Edwards has done a fantastic job of adding man pages to gEDA, and now all of the programs we install have a properly-written man page.
  • The first parts of my rework of keymapping will also be included. Now key names will be properly internationalised when they're displayed, and having Caps Lock enabled will no longer mess up your keybindings. You can also now bind key combinations that use the Mod, Super or Hyper modifier keys.
  • You can now read gEDA documentation on Windows! (About time, really).

What's next once 1.7.2 has been released? Well, the most important thing is going to be to get a new stable release out. Many distributions are keen to move to Guile 2.0.x, and the current stable version of gEDA (1.6.2) doesn't support it. My current hope is to go into a string freeze for gEDA 1.8.0 very soon after gEDA 1.7.2 is released, and aim to get all the translations updated in time for a stable release at the end of January 2012.

There are many exciting plans for the 1.9.x branch of gEDA, but I'll leave them for another time...

Thursday, June 30, 2011

Scheme API merge, and keybinding in gschem

Earlier this week I merged the gEDA Scheme API that I've been working on since late 2009 into the main gEDA unstable branch. It's been a pretty great chunk of work, and I'm glad to have been able to get it into a state where it's generally useful. It's been pretty fun working out how to answer users' feature requests on gEDA-user with a short Scheme function, too! Hopefully people will take this new feature and do cool things with it.

One of the main obstacles to doing cool things with the new Scheme API is that it's difficult to modify key bindings on the fly — gschem doesn't have any equivalent to Emacs' global-set-key, for example. That makes it hard for e.g. an rc file or Scheme plugin loaded after system-gschemrc to add new keybindings or modify existing ones. There's also only a single keymap, so there's no possibility for user-modifiable state-specific keymaps (e.g. for binding keys that the user presses while placing a symbol).

While on the subject of keybindings, the way gschem currently displays keybindings in the menu bar is a little bit rubbish — for example, it would be good to display "]" instead of "bracketright", and "o S" instead of "oShift S". It's also a shame that we don't support a wider range of modifier keys, including "super" ("⊞" on most keyboards).

I'm planning to fix all these issues by adding a comprehensive set of Scheme functions for working with keys, key sequences and key bindings to gschem. Keymaps will probably become a hashtable-based record type, and there will be functions for creating a key sequence from a string and vice versa. There will also be a function for obtaining a "pretty" string describing a key, suitable for displaying in a menu or in the status line.

Saturday, June 25, 2011

The gEDA Scheme API manual is here

I'm currently working to get my guile-scheme-api branch ready to merge into the main unstable gEDA branch (master). One of the main jobs that needed to get done before that's possible was to write a reference manual for all of the Scheme functions and variables that the new Scheme API provides, and I've now done that!

If you're interested in writing extensions for gEDA, please take a look at the draft gEDA Scheme Reference Manual and let me know what you think. I'm keen to receive any feedback you might have, whether about the manual or about the API itself.

I'm hoping to merge my work so far in this branch into unstable within the next week. At the same time, I'm aiming to add the following missing functionality as soon as possible:

  • Object transformations (translation, rotation and mirroring)
  • Picture object creation and manipulation
  • Path object creation and manipulation (quite tricky, unfortunately)
  • File saving and loading
  • Picture and component embedding/unembedding

These are all things that I'm sure extension authors would be quite interested in having available to them! Hopefully we'll see all sorts of interesting new capabilities for gEDA users coming out soon.

Thursday, June 16, 2011

gEDA and Guile — opportunities to get involved

This is the eighth and last in a series of posts on extensibility in gEDA using Guile Scheme.

  1. Finding Scheme API code in gEDA
  2. Compiling against multiple Guile versions
  3. Safe handling of non-local exits
  4. Dealing with deprecated libguile functions
  5. Checking arguments to Scheme functions in C
  6. How and when to use Scheme errors
  7. Reducing boilerplate with "snarfing macros"
  8. Opportunities to get involved

In this post, I'll summarise the series. In addition, I'll recap on the opportunities to get involved in gEDA development offered by the problems I've identified with the current gEDA codebase.

In my first and second posts, I described whereabouts in the gEDA codebase to find code which uses Guile, and also how to set up a build environment to easily compile gEDA against either Guile 1.8 or Guile 2.0. Being able to test with the latest stable release of Guile is quite important, and should definitely be considered by all gEDA developers!

In my third post, I dealt with probably the most important issue: dealing with non-local execution due to Scheme exceptions and continuations. There are a lot of places in gEDA where it's assumed that a call to a libguile or Scheme function will return normally, and that's not a safe assumption. Failing to protect against that (e.g. by using the "dynamic wind" mechanism) can easily result in resources being leaked.

The following post explained how to get notified when gEDA uses deprecated Guile APIs. It also gave an example of an instance where using deprecated API long after its deprecation has lead to actual bugs — the SCM_STRING_CHARS() and SCM_SYMBOL_CHARS() macros.

I went on to talk about the use of Scheme errors. I described the use of the SCM_ASSERT() macro to check function arguments, and also how to use Scheme errors more generally. The main points to remember here were that libguile has some really useful undocumented helper functions for raising Scheme errors from C code (see libguile/error.h), and that just because SCM_ASSERT() has "assert" in its name doesn't mean that you don't have to be careful about resource management if the test fails.

Finally, I talked about the guile-snarf tool that's used by my guile-scheme-api branch to generate boilerplate code for defining Scheme functions and variables from gEDA's C code.

There are several opportunities to get involved in making gEDA better. All these jobs are good for people who aren't that familiar with gEDA development, but who want to get to learn the codebase and about the development process, because they can mostly be done just on a function-by-function basis.

  • You can help change places where SCM_STRING_CHARS() is still used, to use scm_to_utf8_string() instead. I've already had some patches from Ivan Stankovic to deal with some of these!
  • You can look at the places where gEDA calls into libguile or calls Scheme functions to make sure that any potential errors are handled properly.
  • You can double-check that all gEDA functions that take SCM arguments validate them properly using SCM_ASSERT().
  • Or you can help make sure that gEDA Scheme functions raise Scheme errors properly when things go wrong.

If you're interested in submitting patches addressing any of these, or have any questions, please get in touch.

As previously mentioned, my focus for gEDA development in the near future is going to be on getting my guile-scheme-api branch merged, and I encourage everybody to try it out and let me know what you think! For information on my various branches and how to get hold of them, please see my gEDA development page.

Wednesday, June 15, 2011

gEDA and Guile — reducing boilerplate with "snarfing macros"

This is the seventh in a series of blog posts on extensibility in gEDA using Guile Scheme.

  1. Finding Scheme API code in gEDA
  2. Compiling against multiple Guile versions
  3. Safe handling of non-local exits
  4. Dealing with deprecated libguile functions
  5. Checking arguments to Scheme functions in C
  6. How and when to use Scheme errors
  7. Reducing boilerplate with "snarfing macros"
  8. Opportunities to get involved

In this post, I will explain how you can use the guile-snarf tool to reduce the boilerplate needed to define Scheme functions and variables from C.

At the time of writing, this post isn't really relevant to the main gEDA unstable development branch, since it doesn't currently use "snarfing". It might be an interesting read if you're interested in hacking on my guile-scheme-api branch.

Why is "snarfing" needed?

One of the problems with using libguile to define Scheme functions from C is there's a certain amount of boilerplate setup code. For example, if I wish to define a function called "myfunc" in C to be usable from Scheme, I need to do something like:

static SCM
my_func (SCM arg)
{
  SCM_ASSERT (scm_is_string (arg), arg, SCM_ARG1, "my-func");

  /* ... do something useful ... */
}

/* Called during application initialisation */
void
my_init ()
{
  scm_c_define_gsubr ("my-func", 1, 0, 0, (SCM (*)()) my_func);
}

There are a few problems with this.

  • If the number of arguments to the function changes, an update needs to be made in two different places.
  • If the string name of the function changes ("myfunc" in this case), an update needs to be made in two places, and also to every other place the name is used (e.g. SCM_ASSERT() for arguments and any other place a Scheme error is raised).
  • If an extra function is added...

You get the idea.

Similarly, if there is a permanent Scheme value that's needed in multiple functions (such as a symbol) it's necessary to separately define the static variable and add a line to the initialisation function.

Finally, if the libguile API for defining a function or variable changes, it's necessary to individually update the initialisation code for every single one.

The guile-snarf tool

Guile comes with a tool called guile-snarf, which is run against a C source file to generate an additional C source file for inclusion, usually given a .x extension.

The tool recognises some special macros in the C source, which expand normally when compiled, but which are used by the tool to generate Scheme initialisation boilerplate. The example above would become:

SCM_DEFINE (my_func,   /* Function name in C */
            "my-func", /* Function name in Scheme */
            1, 0,      /* No. of required/optional args */
            0,         /* Whether accepts "rest" arg */
            (SCM arg), /* C argument list */
            "Do something exciting.") /* Docstring */
{
  SCM_ASSERT (scm_is_string (arg), arg, SCM_ARG1, s_my_func);

  /* ... do something useful ... */
}

/* Called during application initialisation */
void
my_init ()
{
  #include "mycfile.x";
}

This may seem like a small gain for this trivial example, but for files which define a large number of Scheme procedures there's a definite benefit! There are a few things to note:

  • Functions defined using SCM_DEFINE() are always static. This shouldn't normally be a problem, since they will usually be intended to be called from Scheme, not from other C functions.
  • For each function foo, SCM_DEFINE() also defines a static string called s_foo. This can be pretty handy, especially when needing to pass the Scheme name of the function to SCM_ASSERT() macros. (Its use is discouraged by the official documentation, but libguile itself makes use of it, so hey — it can't be that much of a problem!)

Although SCM_DEFINE is the main reason to use guile-snarf, there are several other really useful macros, such as SCM_SYMBOL("foo"), which creates & defines a variable containing the symbol "foo". In my guile-scheme-api branch, you will fairly commonly see a block at the top of a C source file looking like:

SCM_SYMBOL (lower_left_sym , "lower-left");
SCM_SYMBOL (middle_left_sym , "middle-left");
SCM_SYMBOL (upper_left_sym , "upper-left");
SCM_SYMBOL (lower_center_sym , "lower-center");
SCM_SYMBOL (middle_center_sym , "middle-center");
SCM_SYMBOL (upper_center_sym , "upper-center");
SCM_SYMBOL (lower_right_sym , "lower-right");
SCM_SYMBOL (middle_right_sym , "middle-right");
SCM_SYMBOL (upper_right_sym , "upper-right");

This then allows functions in the file to use the symbols to construct Scheme expressions to be evaluated without needing to recreate the symbols (e.g. using scm_from_utf8_symbol()) every time the function is run.

Conclusion

Currently, when wishing to alter the prototype a Scheme function provided by a gEDA program, you need to make changes in at least three places:

  1. The function definition itself.
  2. The function prototype in a header file.
  3. The function metadata exported to Scheme, usually in a file called g_register.c.

Using the SCM_DEFINE() "snarfing" macro and the guile-snarf tool would allow two of those places to be eliminated, and that's why my guile-scheme-api branch uses them.

The next and final post in this series will summarise the things I've written about, and give a list of easy introductory gEDA development tasks based on some of the issues I've raised.

Monday, June 13, 2011

gEDA and Guile — how and when to use Scheme errors

This is the sixth in a series of blog posts on extensibility in gEDA using Guile Scheme.

  1. Finding Scheme API code in gEDA
  2. Compiling against multiple Guile versions
  3. Safe handling of non-local exits
  4. Dealing with deprecated libguile functions
  5. Checking arguments to Scheme functions in C
  6. How and when to use Scheme errors
  7. Reducing boilerplate with "snarfing macros"
  8. Opportunities to get involved

In this post, I will try to make it clear how and when to raise Scheme errors in procedures implemented in C.

As explained in an earlier blog post in this series, Guile has a fully-fledged exception mechanism, which is used as the basis for its error reporting system. Because this allows execution to jump non-locally when an error occurs, it requires resources to be carefully managed.

Other than checking types using the SCM_ASSERT() macro, when else might it be useful to raise Scheme errors from C functions in gEDA?

When to raise Scheme errors

In Scheme functions written in Scheme, the answer is obvious: raise a Scheme error using error or scm-error. In C, it's not so simple.

Recall that there are currently four approaches to error handling in gEDA's C source code:

  1. Give up (log an error message and quit immediately).
  2. The GError mechanism from GLib.
  3. Best-effort (log an warning message and return a default value).
  4. Raise a Scheme error.

We try and avoid the "giving up" option, because quite often the situation is not actually hopeless, and the error could be handled by a function further up the stack or at the very least the user might be able to save his or her data.

In functions that are actual C API (i.e. designed primarily to be called by other C functions), Scheme errors are never appropriate, because they're quite complex to catch and handle in C. It's much easier to get from a GError to a Scheme error than the other way round! These sorts of functions should either use GError (for run-time errors outside the developers' control, such as being asked to load a file that doesn't exist), or use a "best-effort" approach (for programming errors, e.g. return NULL when asked for a list of objects in a NULL page).

On the other hand, functions that are designed primarily to be called from Scheme code should exclusively use Scheme errors. If there's a problem, either it'll be caught and handled, or it won't and an error message will be generated (and the program may exit).

This means that you shouldn't use best-effort helper macros like g_return_if_fail() — if you're called with invalid arguments, generate a Scheme error (e.g. using SCM_ASSERT()). You can apply a simple rule of thumb: if the SCM type appears in the function prototype, you should normally use Scheme error reporting.

How to raise Scheme errors

With libguile, the main function to use to raise Scheme errors is scm_error_scm(). However, the libguile headers provide some additional undocumented helper functions which make raising errors from C functions much more convenient. For example:

  • scm_error() is similar to scm_error_scm() but allows you to pass the raising function's name as a char * string rather than a Scheme string.
  • scm_misc_error() is the easiest way to generate errors for which it's not worth assigning a specific error key, and is the C equivalent to the error function in Scheme.

A full list of functions for raise errors is available in libguile/error.h, and since they are much better than scm_error_scm(), use them.

Don't forget that raising a Scheme error causes a non-local exit, and make provisions (such as using dynamic wind) to release any resources you hold correctly.

Conclusion

Guile provides a full-featured error-reporting system, which gEDA applications should use in their Scheme APIs more than they currently do. The best way to raise a Scheme error isn't via scm_error_scm(), but through the undocumented helper functions that libguile provides (and I should probably submit a documentation patch to upstream Guile to fix that...)

In my next post, I'll talk about the guile-snarf tool, and how to use Guile's "snarfing macros" to simplify the boilerplate needed to export C functions as callable from Scheme.

Sunday, June 12, 2011

gEDA and Guile — checking arguments to Scheme functions in C

This is the fifth in a series of blog posts on extensibility in gEDA using Guile Scheme.

  1. Finding Scheme API code in gEDA
  2. Compiling against multiple Guile versions
  3. Safe handling of non-local exits
  4. Dealing with deprecated libguile functions
  5. Checking arguments to Scheme functions in C
  6. How and when to use Scheme errors
  7. Reducing boilerplate with "snarfing macros"
  8. Opportunities to get involved

In this post I'll explain how to use the SCM_ASSERT() macro to check arguments passed to Guile functions written in C using libguile.

Using SCM_ASSERT()

SCM_ASSERT() is a shorthand way to raise a Scheme wrong-type-arg error for a Scheme variable if a test fails. For example, if there is a function which takes a string as an argument, you might write it as:

SCM
myfunc (SCM str)
{
  SCM_ASSERT (scm_is_string (str), str, SCM_ARG1, "my-func");

  /* ... do things with str ... */
}

All functions in gEDA which are exposed as Scheme procedures must use SCM_ASSERT() to thoroughly check the types of their arguments.

There are a few things to note:

  • The first parameter to SCM_ASSERT() is a test, and should evaluate to an integer, not a Scheme value. If the test is zero, an error is signalled.
  • If you need to use a test which evaluates to a Scheme value (e.g. scm_list_p()), don't compare it to SCM_BOOL_F or SCM_BOOL_T directly; use scm_is_true() or scm_is_false().
  • The second parameter is the variable which has the wrong type. Although the test might not be on it directly (e.g. if you are checking for a list of strings), the second parameter must always be the actual argument which was passed to myfunc.
  • The third parameter to SCM_ASSERT() indicates which argument to myfunc had the wrong type — and remember that this is 1-indexed. Although this parameter is just an integer, you should always use the SCM_ARG1, SCM_ARG2 etc. macros for arguments up to 7, since this makes the purpose of the parameter more readable.
  • If you have a function that takes an arbitrary number of parameters, you should use the SCM_ARGn macro as the argument index.
  • The final parameter is a char * string containing the name of the function as seen by Scheme ("my-func" in this case).

If you have a C helper function which is used by many similar Scheme functions, you can generate more useful error messages if you pass it the name of the function that's actually visible to Scheme code. For example:

void
helperfunc (SCM str, const char *funcname)
{
  SCM_ASSERT (scm_is_string (str), str, SCM_ARGn, funcname);

  /* ... do things ... */
}

void
myfunc (SCM str)
{
  helperfunc (str, "my-func");

  /* ... do more things ... */
}

Don't leak resources!

The most important thing to remember is that SCM_ASSERT() does not behave like most normal assertion mechanisms when programming in C (g_assert() in GLib for example), in that it doesn't end the program if it fails — it simply causes a non-local exit by raising an error.

Because SCM_ASSERT() might not return, it's really important to be careful about managing resources when writing a Scheme function in C. For example, consider a function that needs to convert a Scheme list of strings into a GList of C strings. Here's a naive, broken implementation:

/* Don't do this -- it leaks memory */
void
process_string_list (SCM lst)
{
  GList *glst = NULL;
  SCM_ASSERT (scm_is_true (scm_list_p (lst))), lst, SCM_ARG1,
              "process-string-list");

  for (SCM iter = lst; iter != SCM_EOL; iter = SCM_CDR (iter)) {
    SCM str = SCM_CAR (iter);
    SCM_ASSERT (scm_is_string (str)), lst, SCM_ARG1, "process-string-list");
    glst = g_list_prepend (glst, scm_to_utf8_string (str));
  }

  /* ... do something with glst ... */
}

If the SCM_ASSERT() inside the loop raises an error, the stack will unwind out of process_string_list() and the memory allocated in glst will be leaked. Instead, you could check the input, and then build the list:

void
process_string_list (SCM lst)
{
  GList *glst = NULL;
  SCM_ASSERT (scm_is_true (scm_list_p (lst))), lst, SCM_ARG1,
              "process-string-list");

  /* Check the input is actually a list of strings */
  for (SCM iter = lst; iter != SCM_EOL; iter = SCM_CDR (iter)) {
    SCM_ASSERT (scm_is_string (SCM_CAR (iter)), lst, SCM_ARG1, "process-string-list");
  }

  /* Convert to a GList */
  for (SCM iter = lst; iter != SCM_EOL; iter = SCM_CDR (lst)) {
    glst = g_list_prepend (glst, scm_to_utf8_string (SCM_CAR (iter)));
  }

  /* ... do something with glst ... */
}

The problem with this is that it is inefficient, as it requires iterating over the linked list twice. A more efficient approach would be to use dynamic wind, as I discussed in a previous post in this series.

/* A function for freeing a GList containing strings */
static void
free_string_glist (void *data)
{
  GList *glst = *((GList *) data);
  for (GList *iter = glst; iter != NULL; iter = g_list_next (iter)) {
    free (iter->data);
  }
  g_list_free (glst);
}

void
process_string_list (SCM lst)
{
  GList *glst = NULL;
  SCM_ASSERT (scm_is_true (scm_list_p (lst))), lst, SCM_ARG1,
              "process-string-list");

  /* Begin a dynamic extent, and register a handler to clean up on error */
  scm_dynwind_begin (0);
  scm_dynwind_unwind_handler (free_string_glist, (void *) &glst, 0);

  /* Convert to a GList, checking types */
  for (SCM iter = lst; iter != SCM_EOL; iter = SCM_CDR (iter)) {
    SCM str = SCM_CAR (iter);
    SCM_ASSERT (scm_is_string (str)), lst, SCM_ARG1, "process-string-list");
    glst = g_list_prepend (glst, scm_to_utf8_string (str));
  }

  /* End dynamic extent */
  scm_dynwind_end ();

  /* ... do something with glst ... */
}

Update: Thanks to Ivan Stankovic for pointing out a bug in this example!

Unfortunately, there appear to be quite a few places in gEDA where there is the potential for resource leakage like this to occur, and checking that all functions which use SCM_ASSERT() do so in a safe manner would be a really useful thing to do.

Conclusion

As you can tell, even doing something as conceptually simple as checking your arguments safely can be a tricky thing to get right when using libguile — another argument, if you needed one, for keeping the number of functions in gEDA's Scheme API as small as possible while still exposing all of the necessary functionality.

In my next post, I'll discuss how (and when!) in general to throw Scheme errors from C.

Saturday, June 11, 2011

gEDA and Guile — dealing with deprecated libguile functions

This is the fourth in a series of blog posts on extensibility in gEDA using Guile Scheme.

  1. Finding Scheme API code in gEDA
  2. Compiling against multiple Guile versions
  3. Safe handling of non-local exits
  4. Dealing with deprecated libguile functions
  5. Checking arguments to Scheme functions in C
  6. How and when to use Scheme errors
  7. Reducing boilerplate with "snarfing macros"
  8. Opportunities to get involved

In this post I'll explain how to set up your environment to get warnings about use of deprecated libguile API, and some of the most problematic deprecated API usage in gEDA.

Getting notified about deprecated API usage

Most gEDA applications use something like the following to disable deprecation warnings unless you specifically set your environment to request them:

if (getenv ("GUILE_WARN_DEPRECATED") == NULL)
  putenv ("GUILE_WARN_DEPRECATED=no");

The simplest way to enable deprecation warnings is, therefore, to add something like this to your shell rc file:

GUILE_WARN_DEPRECATED="detailed"
export GUILE_WARN_DEPRECATED

Now when you start gschem, you'll get a few warnings on stderr (depending on which version of Guile you've compiled against). Here's an example of what you see when starting gschem if you compile against Guile 2.0:

$ gschem
`(debug-enable 'debug)' is obsolete and has no effect. (1)
Remove it from your code.
SCM_STRING_CHARS is deprecated. See the manual for alternatives. (2)
SCM_SYMBOL_CHARS is deprecated. Use scm_symbol_to_string. (3)

Some of this is a problem, some of it isn't.

  1. We use (debug-enable 'debug) in system-gafrc to enable backtraces in Guile 1.8, but it does nothing in Guile 2.0. It would be nice to find a way to only call it if necessary, but it's not really a priority.
  2. SCM_STRING_CHARS() is used a lot in gEDA to obtain a pointer to the underlying character buffer of a Scheme string, and it's been deprecated since Guile 1.8.0. gEDA's reliance on this is a bit of a problem, unfortunately, and I'll explain why later.
  3. SCM_SYMBOL_CHARS() is similar to SCM_STRING_CHARS() (but for the string representation of a Scheme symbol), and has also been deprecated for a long time.

The problem with SCM_STRING_CHARS() and SCM_SYMBOL_CHARS()

So, why is using SCM_STRING_CHARS() a problem? The main issue is that its use assumes that Guile's internal string representation is the same as gEDA's, i.e. an array of char, and that Guile's internal string encoding is the same as gEDA's, i.e. UTF-8. Neither of these assumptions are reliable.

There are two different internal representations for strings in Guile 2.0. All strings are stored as an array of Unicode code points.

  • If all the code points are in the range 0-255 inclusive, the code points are stored with one byte per code point, i.e. as Latin-1 or ISO-9959-1. This is not UTF-8.
  • If any of the code points is outside that range, the whole string is stored with four bytes per code point, i.e. as UTF-32. This is also not UTF-8.

Additionally, Guile 2.0 introduces read-only strings (which don't work with SCM_STRING_CHARS()) and shared substrings (which don't work with SCM_STRING_CHARS()).

So how can we break a function func that takes a single string argument and uses SCM_STRING_CHARS()? Let me count the ways.

  1. We can pass it a string containing code points above 255. Since we target a worldwide user base these days, that's not particularly unlikely.

    (func "你好")
  2. We can pass it a shared substring.

    (func (substring/shared "foo bar" 0 3))
  3. We can pass it a read-only substring

    (func (substring/read-only "foo bar" 0 3))

SCM_SYMBOL_CHARS() shares all these problems. Both these macros have been deprecated since Guile 1.8.0 exactly because the Guile developers wanted to be able to change Guile's original internal string representation to support Unicode fully. (In case you're wondering why they don't use UTF-8, it's because Scheme requires a bunch of string operators that operate on the nth character in a string, and using UTF-8 would make those operators much slower).

Replacing SCM_STRING_CHARS()

What's the alternative? Ideally, we'd use the rather handy scm_to_utf8_stringn() function, but that was only introduced in Guile 2.0 (along with Unicode support), so it's not an option. Instead, we have to rely on scm_to_locale_string(). The main difference between the new functions and SCM_STRING_CHARS() is that the new functions allocate memory, which must be freed with free() (n.b. not g_free()).

Update: gEDA now provides scm_to_utf8_string() and scm_from_utf8_string() even if Guile doesn't, so always use them unless you actually want to work with locale-encoded strings.

So suppose we started off with a version of myfunc() that uses SCM_STRING_CHARS():

void
myfunc (SCM arg)
{
  /* N.b. we should check that arg is in fact a string */
  printf ("%s", SCM_STRING_CHARS (arg));
}

It should be replaced by:

void
myfunc (SCM arg)
{
  char *arg_str;
  /* N.b. we should check that arg is in fact a string */
  arg_str = scm_to_utf8_string (arg);
  printf ("%s", arg_str);
  free (arg_str);
}

In reality, you'll want to do something more complicated than just print the string. Don't forget that if you do more than trivial calls into libguile in between creating arg_str and freeing it you should probably use dynamic wind to make sure that it is properly cleaned up.

You should also be aware that because scm_to_utf8_string() tries to return a null-terminated string, it throws an error if the string contains #\nul characters. It can throw an error if the string can't be converted to the requested encoding (which is locale-dependent for scm_to_locale_string()). This introduces its own challenges.

Replacing SCM_SYMBOL_CHARS()

Replacing SCM_SYMBOL_CHARS() is similar to replacing SCM_STRING_CHARS(); simply use scm_symbol_to_string() to convert the symbol to a string, and then use scm_to_utf8_string() as before.

Conclusion

Library APIs are rarely deprecated without a good reason, and being aware of and proactive about updating deprecated API usage can help avoid some serious problems. Updating gEDA to remove the use of the SCM_STRING_CHARS() and SCM_SYMBOL_CHARS() macros is an important job. Nevertheless, it would still be quite accessible for someone less familiar with the gEDA code base, as it can be done by dealing with one function at a time.

In my next post, I will describe how to use SCM_ASSERT to check types of SCM arguments to functions that use libguile.

Friday, June 10, 2011

gEDA and Guile — safe handling of non-local exits

This is the third in a series of blog posts on extensibility in gEDA using Guile Scheme.

  1. Finding Scheme API code in gEDA
  2. Compiling against multiple Guile versions
  3. Safe handling of non-local exits
  4. Dealing with deprecated libguile functions
  5. Checking arguments to Scheme functions in C
  6. How and when to use Scheme errors
  7. Reducing boilerplate with "snarfing macros"
  8. Opportunities to get involved

In this post I'll be talking about an issue which the current gEDA codebase doesn't deal with very well: dealing with non-local exits in C code that uses libguile.

What are non-local exits?

Non-local exits are a way for execution to jump out of the current execution context to another point in the program. The only way of doing this in C is through the quite limited setjmp/longjmp mechanism .Guile supports two fully-featured constructs for non-local control flow: exceptions and continuations.

Exceptions will be fairly familiar to many developers, as they are heavily used in languages such as Java and Python. When an exception is thrown, the stack is unwound until an exception handler is found for the exception. If none is found, then the program will usually exit with an uncaught exception error. Guile uses exceptions to implement its error handling mechanism, and you can read about that in more detail in the Guile manual (Exceptions, Error Reporting).

The other way that a non-local exit might occur is via a continuation. These are pretty hard to get your head around, unfortunately. The Guile manual describes them as follows:

A “continuation” is the code that will execute when a given function or expression returns. For example, consider

(define (foo)
  (display "hello\n")
  (display (bar)) (newline)
  (exit))

The continuation from the call to bar comprises a display of the value returned, a newline and an exit. This can be expressed as a function of one argument.

(lambda (r)
  (display r) (newline)
  (exit))

In Scheme, continuations are represented as special procedures just like this. The special property is that when a continuation is called it abandons the current program location and jumps directly to that represented by the continuation.

A continuation is like a dynamic label, capturing at run-time a point in program execution, including all the nested calls that have lead to it (or rather the code that will execute when those calls return).

Guile implements continuations that are usable from C code with some pretty crazy techniques for saving and restoring the C stack.

The important point to take away from this is that when you call a Scheme function from C, you cannot depend on it returning normally. That means that totally legitimate resource management in a normal C/GLib program, such as:

void myfunc()
{
  gchar *buf = g_strdup ("const string");
  anotherfunc (buf);
  g_free (buf);
}

becomes unsafe if anotherfunc() calls Scheme code that might either call a continuation or throw an exception, since in that case the call to g_free() would never occur and the memory assigned to buf will be leaked.

In gEDA there are several places which do exactly this, and they need fixing.

Dealing with non-local exits safely

Fortunately, Guile provides two mechanisms for dealing with this, called Continuation Barriers and Dynamic Wind.

Continuation barriers are the less powerful of the two mechanisms, but are simpler to understand. They simply block both exceptions and continuations from entering or leaving the context of the barrier call. Here is how to modify the previous example to use a continuation barrier:

void myfunc()
{
  gchar *buf = g_strdup ("const string");
  scm_c_with_continuation_barrier (anotherfunc, buf);
  g_free (buf);
}

There's an obvious disadvantage to this approach: you can't catch and handle exceptions which occur, or allow exceptions to propagate upward to the function that called myfunc() in case they can be caught and handled there!

Dynamic wind provides a more flexible method. The scm_dynwind_start() and scm_dynwind_end() functions delimit a dynamic extent, and it's possible to register actions to be carried out whenever the dynamic extent is entered or left (e.g. due to a continuation or exception). It's also possible to indicate that non-local exits from the dynamic extent are permitted, but non-local entries aren't (this is the usual thing to do when using dynamic wind from C code).

Here is myfunc() modified to use dynamic wind:

void myfunc()
{
  /* Begin a dynamic extent that can't be re-entered */
  scm_dynwind_begin (0);

  gchar *buf = g_strdup ("const string");

  /* Make sure that buf is freed when the dynamic extent is
   * left either locally or non-locally */
  scm_dynwind_unwind_handler (g_free, buf, SCM_F_WIND_EXPLICITLY);

  anotherfunc (buf);

  /* End dynamic extent */
  scm_dynwind_end ();
}

This ensures that buf is always freed. If anotherfunc() raises an exception, or it calls a continuation that jumps out of myfunc(), the unwind handler will make sure that the string is freed. Since the SCM_F_WIND_EXPLICITLY flag was passed to scm_dynwind_unwind_handler(), the handler will also be called if the call to scm_dynwind_end() is reached and the function returns normally.

Conclusion

One task that's well past due is a review of functions in gEDA that call into Scheme to make sure that they safely handle non-local exits, either by using continuation barriers, dynamic wind, or the "protected" functions provided in libgeda (see libgeda/src/g_basic.c). This would be ideal for someone less familiar with the code base to tackle, since it can be approached on a function-by-function basis.

In my next post, I will describe how to set up your environment to get notifications about use of deprecated libguile API usage in gEDA, and in particular the problems caused by the continued use of the long-deprecated SCM_STRING_CHARS macro.

Thursday, June 09, 2011

gEDA and Guile — compiling against multiple Guile versions

This is the second in a series of blog posts on extensibility in gEDA using Guile Scheme.

  1. Finding Scheme API code in gEDA
  2. Compiling against multiple Guile versions
  3. Safe handling of non-local exits
  4. Dealing with deprecated libguile functions
  5. Checking arguments to Scheme functions in C
  6. How and when to use Scheme errors
  7. Reducing boilerplate with "snarfing macros"
  8. Opportunities to get involved

In this post, I'll explain how to set up your build environment in such a way that you can easily compile gEDA against either Guile 1.8 or Guile 2.0, since we need to support using gEDA with either version of Guile at the moment.

Firstly, you need to build and install Guile 2.0 and Guile 1.8 to their own prefixes, e.g. /opt/guile-2.0/ and /opt/guile-1.8/.

$ wget ftp://ftp.gnu.org/gnu/guile/guile-2.0.1.tar.gz
$ tar -zxf guile-2.0.1.tar.gz
$ cd guile-2.0.1
$ ./configure --prefix=/opt/guile-2.0
$ make
$ sudo make install

$ wget ftp://ftp.gnu.org/gnu/guile/guile-1.8.8.tar.gz
$ tar -zxf guile-1.8.8.tar.gz
$ cd guile-1.8.8
$ ./configure --prefix=/opt/guile-1.8
$ make
$ sudo make install

Next, you need to have some way of quickly adding and removing the different versions of Guile to your environment. Specifically, you need to be able to modify INFOPATH, PATH, MANPATH, LD_LIBRARY_PATH and PKG_CONFIG_PATH.

The approach I use is to use Environment Modules (henceforth referred to as modules). The idea is to be able to do something like this:

$ module load guile/1.8
$ which guile
/opt/guile-1.8/bin/guile
$ module switch guile guile/2.0
$ which guile
/opt/guile-2.0/bin/guile

One particularly neat thing about modules is that although I give all my examples here for POSIX sh and/or bash, the same module files can be used with other shells such as tcsh with no modification.

Installing & setting up Environment Modules

Unfortunately, modules can be a bit of a pain to install, and many distributions don't appear to package it. This is a guide to getting it working on Ubuntu 10.04; it's much easier on Fedora, since there you can just install the environment-modules package.

Due to a really obnoxious default file layout that's not at all trivial to fix, it's easiest to install modules to its own prefix, e.g. /opt/Modules. You'll need to do something like this (after downloading the source code):

$ tar -zxf modules-3.2.8a.tar.gz
$ ./configure --disable-versioning --prefix=/opt
$ make
$ sudo make install

It's then necessary to set up your ~/.bash_profile and ~/.bashrc to enable modules for your sessions. Firstly, you need to add some boilerplate to the top of ~/.bash_profile:

if [ -f /opt/Modules/init/bash ]
then
# Set the file to store your initial environment in
export MODULESBEGINENV=$HOME/.modules/beginenv
# Initialise environment for modules
. /opt/Modules/init/bash
# Set a directory for your personal module files (create this directory)
module use $HOME/.modules
# Load some default modules
module load guile
fi

It's also useful add this to your ~/.bashrc, to make sure that when you start a new shell your environment is preserved:

if [ ! -z "$MODULESHOME" ]; then
module() { eval `$MODULESHOME/bin/modulecmd bash $*`; }

module update # Need to reload modules because of setuid utmp
fi

Creating modules for Guile versions

The next step is to make the module files for the two versions of Guile. These will be called /opt/Modules/modulefiles/guile/1.8 and /opt/Modules/modulefiles/guile/2.0, and will look like this:

#%Module1.0
prepend-path PATH /scratch/opt/guile-2.0/bin
prepend-path LD_LIBRARY_PATH /scratch/opt/guile-2.0/lib
prepend-path PKG_CONFIG_PATH /scratch/opt/guile-2.0/lib/pkgconfig
prepend-path INFOPATH /scratch/opt/guile-2.0/share/info
prepend-path MANPATH /scratch/opt/guile-2.0/share/man

with 2.0 replaced by 1.8 for /opt/Modules/modulefiles/guile/1.8.

Finally, it's necessary to tell modules which version of Guile should be the default, for when a user asks for module load guile. We can do this by adding a /opt/Modules/modulefiles/guile/.modulerc file.

#%Module1.0
set ModulesVersion 2.0

Putting it all together

After re-initialising your session (probably easiest to log out and back in again), you should be able to swap between versions of Guile using module load and module switch as described above.

You should now be easily able to compile gEDA against either version of Guile without too much difficulty — simply load the appropriate environment module, and then run:

$ ./config.status --recheck && ./config.status && make

In my next post I will discuss one of the problems that's quite common in gEDA's use of Guile: unsafe behaviour on non-local exits such as Scheme exceptions.

Wednesday, June 08, 2011

gEDA and Guile — finding Scheme API code in gEDA

The response to my previous post was overwhelmingly in favour of continuing and merging my work on gEDA extensibility using Guile Scheme, both here and on the gEDA mailing lists. Some people asked how they could help out, and so I'm going to write a series of blog posts with some information on how gEDA uses Scheme and some easy introductory tasks that would be useful.

  1. Finding Scheme API code in gEDA
  2. Compiling against multiple Guile versions
  3. Safe handling of non-local exits
  4. Dealing with deprecated libguile functions
  5. Checking arguments to Scheme functions in C
  6. How and when to use Scheme errors
  7. Reducing boilerplate with "snarfing macros"
  8. Opportunities to get involved

In this blog post, I'll explain where Guile is used in gEDA at the moment, both C code using libguile and Scheme code itself.

What is Scheme used for?

Currently, Guile is used for:

  • Executing "configuration" files (actually initialisation files) written in Scheme, in all libgeda applications.
  • Setting up the menus and keybindings in gschem. Key sequences and menu items call Scheme "thunks" (functions of no arguments), which are mostly implemented in C. It's not currently feasible to create a meaningful new action in gschem without writing it mostly in C.
  • Exporting netlists in gnetlist. A gnetlist backend generates particular on-disk netlist file format using a limited API that provides access to the compiled netlist.
  • A few other things (mostly incomplete). For example, there is a very limited existing Scheme API spread between gschem and libgeda for taking actions on changes to pages and/or attributes.

It would be nice to do more things (like extending gschem with additional actions written in pure Scheme, or being able to modify the way that gnetlist compiles a netlist from input schematics).

Where is the Guile code in gEDA?

In the C sources, files which contain code using libguile usually begin with the prefix "g_". For example, "g_rc.c" contains definitions of functions used for configuration parameters, and "g_register.c" contains code for registering C functions and variables to be visible from Scheme. There are a bunch of exceptions to be aware of, which include:

  • libgeda/src/s_clib.c contains some libguile code that permits component libraries to be defined as a set of Scheme functions.
  • libgeda/src/s_menu.c contains some basic infrastructure for menu definitions in Scheme rc files that's past due to move into gschem.
  • libgeda/src/s_menu.c has functions for converting gEDA colour map data to and from Scheme representations.
  • Several gschem source files contain snippets of libguile code for firing hooks when the user carries out certain actions.
  • gschem/src/x_menus.c implements menus defined in Scheme, and gschem/src/x_stroke.c provides support for assigning gestures to actions in Scheme (which capability I'm not sure I've ever used).

More generally, you can usually find code that uses libguile by searching for the "scm_" or "SCM_" prefix used by libguile functions, macros and variables.

Most of the application directories in the gEDA source tree have a "scheme" subdirectory where most of the ".scm" files with Scheme code live. There are also the system rc files for each application, which are also written in Scheme but for some reason live in "lib" subdirectories.

So that's a brief overview of where the Scheme-related code lives in gEDA. In my next post, I will talk about how to set up your build environment to be able to easily build against either Guile 1.8 or Guile 2.0.

Sunday, June 05, 2011

Porting gEDA to Guile 2.0, and future plans

Guile, a Scheme implementation that's part of the GNU project, recently put out a new 2.0 release with many exciting new features and improvements.

gEDA relies on Guile for extensibility. In the past we have had a few problems moving to new versions of Guile, often due to differences between memory management APIs or library functions.

I first started looking at Guile 2.0 support in gEDA in December 2010. At the time, I had great difficulty getting libgeda to work due to differences in the debugging APIs between Guile 1.8 and 2.0. However, during January I did a considerable amount of work on how Scheme errors are reported in libgeda, and as part of that I managed to do things in a way compatible with both versions of Guile.

Last Friday I was therefore able to get gEDA fully working with Guile 2.0. Actually, the main remaining obstacle was that the new version of Guile is much stricter about correct Scheme syntax, and some of the gnetlist backends needed fixing up.

But that's done, and gEDA is now fully compatible with Guile 2.0 and can take advantage of the performance improvements it brings!

What's next for me in terms of gEDA development? Well, I still have several different development branches on the go, and I'm not sure which of them (if any) to focus on.

  1. I have a branch which extensively refactors, extends and improves the libgeda scheme API, with the intention of making it possible to write much more fully-featured plugins & extensions for gschem and gnetlist.
  2. I have done a lot of work on splitting the cairo rendering code out of gschem into a separate library called libgedacairo, in order to make it easier to write applications that display gEDA schematics or convert them to other graphics formats. (As part of of this work, I'm also modifying gschem to take advantage of the GTK+ printing APIs).
  3. I have a design for a new configuration system for gEDA, which is non-executable and can provide per-schematic-page configuration. This is intended to remove a security flaw in gEDA at the moment when opening designs received from others.

Or are there other, more urgent things that need work? let me know what you think.

Sunday, August 09, 2009

gEDA build system redesign

One of the items on the gEDA "Todos" list as a prerequisite for the 1.6.0 release has been a refactor of the gEDA build system. Instead of using several distributed source archives (one for each major component), it was decided to distribute the gaf suite as a single archive, with one unified build system.

Why?

The advantages of this strategy were seen as follows:

  1. The system would be less confusing for new testers or "power" users who want to try out the latest changes in git.
  2. The whole suite could be built and tested before installing any files. In the past, it's been necessary to install the libgeda and symbols packages before any of the other packages could be built.
  3. It would reduce the labour involved in keeping the version numbers, ChangeLogs, and required dependency checks up-to-date. Maintaining multiple configure scripts in parallel is labour intensive, and there is the ever-present risk of missing one.
  4. It would simplify the job of distribution packagers. In the RPM case, it would only be necessary for a packager (like the maintainer of the Fedora Electronic Lab distribution, Chitlesh Goorah) to maintain a single SPEC file, rather than nine.
  5. It would simplify building using parallel make (see the -j option to make), which can significantly speed up compiling gEDA on machines with multiple or multi-core processors.
  6. It would improve detection of targets requiring rebuilding. This is particularly important for working on libgeda; only changes which affect the public interface require a rebuild of the applications, but it's often hard to keep track of what's changed. What often happens with the 1.4.x build system is that people end recompiling the applications completely just to make sure, at a cost of time and energy.

Approach

The rework had been on the "Todos" list for several months when I started looking into it at the beginning of June 2009 (just after handing in my MEng project), and some ideas came to mind about how the current structure of the build system could be improved.

  • One idea that crossed my mind extremely briefly was not to use GNU Autoconf and Automake, but to adopt another build infrastructure. Autotools was retained, because not only is it well known by current gEDA developers, but it has worked well enough for us in the past, and it doesn't require developers or users to install yet another tool in order to be able to compile and install gEDA.
  • I had recently been looking at the configure.ac file for DTN2 (the reference implementation for the Delay Tolerant Networking Bundling Protocol), and I noticed that they used a very minimalist, clearly-laid-out configure script will almost all of the configure checks being described in separate m4 files, as self-contained Autoconf macros. This appealed to me, because configure.ac files are often very confusing to a newcomer, and this method offered the potential to make it much, much clearer.

Problems encountered

There were two major problems encountered while carrying out the work.

Firstly, internationalisation of desktop integration files (.desktop files to appear in menus etc., and MIME-type info files to make file browsers show the right icon. In previous releases, intltool has been used to do this, but intltool suffers from a major flaw: it doesn't support more than one gettext domain in a single package, and we would need to use at least three (one each for libgeda, gschem and gattrib).

I was very surprised to find that there wasn't an existing solution to this problem in widespread use. In the end, I had to actually write my own tool from scratch to carry out this task, a 550-line shell script called desktop-i18n. Although enormously hacky, it works remarkably effectively!

The second major problem was in getting the testsuite to pass without gEDA being installed. The number of issues encountered was innumerable (one of my "favourite" examples was the way that libgeda used to call exit(-1) if it couldn't find the gEDA font, even in gnetlist which doesn't need to display any text objects anyway). By a combination of actually fixing bugs (few) and horrible nasty hacks (many), I eventually surmounted this issue too. With this, there were more than a few occasions where I felt like I was banging my head against a brick wall.

Results

On Friday 7th August I merged the "build-system" branch into the main git repository, and the new build system seems to be a genuine and successful enhancement (despite a few minor issues). However, I can hardly believe how much work it took to do!

Sunday, July 26, 2009

Simple unit tests in Scheme

One nice feature of the Scheme programming is the ridiculous, over-the-top power of its "hygienic" macro system, which allows a programmer to effectively redefine the language to suit the task at hand.

A few days ago, I needed to write some simple unit tests for some Scheme code I was in the process of developing. The Guile Scheme implementation doesn't come with a library for unit testing by default, so it was necessary to create one of my own. This is it:

(define *failed-tests* '())
(define *passed-tests* '())

(define (assert-true result)
  (if (not result)
      (throw 'test-failed-exception
             (with-output-to-string
              (lambda ()
                (display "  assert-true: ")
                (display "got: ")
                (write result))))))

(define (%begin-test name test-thunk)
  (let ((test-success #t)
        (test-fail-msg #f))
    (display name) (display "... ")

    (catch #t test-thunk
           (lambda (key . args)
             (set! test-success #f)
             (set! test-fail-msg
             (if (eqv? key 'test-failed-exception)
                 (car args)
                 (with-output-to-string
                  (lambda ()
                    (display "  unexpected exception: ")
                    (write (cons key args))))))))

    (if test-success
        (begin
          (display "passed")
          (set! *passed-tests* (cons name *passed-tests*)))
        (begin
          (display "failed")
          (if test-fail-msg
              (begin
                (newline)
                (display test-fail-msg)))
          (set! *failed-tests* (cons name *failed-tests*))))
    (newline)))

(define-syntax begin-test
    (syntax-rules ()
      ((_ name . test-forms)
       (%begin-test name (lambda () . test-forms)))))

(define (report-tests)
  (display "Test summary")(newline)
  (display "Passed: ") (display (length *passed-tests*)) (newline)
  (display "Failed: ") (display (length *failed-tests*)) (newline))

Note that this effectively adds a keyword (begin-test) to the Scheme language, and this very simple macro has a massive effect on the readability of the resulting testcase code. For example, a trivial unit test using this minimalistic framework might look like:

(begin-test 'SuccessfulTest
  (assert-true #t)
  (assert-true (equal? 1 1)))
(begin-test 'FailTest
  (assert-true (equal? #t "string")))
(report-tests)

The framework is completed with the addition a few more assert- functions (left as an exercise to the reader). This code will form part of a future release of gEDA.

(In case you want to use this code in your own project, it is licensed under the GNU LGPL v3 or later). But there are probably better unit test systems for Scheme out there!

Saturday, January 24, 2009

gschem, log files and X selections

Just over a week ago, I posted a series of [RFC] posts to the gEDA-user mailing list suggesting six reasonably non-trivial changes to libgeda and gschem. Although the discussions they started off weren't quite as productive as I was hoping, they did attract quite a bit of interest.

I haven't been afraid to put my money where my mouth is. Since then, I've implemented two of the changes I suggested: a modification of the way gEDA apps create logfiles, and support for copying and pasting bits of schematics between gschem windows. I thought I'd post here to explain the amount of careful design work that goes into even relatively minor changes to the gEDA suite.

Logging

Historically, gschem (and the other gEDA apps) have created logfiles in the current working directory when they were launched. This behaviour quickly resulted in a user's home directory becoming littered with the files, gschem.log files especially.

My initial suggestion was to just not create log files by default, and require users to specifically enable them. However, it was quickly pointed out that since one of the most common reasons for gschem failing to start was that a new user had made a mistake in their configuration file, and that it would be best to carry on with generating logfiles as the default behaviour.

Instead, it was suggested that the logfiles should be put in a centralised, out-of-the-way location, and ~/.gEDA/logs/ was suggested, with the caveat that it would be necessary to make sure that multiple copies of gschem accessing the directory would have to be supported safely, even on different machines, but also that a user could quickly easily find the most recent log file -- so just adding a random suffix with mktemp wouldn't work either!

Having worked out a way of doing this (a secure algorithm for generating sequentially numbered, dated log files), I then realised that on some operating systems, such as Microsoft Windows, $HOME might not even be defined, and that there might be platform-specific differences in the standardised place to keep configuration data. Since the gEDA applications had no existing way of handling these variations, I ended up abstracting out the methods for locating the system and user data and configuration files, and updating all of the gEDA apps to use them.

All in all, changing the location where log files are stored used up at least eight hours of development time.

Clipboard copy, cut and paste

For a long time, gschem has supported copying and pasting schematic data between pages and windows in the same gschem process, but it was never made clear to users that this mechanism didn't use the X clipboard, and thus sometimes copying and pasting didn't work when it would have made sense for it to.

X Window System selections are complicated. There are two selections normally used by X applications: the PRIMARY selection and the CLIPBOARD selection. Traditionally, the PRIMARY selection is contains the most recent piece of text selected by the user with the mouse, and can be pasted by clicking the middle mouse butten, while the CLIPBOARD selection is accessed via "copy" and "paste" actions explicitly invoked by the user. In order to match up with other applications (and because nothing like the PRIMARY selection is available on Microsoft Windows or OS X), we decided to use the CLIPBOARD.

For the same portability reasons (and also because the raw X API is horrendously complicated), we decided to use the GTK Clipboard API to manage the clipboard.

A neat feature of clipboards (and X selections in particular) is that it's possible for the source application to provide the data in different forms, depending on what the "pasting" application wants. We needed a format for transferring between gschem instances, so we decided to use the gEDA schematic format (application/x-geda-schematic). Fortunately, as part of my rewrite of the symbol library back in mid 2007, I'd made libgeda able to "load" and "save" schematics from memory buffers, so I simply hooked the clipboard code into that and things started working!

I then, of course, had to work through the boilerplate required to update the GUI to support it. This took as long as implementing the actual feature.

I've also been working on some other features, using some code that I've had floating around for a long time. But more of that another time!

Friday, December 07, 2007

Error-reporting progress

Further to my previous blog post on error-reporting in libgeda, I've made some pretty good progress on implementing the ideas that I described. In particular:

  • The file opening functions (f_open() and f_open_flags) now use GError to return error messages rather than just spewing them to the console.
  • gschem now shows an informative dialog box when it fails to load a file.
  • All programs now actually tell you the real reason for failing to load a file (e.g. "Permission denied" or "File does not exist") rather than just giving a generic "Could not open file" message.
  • The libgeda logging mechanism now handles most levels of GLib log message rather than just the "message" level (seeing as the log is most useful when some sort of error has occurred, this seems sensible).
  • gschem now highlights warnings and errors in the log window.
  • Lots of ways in which libgeda could randomly kill your program have been eradicated.

These changes are not yet in the unstable branch, but barring someone objecting to my changes I'll push them upstream tomorrow sometime.

The next (probably thankless) task is to work through libgeda converting all calls to directly output debugging/error-reporting information to the console to use the GLib logging functions instead, and to make log messages be sent at appropriate levels rather than all at the "message" level (this will allow filtering of log messages in the gschem log window, for instance).

I'm still of two minds about the best way to do this. Currently, libgeda defines a macro s_log_message() which is equivalent to GLib's g_message(). For adding logging at multiple levels, the two options are:

  1. Get rid of s_log_message entirely, and just use the GLib logging macros directly.
  2. Define s_log_debug, s_log_warning etc so that the exact implementation could be easily modified at a later date (e.g. for the purpose of making libgeda always log in the "libgeda" log domain).

I'm currently leaning towards the latter, but haven't as yet made up my mind about exactly which will be the best way to go.

Tuesday, November 27, 2007

Error reporting & handling in libgeda

Apparently some people do read my blog: Peter Clifton and I met up for a drink at Borders in town this evening, and he recommended that I write more often. So here's this evening's contribution, on a slight refactor of libgeda error reporting.

It's been a while since I last blogged about gEDA development, and since then I've had the good fortune to be able to travel to Cambridge, MA and meet several of the other developers based around there. In the last couple of days I've started ramping up my involvement in preparation for having some time for serious development over the Christmas vacation.

So, to begin: in general, libgeda kills the process without warning far too often. This is a bad thing for a library to do; in particular, it's very bad to do it just because someone passed some bad data in. The libgeda doesn't make any attempt to hide how the data structures are organised, and thus we have to assume that user code will make use of this knowledge to try and hack them directly.

My pondering of this problem over the last couple of days has lead me to think up four basic rules to consider when working out how to handle errors occuring in libgeda code:

  1. If possible, succeed.
  2. If failure is inevitable, fail gracefully.
  3. If normal operation may result in failure, use GError.
  4. Assume that libgeda works.

Of course, some of these need a little explanation as to their interpretation and why they make sense.

Firstly, "If possible, succeed." This seems obvious, but actually has some subtlety. What I mean here is that if there's a sensible, clean way of carrying on despite a problem, you should do so. This only really applies to user-facing code, as code which can't be called from outside libgeda really should have had its inputs checked by the calling function already. However, since all of libgeda's code is user-facing -- we don't have any private headers or the like -- this point is rather moot. In addition, "succeeding" doesn't preclude printing messages to the effect of, "Someone's playing silly buggers but I'm going to try my best anyway." g_critical() should be used for this. One example would be when an unknown object type is encountered; at the moment, libgeda often kills the program by calling g_assert_not_reached(), when often it would be valid to continue by logging a critical message and then skipping over the offending object.

Secondly, "If failure is inevitable, fail gracefully." Failing gracefully requires no dangerous side-effects from the failure. If possible, the system should be returned to its prior state. This often requires that user data needs to be checked before doing anything destructive, possibly at the expense of some CPU time.

GError is a nice mechanism in that it allows errors to be ignored if necessary. A good example of when it should be used would be in code which reads and writes files; because GLib's file access code already uses GError extensively, this would not be hard to implement.

Finally, "Assume that libgeda works." Similarly to rule #1, what is meant here is that libgeda functions should check their own behaviour. If they do so, there is no need for libgeda functions which call them to check again -- it's a waste of CPU time and developer effort.

So, given the above points, when is it appropriate to use g_assert()? It would not be appropriate to use it to check the arguments passed to a function, but it would be valid to use it to check that the function has successfully done what it is supposed to before returning a result. For instance, when a complex algorithm is in use, putting some assertions in to make sure that the algorithm actually does what you think it does might be a very good idea.

Of course, this has lead to a number of action items in terms of libgeda refactoring:

  • Cleaning up uses of g_assert() in libgeda, and replacing them with g_critical() logging and graceful failure where possible. I've already made a start on this.
  • Moving all of the file handling code to use GError. This will allow gschem to show a message dialog when a file operation fails, rather than forcing the user to look at the log window or at the console.
  • In the medium term, moving all of libgeda to use the GLib logging API (of which g_critical() is part) will allow the development of a shiny new gschem log window which allows logging at different levels with colour coding of criticality. This would be nice.
  • Ideally, the error messages used for GErrors emitted from libgeda should be translated. This means libgeda will need to use gettext.

As usual, I'd be interested to hear peoples' thoughts on this. Please let me know on the -dev list or by e-mail.

Saturday, June 30, 2007

Making embedding more useful

One of the trickiest problems electronic engineers have is with keeping their symbol libraries up-to-date while not breaking old schematics (e.g. with changing pin positions).

Currently, most people use one of two approaches to managing the problem:

  1. gEDA has the ability to embed symbol data in the schematic file, making the schematic independent of changing symbol libraries, and some people choose to embed all their symbols. However, a full copy of the symbol is embedded for each time it's instantiated in the schematic, making the file size bloat enormously. In addition, not only is neither possible to check whether any of the symbols have updated versions available in the library nor to update all symbols of a given type together, it is also impossible to make sure that all symbols of the same type are embedded from the same version of that symbol.
  2. gEDA also has the ability to use a project-specific configuration file which lives in the same directory as the schematics. This can be used to implement a project-specific component library where all symbols used by the project live. Although this solves the problem of keeping all symbols in the project in sync with each other, it is quite high maintenance to keep the project library in sync with a "master" library. It is also still impossible to automatically check for newer versions and update them.

My solution to this would be to add the missing features to the embedding functionality (presupposing that these embedding bugs [1692626] are also fixed).

By default, each symbol used would automatically have a single copy embedded in the schematic, which instances of that symbol would reference. This would mean that all instances of a given symbol would be upgraded in lockstep, eliminating inconsistencies.

A hash algorithm would be used to compare the embedded version of a symbol with the latest library version. If they differ, an icon would be shown in a new component manager (derived from the current component selector) and users would be able to update symbols from there, either all symbols at once or one symbol at a time.

I have yet to work out whether it should be possible to edit symbols which do not exist in the local library, but it should certainly be possible to open them for read-only viewing or export them to discrete symbol files.

I believe this solution would eliminate most of the current deficiencies of gEDA and gschem for keeping archived schematics safe and keeping current schematics in sync with libraries.

Friday, June 29, 2007

Hacking in the unstable branch

For most of June, gEDA has been in the process of switching from using CVS to git for version control, and quite a lot of developers' time seems to have been spent on getting used to the new system. I really like the new system; it makes reviewing changes and maintaining patchsets so much less of a chore, and my productivity has gone through the roof!

I've made really good progress on the component library and Scheme work which I said that I had planned to do during June:

  1. The component selector preview widget now works properly for all possible component library backends. My changes coincidentally fix a security hole exploitable by malicious library distributors (they could use the library directory RC file mechanism to execute arbitrary code on your system).
  2. system-gafrc is now installed by libgeda, and loads separate Scheme files for the default libraries and default font. Hopefully, this should make it easier to avoid using the default symbol library at all if, for some reason, you don't want it.
  3. I zapped a bunch of deprecated Guile functions, and changed the configure scripts to check for Guile by version, but it's pretty boring stuff and I don't really have any motivation to finish it of at the moment.
  4. Replacing the directory and command component library backends with equivalent Scheme implementations has had to be suspended, due to the fact that the Guile functions for working with pipes really don't work very well on Windows, and Windows support looks like it's going to become more important over the next few months.
  5. I've replaced the s_clib_glob() with a similar function, s_clib_search(), which can operate either in exact matching mode or glob matching mode. This will allow people to use symbols with glob special characters without strange breakage. I've also added a caching mechanism for the results of the search. I went through all the places where pointers to CLibSource or CLibSymbol structures might be cached, and tried to make sure that they weren't; this was to make it possible to add, remove and refresh component sources without causing breakage.

I have introduced a patch to my personal tree which adds a list of currently used symbols to the component selector. I find this really useful already for speeding up adding frequently-used parts to my schematics, such as resistors and capacitors. I'm also hoping to add a button to the component selector dialog which allows you to directly open a selected symbol for editing.

My recent work (and discussions on the mailing lists) have inspired two more ideas, about which I'll go into more detail on a future occasion:

  • Per-directory configuration contexts.
  • Overhaul of the embedding system to make it more useful.