Tuesday, September 29, 2015

Using C library functions from LiveCode Builder

This blog post is part of an ongoing series about writing LiveCode Builder applications without the LiveCode engine.

Currently, the LiveCode Builder (LCB) standard library is fairly minimal. This means that there are some types of task for which you'll want to go beyond the standard library.

In a previous post, I described how to use LiveCode's foundation library. This lets you access plenty of built-in LiveCode functionality that isn't directly exposed to LCB code yet.

Someone else's problem

Often someone's already wrapped the functions that you need in another program, especially on Linux. You can run that program as a subprocess to access it. In LiveCode Script, you could use the shell function to run an external program. Unfortunately, the LCB standard library doesn't have an equivalent feature yet!

On the other hand, the standard C library's system(3) function can be used to run a shell command. Its prototype is:

int system(const char *command);

In this post, I'll describe how LCB's foreign function interface lets you call it.

Declaring a foreign handler

As last time, you can use the foreign handler syntax to declare the C library function. The com.livecode.foreign provides some important C types.

use com.livecode.foreign

foreign handler _system(in pCommand as ZStringNative) \
      returns CInt binds to "system"

Some things to bear in mind here:

  • I've named the foreign handler _system because the all-lowercase identifier system is reserved for syntax tokens
  • The ZStringNative type automatically converts a LCB string into a null-terminated string into whatever encoding LiveCode thinks is the system's "native" encoding.
  • Because the C library is always linked into the LiveCode program when it's started, you don't need to specify a library name in the binds to clause; you can just use the name of the system(3) function.

Understanding the results

So, now you've declared the foreign handler, that's it! You can now just _system("rm -rf /opt/runrev") (or some other helpful operation). Right?

Well, not quite. If you want to know whether the shell command succeeded, you'll need to interpret the return value of the _system handler, and unfortunately, this isn't just the exit status of the command. From the system(3) man page:

The value returned is -1 on error (e.g., fork(2) failed), and the return status of the command otherwise. This latter return status is in the format specified in wait(2). Thus, the exit code of the command will be WEXITSTATUS(status). In case /bin/sh could not be executed, the exit status will be that of a command that does exit(127).

So if the _system handler returns -1, then an error occurred. Otherwise, it's necessary to do something equivalent to the WIFEXITED C macro to check if the command ran normally. If it didn't, then some sort of abnormal condition occurred in the command (e.g. it was killed). Finally, the actual exit status is extracted by doing something equivalent to the WEXITSTATUS C macro.

On Linux, these two macros are defined as follows:

#define WIFEXITED(status)     __WIFEXITED (__WAIT_INT (status))
#define WEXITSTATUS(status)   __WEXITSTATUS (__WAIT_INT (status))
#define __WIFEXITED(status)   (__WTERMSIG(status) == 0)
#define __WEXITSTATUS(status) (((status) & 0xff00) >> 8)
#define __WTERMSIG(status)    ((status) & 0x7f)
#define __WAIT_INT(status)    (status)

Or, more succinctly:

#define WIFEXITED(status)   (((status) & 0x7f) == 0)
#define WEXITSTATUS(status) (((status) & 0xff00) >> 8)

This is enough to be able to fully define a function that runs a shell command and returns its exit status.

module org.example.system

use com.livecode.foreign

private foreign handler _system(in pCommand as ZStringNative) \
      returns CInt binds to "system"

/*
Run the shell command  and wait for it to finish.
Returns the exit status of if the command completed, and nothing
if an error occurred or the command exited abnormally.
*/
public handler System(in pCommand as String) \
      returns optional Number

   variable tStatus as Number
   put _system(pCommand) into tStatus

   -- Check for error
   if tStatus is -1 then
      return nothing
   end if

   -- Check for abnormal exit
   if (127 bitwise and tStatus) is not 0 then
      return nothing
   end if

   -- Return exit status
   return 255 bitwise and (tStatus shifted right by 8 bitwise)

end module

Tip of the iceberg

This post has hopefully demonstrated the potential of LiveCode Builder's FFI. Even if you use only the C standard library's functions, you gain access to almost everything that the operating system is capable of!

Using a C function from LCB involves reading the manual pages to find out how the function should be used, and how best to map its arguments and return values onto LCB types; often, reading C library header files to understand how particular values should be encoded or decoded; and finally, binding the library function and providing a wrapper that makes it comfortable use from LCB programs.

LiveCode Builder can do a lot more than just making widgets and — as I hope I've demonstrated — can do useful things without the rest of the LiveCode engine. Download LiveCode 8 and try some things out!

Wednesday, September 23, 2015

Roasted vegetable and chickpea tagine

It's been a while since I last posted a recipe here! Recently I've been having quite a lot of success with this Morrocan-inspired vegetarian recipe.

This recipe makes 6 portions.

Ingredients

For the roasted vegetables:

  • 350 g new potatoes, halved
  • 1 fennel bulb, trimmed & cut into batons
  • 1 medium carrot, cut into chunks
  • 1 large red pepper, cut into chunks
  • 1 large red onion, cut into chunks
  • 3 tbsp exra-virgin olive oil
  • 1 tsp cumin seeds
  • 1 tsp fennel seeds
  • 1 tsp coriander seeds, crushed

For the sauce:

  • 4 garlic cloves, chopped
  • 400 g canned chopped tomatoes
  • 400 g canned chickpeas, drained and rinsed
  • 250 ml red wine
  • 1 pickled lemon, finely chopped
  • 0.5 tbsp harissa paste
  • 1 tsp ras el hanout
  • 1 cinnamon stick
  • 40 g whole almonds
  • 10 dried apricots, halved

To serve:

  • Greek-style yoghurt
  • 2 tbsp coriander, finely chopped

Method

Preheat the oven to 200 °C fan. Put all the ingredients for the roasted vegetables into a large, heavy roasting tin, season to taste, and toss together to coat the vegetables in oil and spices. Roast for 30 minutes until the potatoes are cooked through and the vegetables generally have a nice roasted tinge.

While the vegetables are roasting, heat a large pan over a medium heat. Fry the garlic for 20–30 seconds until fragrant. Add the remaining ingredients, bring to the boil, and simmer while the vegetables roast.

When the vegetables are roasted, add them to the sauce and stir. Return the sauce to the simmer for another 15–20 minutes.

Serve in bowls, topped with a dollop of yoghurt and some chopped coriander. Couscous makes a good accompaniment to this dish if you want to make it go further.

Monday, September 14, 2015

Compiling multi-module LiveCode Builder programs

This blog post is part of an ongoing series about writing LiveCode Builder applications without the LiveCode engine.

Multi-module programs

When writing a large program, it's often useful to break it down into more than one module. For example, you might want to make a module that's dedicated to loading and saving the program's data, which has quite a lot of internal complexity but exposes a very simple API with Load() and Save() handlers. This is handy for making sure that it's easy to find the source file where each piece of functionality is located.

However, it can become tricky to compile the program. Each module may depend on any number of other modules, and you have to compile them in the correct order or the compilation result may be incorrect. Also, if one module changes, you have to recompile all of the modules that depend on it. If you tried to do this all by hand, it would be nigh-on impossible to correctly compile your program once you got above about 10 source files.

Fortunately, there are two really useful tools that can make it all rather easy. GNU Make (the make command) can perform all the required build steps in the correct order (and even in parallel!). And to help you avoid writing Makefiles by hand, lc-compile has a useful --deps mode.

Most of the remainder of this blog post will assume some familiarity with make and common Unix command-line tools.

The --deps option for lc-compile

make lets you express dependencies between files. However, you already express the dependencies between LCB source files when you write a use declaration. For example:

use com.livecode.foreign

says that your module depends on the .lci (LiveCode Interface) file for the com.livecode.foreign module.

So, the LCB compiler (a) already knows all the dependencies between the source files of your project and (b) already knows how to find the files. To take advantage of this and to massively simplify the process of creating a Makefile for a LCB project, lc-compile provides a --deps mode. In --deps mode, lc-compile doesn't do any of the normal compilation steps; instead, it outputs a set of Make rules on standard output.

Consider the following trivial two-file program.

-- org.example.numargs.lcb

module org.example.numargs

public handler NumArgs()
   return the number of elements in the command arguments
end handler

end module
-- org.example.countargs.lcb

module org.example.countargs

use org.example.numargs

public handler Main()
   quit with status NumArgs()
end handler

end module

To generate the dependency rules, you run lc-compile with almost a normal command line — but you specify --deps make instead of an --output argument, and you list all of your source files instead of just one of them. See also my previous blog post about compiling and running pure LCB programs. For the "countargs" example program you could run:

$TOOLCHAIN/lc-compile --modulepath . --modulepath $TOOLCHAIN/modules/lci --deps make org.example.numargs.lcb org.example.countargs.lcb

This would print the following rules:

org.example.countargs.lci: org.example.numargs.lci org.example.countargs.lcb
org.example.numargs.lci: org.example.numargs.lcb

Integrating with make

You can integrate this info into a Makefile quite easily. There are two pieces that you need: 1) tell make to load the extra rules, and 2) tell make how to generate them. In particular, it's important to regenerate the rules whenever the Makefile itself is modified (e.g. to add an additional source file).

# List of source code files
SOURCES = org.example.countargs.lcb org.example.numargs.lcb

# Include all the generated dependency rules
include deps.mk

# Rules for regenerating dependency rules whenever
# the source code changes
deps.mk: $(SOURCES) Makefile
 $(TOOLCHAIN)/lc-compile --modulepath . --modulepath $(TOOLCHAIN)/modules/lci --deps make -- $(SOURCES) > $@

A complete Makefile

Putting this all together, I've created a complete Makefile for the example multi-file project. It has the usual make compile and make clean targets, and places all of the built artefacts in a subdirectory called _build.

################################################################
# Parameters

# Tools etc.
LC_SRC_DIR ?= ../livecode
LC_BUILD_DIR ?= $(LC_SRC_DIR)/build-linux-x86_64/livecode/out/Debug
LC_LCI_DIR = $(LC_BUILD_DIR)/modules/lci
LC_COMPILE ?= $(LC_BUILD_DIR)/lc-compile
LC_RUN ?= $(LC_BUILD_DIR)/lc-run

BUILDDIR = _build

LC_COMPILE_FLAGS += --modulepath $(BUILDDIR) --modulepath $(LC_LCI_DIR)

# List of source code files.
SOURCES = org.example.countargs.lcb org.example.numargs.lcb

# List of compiled module filenames.
MODULES = $(patsubst %.lcb,$(BUILDDIR)/%.lcm,$(SOURCES))

################################################################
# Top-level targets
all: compile

compile: $(MODULES)

clean:
 -rm -rf $(BUILDDIR)

.DEFAULT: all
.PHONY: all compile clean

################################################################
# Build dependencies rules
include $(BUILDDIR)/deps.mk

$(BUILDDIR):
 mkdir -p $(BUILDDIR)

$(BUILDDIR)/deps.mk: $(SOURCES) Makefile | $(BUILDDIR)
 $(LC_COMPILE) $(LC_COMPILE_FLAGS) --deps make -- $(SOURCES) > $@

################################################################
# Build rules
$(BUILDDIR)/%.lcm $(BUILDDIR)/%.lci: %.lcb | $(BUILDDIR)
 $(LC_COMPILE) $(LC_COMPILE_FLAGS) --output $@ -- $<

You should be able to use this directly in your own projects. All you need to do is to modify the list of source files in the SOURCES variable!

Note that you need to name your source files exactly the same as the corresponding interface files in order for this Makefile to work correctly. I'll leave adapting to the case where the source file and interface file are named differently as an exercise to the reader…

I hope you find this useful as a basis for writing new LiveCode Builder projects! Let me know how you get on.

Sunday, September 06, 2015

Accessing the Foundation library with LiveCode Builder

This blog post is part of an ongoing series about writing LiveCode Builder applications without the LiveCode engine.

The LiveCode Foundation library

LiveCode includes a "foundation" library (called, unsurprisingly, libfoundation) which provides a lot of useful functions that work on all the platforms that LiveCode supports. This is used to make sure that LiveCode works in the same way no matter which operating system or processor you're using. libfoundation is compiled into both the LiveCode engine and LiveCode Builder's lc-run tool, so it's always available.

libfoundation is written in C and C++. The functions available in the library are declared in the foundation.h header file.

Among other capabilities, libfoundation handles encoding and decoding text. This provides an opportunity to fix one of the problems with the "hello world" program I described in a previous post.

Foreign function access to libfoundation

The "hello world" program read in a file and wrote it out to the standard output stream. Unlike "hello world" programs seen elsewhere, it *didn't* write out a string, e.g.:

write "Hello World!" to the output stream

This doesn't work because write needs to receive Data, and converting a String to Data requires encoding (using a suitable string encoding). And unfortunately, the LiveCode Builder library doesn't supply any text encoding/decoding syntax, although I'm working on it.

However, and fortunately for this blog post, libfoundation supplies a suitable function, MCStringEncode. Its C++ declaration looks like:

bool MCStringEncode(MCStringRef string, MCStringEncoding encoding, bool is_external_rep, MCDataRef& r_data);

You can use it in a LiveCode Builder program by declaring it as a foreign handler. The com.livecode.foreign module provides some helpful declarations for C and C++ types.

use com.livecode.foreign

foreign handler MCStringEncode(in Source as String, \
      in Encoding as CInt, in IsExternalRep as CBool, \
      out Encoded as Data) returns CBool binds to "<builtin>"

CInt and CBool are C & C++'s int and bool types, respectively.

Encoding a string with UTF-8

Next, you can write a LiveCode Builder handler that encodes a string using UTF-8 (an 8-bit Unicode encoding). Almost every operating system will Do The Right Thing if you write UTF-8 encoded text to standard output; the only ones that might complain are some versions of Windows and some weirdly-configured Linux systems.

handler EncodeUTF8(in pString as String) returns Data
   variable tEncoded as Data
   MCStringEncode(pString, 4 /* UTF-8 */, false, tEncoded)
   return tEncoded
end handler

The "4" in there is a magic number that comes from libfoundation's kMCStringEncodingUTF8 constant. Also, you should always pass false to the IsExternalRep argument (for historical reasons).

A better "hello world" program

Putting this all together, you can now write an improved "hello world" program that doesn't get its text from an external file.

module org.example.helloworld2

use com.livecode.foreign

foreign handler MCStringEncode(in Source as String, \
      in Encoding as CInt, in IsExternalRep as CBool, \
      out Encoded as Data) returns CBool binds to "<builtin>"

handler EncodeUTF8(in pString as String) returns Data
   variable tEncoded as Data
   MCStringEncode(pString, 4 /* UTF-8 */, false, tEncoded)
   return tEncoded
end handler

public handler Main()
   write EncodeUTF8("Hello World!\n") to the output stream
end handler

end module

If you compile and run this program, you'll now get the same "Hello World!" message -- but this time, it's taking some text, turning it into Data by encoding it, and writing it out, rather than just regurgitating some previously-encoded data.

Other neat stuff

There's other cool (and, often, terribly unsafe) stuff you can do with direct access to libfoundation functions, like allocate Pointers to new memory buffers and directly manipulate LiveCode types & values. However, most of libfoundation's capabilities are already available using normal LiveCode Builder syntax.

The real power of foreign handler declarations becomes apparent when accessing functions that aren't in libfoundation — and this may be the subject of a future blog post!