Tuesday, September 29, 2015

Using C library functions from LiveCode Builder

This blog post is part of an ongoing series about writing LiveCode Builder applications without the LiveCode engine.

Currently, the LiveCode Builder (LCB) standard library is fairly minimal. This means that there are some types of task for which you'll want to go beyond the standard library.

In a previous post, I described how to use LiveCode's foundation library. This lets you access plenty of built-in LiveCode functionality that isn't directly exposed to LCB code yet.

Someone else's problem

Often someone's already wrapped the functions that you need in another program, especially on Linux. You can run that program as a subprocess to access it. In LiveCode Script, you could use the shell function to run an external program. Unfortunately, the LCB standard library doesn't have an equivalent feature yet!

On the other hand, the standard C library's system(3) function can be used to run a shell command. Its prototype is:

int system(const char *command);

In this post, I'll describe how LCB's foreign function interface lets you call it.

Declaring a foreign handler

As last time, you can use the foreign handler syntax to declare the C library function. The com.livecode.foreign provides some important C types.

use com.livecode.foreign

foreign handler _system(in pCommand as ZStringNative) \
      returns CInt binds to "system"

Some things to bear in mind here:

  • I've named the foreign handler _system because the all-lowercase identifier system is reserved for syntax tokens
  • The ZStringNative type automatically converts a LCB string into a null-terminated string into whatever encoding LiveCode thinks is the system's "native" encoding.
  • Because the C library is always linked into the LiveCode program when it's started, you don't need to specify a library name in the binds to clause; you can just use the name of the system(3) function.

Understanding the results

So, now you've declared the foreign handler, that's it! You can now just _system("rm -rf /opt/runrev") (or some other helpful operation). Right?

Well, not quite. If you want to know whether the shell command succeeded, you'll need to interpret the return value of the _system handler, and unfortunately, this isn't just the exit status of the command. From the system(3) man page:

The value returned is -1 on error (e.g., fork(2) failed), and the return status of the command otherwise. This latter return status is in the format specified in wait(2). Thus, the exit code of the command will be WEXITSTATUS(status). In case /bin/sh could not be executed, the exit status will be that of a command that does exit(127).

So if the _system handler returns -1, then an error occurred. Otherwise, it's necessary to do something equivalent to the WIFEXITED C macro to check if the command ran normally. If it didn't, then some sort of abnormal condition occurred in the command (e.g. it was killed). Finally, the actual exit status is extracted by doing something equivalent to the WEXITSTATUS C macro.

On Linux, these two macros are defined as follows:

#define WIFEXITED(status)     __WIFEXITED (__WAIT_INT (status))
#define WEXITSTATUS(status)   __WEXITSTATUS (__WAIT_INT (status))
#define __WIFEXITED(status)   (__WTERMSIG(status) == 0)
#define __WEXITSTATUS(status) (((status) & 0xff00) >> 8)
#define __WTERMSIG(status)    ((status) & 0x7f)
#define __WAIT_INT(status)    (status)

Or, more succinctly:

#define WIFEXITED(status)   (((status) & 0x7f) == 0)
#define WEXITSTATUS(status) (((status) & 0xff00) >> 8)

This is enough to be able to fully define a function that runs a shell command and returns its exit status.

module org.example.system

use com.livecode.foreign

private foreign handler _system(in pCommand as ZStringNative) \
      returns CInt binds to "system"

Run the shell command  and wait for it to finish.
Returns the exit status of if the command completed, and nothing
if an error occurred or the command exited abnormally.
public handler System(in pCommand as String) \
      returns optional Number

   variable tStatus as Number
   put _system(pCommand) into tStatus

   -- Check for error
   if tStatus is -1 then
      return nothing
   end if

   -- Check for abnormal exit
   if (127 bitwise and tStatus) is not 0 then
      return nothing
   end if

   -- Return exit status
   return 255 bitwise and (tStatus shifted right by 8 bitwise)

end module

Tip of the iceberg

This post has hopefully demonstrated the potential of LiveCode Builder's FFI. Even if you use only the C standard library's functions, you gain access to almost everything that the operating system is capable of!

Using a C function from LCB involves reading the manual pages to find out how the function should be used, and how best to map its arguments and return values onto LCB types; often, reading C library header files to understand how particular values should be encoded or decoded; and finally, binding the library function and providing a wrapper that makes it comfortable use from LCB programs.

LiveCode Builder can do a lot more than just making widgets and — as I hope I've demonstrated — can do useful things without the rest of the LiveCode engine. Download LiveCode 8 and try some things out!


Monte said...

Thanks Peter, looking forward to more of these posts.

I've had a little play with libyaml to see if I could work out how to include a library in the extension resources but it failed miserably: http://forums.livecode.com/viewtopic.php?f=93&t=25317&sid=ac3eb978326ac14d05098ef648a035c6

I tried a few different options for the bind string which I thought was meant to be libraryName>functionName but as you just used functionName I also just tried that but still no dice.

Also interested in anything you have to say about working with structs.

Peter Brett said...

Hi Monte, I'm glad you're still enjoying these posts! You'll be pleased to hear that I already have some more planned. At the moment unfortunately there's no way to include external native code files in LCB package resources, sorry. I started work on it a few months ago, but then build automation happened, and then HTML5, etc. etc.

Your best bet is to install the library in the system's dynamic linker's search path and then bind to it using "libraryName>functionName".

As far as structs are concerned, at the moment you'll need to pack them manually. In my not-so-secret extremely-broken "undergrowth" library there's an example of working with "struct pollfd. This may or may not be helpful.

Monte said...

Hmm... what's this then

I'll check out the undergrowth ;-)

Peter Brett said...

Hmm, I must have been on holiday when that got merged. Or insert other suitable excuse here. ;-)

Monte said...

I can't get it to work anyway so possibly no need for excuses ;-)

Unknown said...

I can't get this to work as well, even after copying exactly what you had posted here and trying to run it on LC 8 DP 7. Any ideas on how to run "Shell" commands in LCB?

Peter Brett said...

Hi Stephen, Monte was talking about the "embedding native code libraries in LCB extensions" that he couldn't get working, not the stuff in this blog post. What problems did you run into?

Unknown said...

I had copied the code snippet as you had it and used it like the following in my LC script;
on mouseUp
put "@powershell -NoProfile -ExecutionPolicy Bypass -Command" && "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" into tChocoInstall
//put the result
put it
end mouseUp

but all i get is an error warning like so;
button "Button": execution error at line 3 (Handler: can't find handler) near "System", char 1

I did not change any parts of your code in your posting, so wonder what i am doing wrong?

I was trying this out on LC 8 DP7 on a Windows 7 (64bit) machine.

I am still trying to wrap my head around how the LCB can hook into C functions. I have read your posts over a couple of times and gone over the man pages like you had listed, but i guess i am facing a block and your clarifications will definitely point me in the right direction.


Peter Brett said...

Hi Stephen, these blog posts have been about writing LiveCode Builder-only programs -- not using the IDE or the "normal" LiveCode engine at all. If you were putting the code into a stack I'm surprised you got as far as you did!

If you created and compiled the module as an extension, and you want to use the Syntax() function from LiveCode Script, then you need to make sure that you use the library keyword instead of module, i.e.:

library org.example.system

use com.livecode.foreign


end library

Then the public handlers that you define will be put into the LCS message path.

Unknown said...

Thanks Peter for your reply, i believe further clarifications as to what i am trying to achieve and where i am so far will enable you point me in the right direction.

I am writting an apt-get for LC in LCB and using web based Chocolatey API service . I have written wrappers around the "apt-get" (for windows only) in pure LC scripts, but it was a cumbersome work and now properly finessed, so in my goal to learn LCB, i am currently trying to re-write the apt-get in LCB.

From your response, it is obvious i am mixing up LCB syntaxes, but i have followed all the examples and even reviewed LC libraries written in LCB, to arrive at where i am today. I have noticed the major issue/stumbling block in my path is calling/executing Shell command in LCB, hence my resort to your post which i thought was addressing this from a library point of perpective, since this isn't the case, can you please point me in the right direction on;

(1) How to run Shell in LCB as a library?
(2) How to wrap REST in LCB as a library?
(3) How to call/start using LCB library in an LC script?
(4) How to wrap OS features(windows in this case) in LCB?

Providing some guidance on the above topic will greatly help. I do hold the second type of the more expensive LC Bbusiness License :), but i find that your support and response are far easier for my understanding than the hollow support email i get from you guys as part of my business license. So any pointers in the right directions is greatly appreciated.
Thank you

Peter Brett said...

Hi Stephen, please drop an e-mail to LiveCode support and I'm sure they'll be able to help you out with your queries. The questions are a bit off-topic from the subject of this blog post, and also our support team have a better breadth of knowledge to be able to answer them properly.