0.6.2: LibRed and Macros

It is with great pleasure that we announce the 0.6.2 release of the Red programming language and toolchain. This release is the second heaviest one we have ever made (after the 0.6.0), weighing about 1200 commits! It was intended initially to be a minor one, but the needs for preprocessing support for Red runtime code arose, so it was the right time to make a first iteration into the land of macros. Moreover, the work on libRedRT (described below) gave us the opportunity to fulfill one of the goals we had for Red: become embeddable. This is now a reality, thanks to libRed. So the main features of this release are:

• Macros and preprocessor support
• Fast compilation using libRedRT
• LibRed for embedding Red anywhere

Macros

The Red macros and preprocessing abilities have been the topic of a previous article, so I invite you to discover them (and hopefully enjoy!) if you have not yet done so.

In addition to this article, it is worth mentioning that a feature called pre-load has been implemented since then (you can read about it down below), which can be used as an entry point for user-provided reader macros, and allow easier support for non-standard syntax. This is a minor feature, as Rebol-like languages already come strongly equipped for string parsing and processing, thanks to the Parse dialect.


Development and release modes for compilation

The biggest change introduced by this new version is the splitting of the compilation process in two modes, which are controlled by different command-line options:

• development: (-c) only user code is compiled (new mode).
• release: (-r) both user code and Red runtime library are compiled together.

When using -c, the Red runtime library is pre-compiled to a dynamic library called libRedRT ("RT" for RunTime), stored in the hidden red/ folder. When libRedRT is already present, only user code is compiled resulting in drastically shorter compilation times.

When -r option is used, the toolchain compiles user code in release mode, compiling with it all dependencies (including Red runtime library).

Here are a couple of simple benchmarks to show the (huge) difference in performance, using hello.red and view-test.red scripts compiled in both modes:

Another benchmark is compiling the Red tests suite (~18000 tests), unified way combines the tests into the minimum number of compilation units, while split way compiles each test file separately:

Measured on a Corei7 4Ghz Win7/64-bit machine.

Such performance results in significant daily productivity gains, both for the Red team, and Red users. It was worth the time and effort it took to properly convert the runtime library into a shared library. Though, the full support for modular compilation will come in 0.8.0, which will result in drastic cuts for the compilation time in release mode too.

Such fast compilation mode also works for Red scripts that embed Red/System code. Two cases are possible:

1. Red/System code does not contain any call to Red runtime library.
2. Red/System code contains one or several calls to Red runtime library.

In the first case, nothing needs to be done, the usage rules described above apply.

In the second case, a custom version of libRedRT is required, but the toolchain will take care of the process, it just requires the user to compile once using -u option, then simply use -c for next compilations. If new functions from the runtime library are called, then a new custom library needs to be rebuilt.

When compiling a custom run-time library, cleaning libRedRT files is needed to get rid of any outdated versions. The toolchain provides a red clear command which will remove all current libRedRT related files. Note that when upgrading the red binary to a newer version, it will automatically upgrade libRedRT on first invocation.

Any Red user code can be compiled in development mode, with the exception of objects with multiple inheritance, which is not supported by libRedRT (so can only be compiled in release mode).


Embedding Red using libRed

LibRed is the embeddable version of the Red interpreter + runtime library. It includes Parse dialect, reactive programming and our GUI system (View engine, VID dialect, Draw dialect). It exposes an interface to the host language through an API, which is C-compatible, so that any language that can embed C shared libraries, can also embed libRed.

The libRed API has been designed to be as simple and as straightforward as possible. See for yourself in the API documentation. The role of that API is to provide the required hooks for interfacing the Red runtime and interpreter with the host language. A high-level binding can eventually be built on top of the libRed C-level API, that will best map Red features, to the host language features.

A libRed HelloWorld in C:

    #include "red.h"

    int main() {
        redOpen();
        redPrint(redString("Hello World!"));
        redClose();
        return 0;
    }

A libRed graphic HelloWorld in C:

    #include "red.h"

    int main() {
        redOpen();
        redDo("view [text {Hello World!}]");
        redClose();
        return 0;
    }

In addition to the C-level API, we provide also a binding for VisualBasic for Applications, which can be used to embed Red into Microsoft Office applications!

Here is a demo showing side-by-side an Excel/VB window and a Red window playing Pong game:

All the code for this demo fits in a single page of VB code! You can get the required files from here. In that same Excel file, you will find two other simple examples of integration of Red with Excel sheets, which look like this:

Building libRed is straightforward:

    red build libred

That will build libRed with the cdecl ABI, suitable for integration with any C-compatible host. For VBA and MS Office, the stdcall ABI is required, which is achieved using:

    red build libred stdcall

The building process will also result in creating a libRed/ directory locally, expanding some extra files required for linking libRed properly.

Currently, full bindings are provided for C and VisualBasic. Experimental bindings are also available for Rust, Adobe AIR and C#.

This is a first iteration for libRed, we have plans for improving it, and for providing proper multi-instances support (currently limited at one instance per process).


Changes in 0.6.2

Core language

• New datatypes: tag!, email!
• New action: to
• New natives: as, call
• New functions: tag?, email?, hex-to-rgb, sqrt, class-of, face?, distance?, expand-directives, to-*, rejoin
• Adds integers auto-promotion to floats on loading and on some integer operations.

If you need to preprocess the input to LOAD, you can now do it easily by plugging a function into system/lexer/pre-load. This feature is mostly meant for pre-processing the console's input, though it could also be used for changing some Red syntactic rules. For example:

    system/lexer/pre-load: func [src part][replace/all src comma space]

    >> [1,2,3,abd,"hello"]
    == [1 2 3 abd "hello"]

Another usage could be to translate words on-the-fly in the console:

    system/lexer/pre-load: function [src part][
        parse src [
            any [
                s: [
                    "affiche"       (new: "print")
                    | "si"          (new: "if")
                    | "tant que"    (new: "while")
                    | "pair?"       (new: "even?")
                    | "impair?"     (new: "odd?")
                ] e: (s: change/part s new e) :s
                | skip
            ]
        ]
    ]

    >> i: 10 tant que [i > 0][si impair? i [affiche i] i: i - 1]
    9
    7
    5
    3
    1

In addition to several small fixes, load now offers a /trap refinement, which enables manual error management when loading a string series. Instead of raising an error, load/trap will just stop and return a block made of:

• a block of successfully loaded values.
• the input string at the position where the lexer failed.
• an error! value (or none! value if the tail of the string has been reached).

Command-line argument processing has been mostly rewritten to provide a consistent experience across platforms and type of binaries (red executables, console executables, compiled user scripts and, to a lesser extent, Red/System executables). The new features are:

• system/options/script refers to the script name (string! or none!).
• system/options/args refers to a list of tokenized arguments (block! or none!).
• system/script/args refers to the original command-line (string! or none!).
• full Unicode support for red executable's arguments on Windows.
• single-quoted arguments are accepted on all platforms (same as double quotes)
• multiple nested quotes are treated as just one level of quoting when splitting the command-line.

Datatypes conversions are now fully supported in Red! The to action is now officially supported, and make action has been completed. The usual to-<type> helper functions you can find in Rebol, are also now defined. All the conversion rules are documented in an Excel matrix for now.

Call

Calling external applications is also now possible thanks to the contributed code for call function by Bruno Anselme, which has now been integrated into the runtime library. Use help call to explore all the options offered. Note that the /console option, which redirects I/O from child process, is not supported currently by the Red GUI console (it works fine from within the Red CLI console).

View and VID dialect

A number of small changes and fixes have been provided both in View and VID, among them:

• box, h1 to h5 styles added to VID.
• Colors in VID can now be specified as hex values, using #rgb or #rrggbb formats.
• Adds support for no-border flag to area and field face types.
• Adds now option to rate keyword in VID, to fire on-time actor at once.
• The wheel event  and on-wheel actor are now available for handling mouse wheel events.
• Default tab size for area changed to 4 spaces.
• View now uses DirectWrite to draw text in base face (except WindowsXP).
• Better handling of default fonts.
• Enhanced GUI console, with new settings window with pre-selected colors picker.
• A hint text can now be specified in a field options block and a hint keyword has been added to VID. For example:

Draw dialect

Big additions were made to Draw, most thanks to massive contributions from Zamlox:

• Matrix operations support: matrix, invert-matrix, reset-matrix, matrix-order, push, rotate, scale, translate, skew, transform.
• A new clip command is available for defining a clipped drawing/filling region.
• A Shape sub-dialect has been added for more complex shapes drawing and filling.
• A crop option is now available for image command.
• pen and fill-pen have been vastly extended to allow drawing and filling with gradients, patterns, arbitrary shapes and images. Have a look at some of the new capabilities (source):

Parse dialect

In addition to some fixes, a few new features were added:

• insert command now also support position argument (like change).
• added pick option to keep, so user can control how keep captures the matched input:
• keep collects matched values as a series if many, or as a value if only one.
• keep pick collects all the matched values separately in a block.
• keep copy <word> collects all the matched values as a single series (of same type as input).

Red/System dialect

• Support for float! / float32! conversions from/to integer!.
• Adds system/cpu/overflow? field for reading CPU's integer overflow state.
• Adds support for importing variables from shared libraries.
• Allows loading of libraries from current folder and PATH environment variable on macOS.
• Now #call directive supports function calls with refinements.
• Default ABI for exported functions is now settable through export-ABI entry in config object.
• Renamed log and log10 imports from libC, respectively to log-2 and log-10.
• Now size? accepts a context path argument.
• Improved error reporting for "missing argument" errors.

Other changes

• A prototype Red/.Net bridge has been introduced.
• New --config [...] command-line option, for passing a block of compiler settings.
• Added -s and --show-expanded command-line options to output expanded version of compiled source code.
• Added /target option to react?.
• Added /seek and /lines option to write.
• Added /expand refinement to do for preprocessor invocation.
• Added math function for evaluating code using math precedence rules.
• CTRL-L key combination can now clear the GUI console's screen.
• Checksum function can now trigger object on-deep-change event.
• Now keep returns its argument (collect function).
• Added temporary rejoin function.
• Added 'class reflective property to objects.
• Added class-of accessor (only objects).
• Nicer handling of line breaks in molded image! binary buffer.
• Now #include is converted to do in interpreted code (using macros).
• Zero? function is now a native and supports time! values.

Also, more than 150 issues have been fixed (and wishes granted) during the last months, 22 issues marked as bugs are left open.

Last but not least, our documentation on Gitbook (which is an ongoing work) has been moved to Asciidoc format now, thanks to the efforts of Tovim. That new format will provide us better options for a more accurate control of the styles and layout.

A big thank goes to all contributors who pushed code or who opened tickets clearly identifying issues.

What's Next?

Since 0.6.1, we have adjusted the Red roadmap to work on two releases in parallel. This means that while 0.6.2 was progressing, 0.6.3 was advancing at the same time, is now almost ready, and will be merged into master in a couple of days (if you happen to have a Mac, it contains the macOS GUI backend for Red!). As soon as 0.6.3 is out, 0.7.0 will start (full async I/O), while 0.6.4 (Android) is being worked on. That should provide us a higher number of new releases this year, while still implementing large new features. So far, such approach has worked pretty well.

In the meantime, and as usual, enjoy playing with Red!