Red v0.3.1: functions support added

This new milestone brings (finally) functions to Red along with other new features and fixes. Let's first have a quick overview of 0.3.1 release content:

• function support
• path get/set notation support
• refinement support for native functions
• expressions in parentheses compilation
• new datatypes: function!, paren!, path!, lit-path!, get-path!, set-path!
• new actions and natives: find, select, copy, reflect, type?, halt,...
• extended mold, form and comparison operators to all new and existing datatypes
• many new mezzanines functions
• modulo and remainder operators on floats implemented for ARM backend
• Quick-Test testing framework ported to Red
• a truckload of new unit tests
• many bugfixes
• about 200 new commits to the Github repository
• updated Red/System formal description document

Functions

They are declared using a similar syntax to REBOL, with some extensions. The specification block looks like this:

    [arg1 [type1] arg2 [type2]...return: [type] /local var1 [type-v1]...]

• Arguments and local variables are declared in the same way as in REBOL
• A return value type can be optionally specified
• Local words can have a type specifier

All type specifiers are optional, adding them will not only allow the compiler to make additional type checkings but also generate faster code in some cases (once the optimizations added to the Red compiler).

Note: argument and return value type checking have not been implemented yet, they need typeset! and error! datatypes to be implemented first.

Functions can be built using several constructors:

• function: automatically collects local variables, like `funct` in REBOL.

    twice: function [a [integer!] /one return: [integer!]][
    c: 2
      a: a * c
    either one [a + 1][a]
    ]

• func: low-level function constructor, everything needs to be manually specified.

     twice: func [a [integer!] /one return: [integer!] /local c][
      c: 2
      a: a * c
      either one [a + 1][a]
    ]

• has: no argument, just lists local words.

    globals: [1 2 3 _]
    
    foobar: has [list][
      if list: find globals '_ [clear list]
    ]

• does: no argument, no local words.

    quit: does [
      print "Goodbye cruel world!"
      halt
    ]

Early exit points are also there in form of exit and return functions.

    foobar: func [a [integer!] b [integer! none!]][
      if none? b [return none]
      if b = 0 [
         print "Error: division by zero"
         exit
      ]
  a / b
    ]
 
Refinements are also fully supported, when not used, their value is set to false (while in REBOL it is set to none).

Still some features are not yet implemented:

• building a function at runtime (requires the ability to JIT-compile source code)
• passing a function as argument (pending)
• tail recursion optimization

Currently functions are using a local context on stack, so their life-time is very short. For indefinite extent support, a closure! type will be added in the next months.

Path support

Path datatypes were added (path!, lit-path!, get-path!, set-path!) to the runtime library and compiler, allowing the use of path notation as syntactic sugar for series access. Getting and setting values using path notation are supported.

Some usage examples:

    list: [a 5 b [c 8] d #"e" name: "John"]
    
    probe list/2 ;-- outputs 5
    probe list/7 ;-- outputs name:
    probe list/a ;-- outputs 5
    probe list/4/2 ;-- outputs 8
    probe list/b/c ;-- outputs 8
    probe list/name ;-- outputs "John"
    
    list/b/c: 0
    probe list/b/c ;-- outputs 0
    
    index: 3
    probe list/:index ;-- outputs b
    list/:index: 'z
    probe list/:index ;-- outputs z
    
Note: notice how words lookups in blocks can work with any word datatypes.

Paren expressions

Paren! datatype and expressions in parentheses compilation support has been added too. They are mainly useful with infix operators in order to force a given execution priority. Example:

    print (1 + 2) * (3 + 4) ;-- outputs 21

New built-in functions

• actions: find, select, copy, reflect

• natives: type?, halt, function, func, does, has, exit, return

• mezzanines: probe, first, second, third, fourth, fifth, last, spec-of, body-of and all <datatype>? type testing functions.

Existing native or action functions were extended with refinements (like mold and form). See the boot.red script for more details.

Testing

The Quick-Test framework from Peter WA Wood was ported to Red and now powers all the current 1800+ unit tests written in Red itself, mostly added by Peter.

About 32 issues/bugs in both Red and Red/System were fixed in this new release.

Red/System formal description

Last but not least, Rudolf updated his Red/System formal description document [PDF, 162KB] to include the additional features like namespaces. Many thanks to him for his remarkable work and support to Red project!

What's next?

Objects support is next priority along with new natives like switch and case. Typesets and error handling will probably be added very soon too. Once that part done, the next focus will be to add I/O support!


Many thanks to all the people that have made significant donations in last months to Red project, allowing me to continue to work full time on it! It is really making a big difference, as shown by Red increased progress speed since this summer. Thanks again!

Cheers!

Red alpha release

We have all waited long for this to happen, but it is finally there: first Red alpha release!

This is the biggest milestone since the initial release of Red System, as it marks the entrance of the higher layer of the Red software stack. There's still another layer pending, the JIT compiler, but that's for 2013, when Red will be self-hosted.

What Red can do so far?

So what happened since the first Red compiler release a little more than a month ago? We have added quite a lot of features actually:

• 15 datatypes: block!, string!, char!, integer!, logic!, none!, word!, lit-word!, get-word!, set-word!, refinement!, action!, native!, op!, unset!.

• 19 actions: make, form, mold, add, divide, multiply, subtract, append, at, back, clear, head, index?, length?, next, pick, poke, skip, tail.

• 21 natives: if, unless, either, any, all, while, until, loop, repeat, foreach, forall, print, prin, equal?, not-equal?, strict-equal?, lesser?, greater?, lesser-or-equal?, greater-or-equal?, not.

• 10 infix operators: +, - , *, /, =, ==, <, >, <=, >=.

String! datatype fully supports Unicode from end to end and in a cross-platform way, according to our plan (see the hello.red demo script).

We also have some unit tests (223 tests right now), but that number should rapidly increase once we will be able to fully port QuickTest framework to Red (in a couple of weeks, once Red will gain functions).

All those features were mostly coded in the last month, some parts are still lacking (e.g. refinements support for actions/natives) but that gives you an idea of the pace at which Red will be further developed.

However, we haven't started yet writting the documentation for Red language, we should decide soon on the kind of documentation we want (hint: you can make propositions about that on the Red mailing-list or FB page). Red API documentation will most probably be generated automatically from source code by extracting doc-strings.

What's not yet there?

A lot of things, but they will be added progressively until the end of the year, where we should have a pretty complete Red (bootstrap) implementation. So, the main pieces to implement are:

• Red runtime lexer (LOAD native)
• Functions and objects support
• I/O support (including networking)
• More complete memory allocator and a garbage collector.
• Concurrency support
• Additional datatypes

Can I try it already?

Red codebase is available at github, just clone it locally or download a packaged version. Some simple instructions to setup and run it can be found here. As we are still at the bootstrap stage, the installation process and usage is more complex than it should be, once self-hosted. Anyway, we should be able to deliver some test binaries soon and simplify the usage for those who want to play with the alpha releases.

How does Red work?

Currently Red scripts are statically compiled using the %red.r front-end wrapper. The Red lexer loads the UTF-8 encoded Red scripts in memory in form of blocks of values. Those blocks are then compiled to a Red/System script, that gets itself compiled to native code and outputs an executable binary. The Red/System compiler has been enhanced to be able to compile source code directly from memory.

Some compilation options are already available, the famous -t option from Red System is also present, allowing Red to cross-compile any script as easily as with Red/System.

This is just the beginning, so for now, whole Red runtime library is compiled with user scripts each time. We will add modular compilation support at some point to get the full compilation speed back.

You can test the few example Red scripts from the repository. Here is what the demo.red script outputs on some of the platforms Red can already run on:

The missing characters in some of the screenshots are Chinese ones as I only have font support for them on Ubuntu.

Red was originally conceived to be statically typed. However, during the early stages of Red implementation, it appeared that an hybrid typing system will be more adequate for a language that will support high-level scripting, without the burden of a sophisticated, but slow, type inference engine. The overhead of such inference engine can be very high (cf Scala slow compilation speed ). The hybrid type system works very simply, typing is optional, but if local variables and function returned values are typed by user, the compiler will be able to generate faster code (more than an order of magnitude faster for, e.g., pure math expressions) and catch more programming errors at compile time.

Red compiler does not do any optimization so far, it basically outputs the equivalent of an unrolled interpreter loop. Some optimizations will be added along the development of the bootstrap version of Red,  but the full extent of possible optimizations will be left for the self-hosted Red.  Firstly because the current compiler code is disposable, so we don't want to invest too much time in it, secondly because the final JIT compiler will allow even deeper optimizations thanks to reflection and profiling information collected at runtime.

Red/System changes

Red/System has been improved significantly during the Red early development:

• function pointer support has been extended and improved, dereferencing is now possible.
• 20 issues have been fixed.
• some long-standing bugs with GTK+ and other bindings on ARM platforms have been fixed.

So, what's coming next?

Red will now mature very quickly, as functions and objects will make their entrance in the next days/weeks. A new Red alpha release is planned very soon, probably for mid-november with a lot of new features.

The work on shared library generation for MacOS and Linux platforms is on-going. Once available, we'll be able to literally "plug" Red in a lot of third-party apps, including other programming languages. Also, this will enable (or make easier) the bridging with some external systems like Java, .Net and objective-c, in order to fully support the main mobile platforms.

As you can see, next months will see some important parts of Red come to life, be sure not to miss them, follow us on our mailing-list, on Twitter, IRC and Facebook.

Cheers!

Plan for Unicode support

Red is growing up fast, even if just born two weeks ago! It is time we implement basic string support so we can do our first, real, hello-word. ;-)

Red strings will natively support Unicode. In order to achieve that in an efficient and cross-platform way, we need a good plan. Here is the list of Unicode native formats used by our main target platforms API:

        Windows       : UTF-16
        Linux         : UTF-8
        MacOSX/Cocoa  : UTF-16
        MacOSX/Darwin : UTF-8
        Java          : UTF-16
        .Net          : UTF-16
        Javascript    : UTF-8
        Syllable      : UTF-8
   
All these formats are variable-width encodings, requiring any indexed access to pay the cost of walking through the string.

Fortunately, there are also fixed-width Unicode encodings that can be used to give us back constant time for indexed accesses. So, in order to make it the most space-efficient, Red strings will internally support only these encoding formats:

        Latin-1 (1 byte/codepoint)
        UCS-2   (2 bytes/codepoint)
        UCS-4   (4 bytes/codepoint)

This is not something new, at least Python 3.3 does it in the same way.

Additionally, UTF-8 and UTF-16 codecs will be supported, in order to deal with I/O accesses on host platforms.

Red will use UTF-8 for exchanging strings with outer world by default, except when accessing a UTF-16 API is necessary. Conversion for input and output strings will be done on-the-fly between one of the internal representation and UTF-8/UTF-16. When reading an input string, Red will select the most space-efficient internal format depending on highest codepoint in the input string. Also users should be able to force the encoding of a string to a given internal format, when possible.

So far, this is the plan for additing Unicode to Red, a prototype implementation will be done quickly, so we can fine-tune it if required.

Comments and suggestions are welcome.

Red is born

Yesterday, I finally got my first real Red program compiling and running:

    Red []
    print 1

This doesn't look like much but it proves that the whole current Red stack is working properly. Red compiler generates Red/System code in memory, that is then compiled to native code and linked in a 14KB executable file (contains the full Red runtime with currently 9 datatypes partially implemented).

The baby needs a few more days in the nursery before I commit the new code to the v0.3.0 branch on Github. Once the existing datatypes will be more complete and once we choose how to deal internally with Unicode strings, we should be able to release the first Red alpha.

The current Red compiler is pretty simple and should remain light in the future. I have finally chosen an hybrid dynamic/static type system, to avoid diving into a complex type inference engine now, as I realized that once we get out of the bootstrap and have the final Red JIT-compiler, it will be much more easier to achieve. Also, I want to pass that bootstrap stage as soon as possible, because it is really limiting the full Red potential.

Stay tuned, the next months will be fun! ;-)

Red/System v0.2.6 released

The need for more structured code support has arisen with the addition of bigger and more sophisticated bindings like GTK one or the (soon to be released) work on alpha version of Red compiler (which emits Red/System code). So, v0.2.6 introduces namespaces support using a very simple, but efficient model.

For example, to encapsulate some variables and functions in a local context:

    b: 1
    a: context [
       b: 2
       inc: func [i [integer!] return: [integer!]][
           i + b
       ]
    ]
    a/inc 5                            ;-- will return 7

Local variables take precedence other global ones with same name. This simple rule also applies to nested contexts, the nearest one has priority, e.g.:

    a: context [
       b: 123

       c: context [
           #enum colors! [red green blue]
           b: "hello"
           foo: func [][print-line b]
       ]

       print-line b                    ;-- will output 123
       c/foo                           ;-- will output hello
    ]

    print-line a/b                     ;-- will output 123
    a/c/foo                            ;-- will output hello
    print-line a/c/b/2                 ;-- will output e
    print-line a/c/blue                ;-- will output 2

As you can see from this example, enumerations can also be defined locally to a context, but not only. Aliases and imported functions can also be defined locally too! With that powerful new feature, you can now modularize your Red/System programs simply and efficiently.

See the namespaces documentation for more features and info.

In this release you will also find several bugfixes, mainly for floats support. Thanks to Rebolek for his aggressive floats testing. ;-)

Enjoy!

Setting up an ARM virtual machine

EDIT (March 8, 2014): Updated for the 1.6.0 QEMU version.

The recent release of the Raspberry Pi board raised a new wave of interest for the ARM platforms. As Red/System is already running on ARMv5+ platforms, a number of Red followers might want to test it on these platforms, so this article aims at giving you a quick way to setup a QEMU emulator for ARM and install a Debian GNU/Linux on top of it.

QEMU

First thing first, let's setup the platform. QEMU is the tool we want for that job. As explained on the QEMU web site: "QEMU is a generic and open source machine emulator and virtualizer." The nice thing about QEMU is that it is able to emulate a good range of ARM platforms with different CPU versions.

QEMU is distributed as a source package, like most of open source tools, however this is not always convenient for Windows users, so you will also find a link to Windows pre-compiled binaries.

• UNIX users: download and compile QEMU from sources. 
• Windows users: download the binaries from this page, or from this direct link to 1.6.0 zip package (52MB)

Installing GNU/Linux

The simplest and quickest way to get a Linux/ARM distribution running on top of QEMU is to used pre-installed Debian images. These Virtual Machine images can be found there along with additional useful information (be sure to read them all). As you can see, there are two kind of images: standard and desktop. The desktop one gives you the full Debian GUI environment pre-installed, but be warned, it is extremely slow, even on my Core i7 box with a medium-level graphic card, so I recommend using the standard version pre-installed with Debian Squeeze (latest Debian release).

The files you need from the debian.org site are (direct links provided for convenience):

• debian_squeeze_armel_standard.qcow2 (169MB)
• initrd.img-2.6.32-5-versatile (2.4MB)
• vmlinuz-2.6.32-5-versatile (1.2MB)

Put them all somewhere in the same folder.

The command line for starting QEMU with the pre-installed VM is provided on the debian.org page, here is a copy of the right command line for the standard VM with 2.6.32 kernel:

qemu-system-armw -L Bios -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile -hda debian_squeeze_armel_standard.qcow2 -append "root=/dev/sda1"

For Windows users, the easiest way to start it is to make a shortcut on qemu-system-arm , add the command-line arguments and insert the path to the VM files in the "working folder" field.

You can now just run the emulator and boot on Debian to test your environment.

For purists, it is also possible to install the Debian distro from scratch following this step-by-step tutorial.

Making the Virtual Machine communicate

As you will quickly discover, the network connection is not working out of the box, so you need to add more parameters to the command-line to make it work. From my own experience, it is a black art to make it work properly (the documentation is really poor and obscure on that topic), but I ended up making the network work (and the Internet connection through the host platform) by using these additional arguments:

    -net nic -net user

Once the VM is rebooted, you can test it using a ping and then issue an `apt-get update` command. In case the ping test passes but not the apt-get, you might have wrong apt source URL, so you can change them by editing the /etc/apt/sources.list config file.

You have now a working ARM virtual platform, but in order to make it useful for Red/System testing, you need a simple way to pass compiled files to the guest OS (Red/System can only cross-compile to ARM due to the lack of a suitable REBOL interpreter).

In order to achieve that, you need to add new parameters to the command-line in order to map a guest OS TCP port to a host OS port. For example, to map guest OS port 22 (SSH) to port 1222 on host OS, just add:

    -redir tcp:2222::22

So, the VM internal port 22 is now reachable from localhost:1222 on your host OS. You can now use your favorite SSH/SFTP/SCP client to get a remote shell and transfer files to the VM. You might need to setup a SSH server in the VM (I seem to recall that it is not installed by default in the above images), you can achieve that using the following command:

    # sudo apt-get install openssh-client openssh-server

I use SSH for moving files in and out the VM using the still excellent SSH Secure Shell tool for Windows, but you could as well use FTP or any other TCP-based suitable protocol.

Additional settings

You can improve your experience with QEMU by adding more memory to the default emulated ARM platform which only has 128MB. Pushing it to 256MB (same as Raspberry PI Model A board now has) is highly recommended, just add to the command-line:

    -m 256

If you are using a non-english keyboard like myself, you can also emulate your preferred keyboard layout using for example, the following command:

    -k fr

will setup the keyboard with a french layout, other country codes are available here. In case you get an error message about the keymap not being found, just copy the Bios/keymaps folder one level up, so that it is just above your qemu-* binaries.

That's all folks, I hope this would have help you get started! Let me know if there is any error or missing info in this article, and let me know if it worked well for you.

Red/System v0.2.5 released

This is a mainly a bug fixing release with several issues and bugs in float numbers handling solved.

In addition to that:

• Libc is now much better integrated with Red/System, the __libc_start_main C internal initialization function is now correctly handled by Red/System on all supported platforms. This ensures that all libraries relying heavily on C will now work as well as if called from a C program. Thanks to Andreas for the nights spent on digging and debugging that.

• The IA-32 floats backend has been improved to keep last float value in FPU rather than passing it to CPU. This change not only simplifies the backend code emitter but also reduces significantly the generated code size for floats. As a side effect, float-intensive programs are now twice as fast as with v0.2.4. The same change could be applied to ARM backend, but with less gains as ARM can easily transfer data between CPU and FPU, as opposed to IA-32 architecture, which requires an intermediary step through memory when using x87 FPU.

By the way, that's our first birthday (technically ten days ago) since Red project was announced for the first time at the ReBorCon 2011 conference in Amsterdam. The initial schedule has been changed several times since then, to better adapt to the very rapid changes that are occurring in the computing world (massive move to mobile devices to name the main one).

After 900+ commits and thanks to all the contributors, we now have solid foundations to build upon, much better than what I expected to have, one year ago. The focus from now is getting the Red language layer up and running as soon as possible. A large window of opportunities is opening in 2012, so let's not miss it!

Cheers!

Floating point support released

New Red/System version 0.2.4 is out. The main new feature is floating point numbers support. This feature was not planned, but as users were strongly pushing for having it now, we have added partial support for floats. Users can now wrap and use libraries like OpenGL.

New floating point datatypes

Two new IEEE-754 datatypes were added:

• float64! datatype implements double precision (64-bit) floating point numbers. It has an alias named float! which is the preferred name (both will be accepted by the compiler anyway).

• float32! datatype implements single precision (32-bit) floating point numbers.

They are first-class datatypes, so they can be used anywhere, as variables, arguments or returned values.

Float operations

Basic math and comparison operations are supported. Transcendental functions are not predefined yet in the runtime, they need to be imported first from an external library. For equality test, no rounding is applied, so only exact float numbers match will return `true`. Adding an "almost-equal" operator is planned, but not available yet.

Also, type casting is allowed between:

• float! <=> float32!
• float32! <=> integer!

Remember to type cast all float32! values to float! when passing them as argument to C variadic functions. For non-variadic C functions, Red/System's compiler will promote float32! to float! automatically.

Limitations

Not all of IEEE-754 standard was implemented yet. See the remaining features to add in this todo-list.

Implementation

At low-level, floating point support is architecture-specific:

• IA-32: currently uses x87 FPU instructions for backward compatibility with older Intel CPU. The default implementation requires a PentiumPro or better. SSE unit support will be added when a proper optimizing compiler will be available. For older CPU support, a new compilation option has been added (`cpu-version`), which triggers the use of older opcodes. For example, in order to use floats on older than PentiumPro CPU, you need to add this line in the target definition:   cpu-version: 5.0

• ARM: uses the VFP unit when available and was reported to run fine on emulated VFP. When calling some C functions passing or returning float numbers, some ABI mismatching were also reported during the tests, as ARM uses different ABI for passing float numbers as arguments. Our current implementation only use the passing-on-stack convention, adding support for other conventions is not planned, but will be done when really required.

Tests coverage extended

Several thousands new tests were added to the test suite, pushing the total number of unit tests to 11755. Thanks to Peter for taking care of that.

Decimal support

As IEEE-754 2008 revision introduced several new decimal floating point formats, we plan to support them later in Red/System.

Other feature released: enumeration support

A contributed feature is also part of this new version of Red/System: C-like enumeration support. This contribution was brought by the lead developer of Amanita Design, small independent game developing studio based in Czech Republic, which released the award-winning game Machinarium. A new compilation directive was added to declare enumerations: #enum.

Last but not least...

We got noticed by Ruby's author, Matz, he tweeted about Red! :-)