ARM support released

We are pleased to announce that Red/System v0.2.3 is out, extending Red to the mobile world with a complete support for Linux on ARMv5+ processors. The new port supports 100% of Red/System features and passes successfully the 8537 unit tests.

For those of you interested in more details, our ARM port targets the v5 family, with 32-bit instructions (no Thumb support) and optional literals pools (use -l or --literal-pool command-line option when compiling) that implement a global allocation strategy (unlike gcc's function-local allocations).

New compilation targets have been provided for Linux and derivative OS running on ARM:

• Linux-ARM
• Android

Android support is using the same code as Linux-ARM, only differing in libc and dynamic linker names.

Currently, as Red/System only works on command-line in Android, you need a special loader to download the executable and run it. This can be achieved using the NativeExe app. You will need to allow temporary installing apps from non market sources (Settings > Applications > Unknown sources). Also, your local 3G provider might be filtering out executables downloaded this way, you can workaround that by either manually loading the NativeExe-0.2.apk file with adb, or share a wired Internet connection with your mobile device.

You can easily install NativeExe app by just typing the following URL in your Android web browser:

http://gimite.net/archive/NativeExe-0.2.apk

or by scanning this QR-code instead:

Once done, input in the second field: http://sidl.fr/red/hello and hit [Run]. 


Here are a few screenshots of HelloWorld tests:

hello.reds script on Android 2.2

hello.reds script on Linux Debian 6.0 in QEMU

Andreas has also reported that it's working fine on Nokia N9 devices.


Supporting Android and iOS API

The next steps to enable building full apps on Android and iOS are:

1. Support PIC compilation mode: indirect access to all global variables and data. This is a requirement for building shared libraries on UNIX (but optional on Windows).
2. Add shared library generation to file format emitters. This will require some new compilation directives to mark the exported code parts.
3. Build a bridge with Java for Android and Objective-C for iOS/OSX. This generic bridge would allow accessing all the objects and methods of the host and send back to Red all I/O events.

Such approach will allow us to build easily Android or iOS apps without having to write a single line of Java or Objective-C code, while providing the full power of  Red. At least, that's the theory, we'll see in practice if it's up to our expectations. Also, cross-compilation should be fully available for Android (producing Android apps from any OS), but code signing and app publishing requirements might make it impossible for iOS and require a MacOS X with Xcode for producing apps (if you know workarounds, let us know).

The PIC support should be doable in a few days, the support for shared library generation might take a bit more time. Anyway, theses tasks will need to be multiplexed with Red runtime & compiler implementation, so don't expect significant progress before a month.

In the meantime, you are welcome to test the ARM port of Red/System and hack Android and upcoming Raspberry Pi devices using it. ;-)

Cheers!

Red/System v0.2.2 released

This release is mainly a bugfix release that solves several old issues. It is also now correctly synchronized with all the current bindings done for Red/System. The main changes are:

• Internal compiler refactoring of: expressions compilation, type casting and ANY/ALL support.
• Greatly improved runtime error reporting: now it reports both source line number and source file name where the error occured. It works in debug mode only (-g command-line option).
• Aliased struct names can now be tested separately in typed (RTTI) functions.
• Callback function attribute removed. It is no more needed and any function can now be used as a callback. In addition a new cdecl attribute is now accepted to allow the switch to C calling convention, when passing a function as argument to an imported C function.
• 21 issue reports closed.
• More than 2000 new unit tests were added (mostly generated using scripts wrote by Peter WA Wood) for a total of now 8613 tests.

Red at Software Freedom Day 2011

I have been invited at the Software Freedom Day 2011 (September 14th) to give a talk about Red. Here are the presentation slides and the video.

If you are experiencing issues seeing the slides in flash format, here's a PDF version.

I am not a great speaker, so you might want to go through the slides first, and eventually have a look at my corresponding comments in the video (it is too long anyway). I need to make shorter and faster presentations in the future.

Red/System v0.2.1 released

This new release is bringing a lot of important bug fixes and some new features for Red/System. The new PRINT function might break some existing scripts, but should be easily fixable.

New features

• Variable-arguments function support
• RTTI function limited support
• Polymorphic and variadic PRINT function
• Command-line arguments access
• Added stack low-level manipulation native functions: push, pop
• Access to stack pointer and stack frame pointer (reading/writing)
• Subtracting two pointers is now possible
• Preprocessor improved to solve macros recursively
• New built-in hexdump debugging functions
• External library access unit tests (thanks to Peter)

PRINT function examples:


print 123
123

print "hello"
hello

print [123 "hello"]
123hello

print [123 tab "hello" lf "world" lf]
123 hello
world

a: 1 < 2
b: "hello"
print [a tab b/2]
true e


Bugfixes

• Complex arithmetic expressions involving pointers, path access and type casting are now much more reliable. 89 new unit tests were written to help fix the bugs and track regressions.

• Complete list of fixed issues available in Github's tracker.

Specification document

• Updated to match all new and changed features

Red runtime

• Memory allocator implemented in Red/System, documentation is pending.

As you can see, no vacation this summer for Red project!

Mac OS X port released!

Red/System Mac OS X port is now completed and on par with Windows and Linux ports. The new Mach-O file emitter supports all current Red/System's features including #syscall and #import directives.

Now that all major 3 OS are supported, I can say that the Mac OS X file ABI and function call ABI are really the most complicated one to support from scratch, the Windows one (PE/COFF) being the simplest (with its own set of oddities, but simpler) and Linux's ELF somewhere in between. Among the issues we had to identify and deal with:

• Stack alignment on 16 bytes constraint at the point of every library call (meet with the infamous "gate keeper" code causing segfaults and bus errors on misaligned calls)

• Extra stack entry for each BSD system call.

• Working around dynamically linked libraries info extraction that need to be copied from each library header to the generated executable image. This could have prevented cross-compilation.

• The need for direct CALL opcode in the generated IA-32 code instead of indirect ones used for PE and ELF library function callings.

After a lot of efforts spent mostly in debugging, here is the result:

As you can see, it is fully functional and all unit tests are passing!

A couple of notes about usage:

• The library you declare in #import definition needs to be located in /usr/lib/ or else, you need to provide a full path to the library.

• The file emitter should support up to 255 imported libraries (Mach-O limit), but it has been tested only with one so far (libc.dylib), so if you have anything odd happening with more than one imported library, please fill a bug report in github's tracker, or post a message to the mailing-list.

Enjoy!

Red/System goes beta!

This is a major milestone reached on the Red roadmap. Red/System is now in a beta state, which means that it is good enough, both in design and implementation to be used for building applications.

A huge collective effort has been made by all contributors to reach that state during June, in a single month, we produced:

• 190 new commits (total = 354 now), more in a month than in the three previous ones!
• 3600 more unit tests (total = 6230 now)
• 2800 more LOC (total ~ 6000 LOC now)
• 70 issues closed on github's tracker
• 8 more revisions of specification document

This is what can be called an "intense" month of work!

There are still a lot of features that are lurking in the corner for inclusion. Red/System features are not set in stone, the language will evolve to match our needs, as a support tool for building Red. It will now follow a bit more organized release cycle to help users and contributors have a more stable base to work with.

The version number has been bumped to v0.2.0. From now on, only non-disruptive changes will be pushed on the main branch, other changes will go into a separate development branch until we decide to make a new release, merge new features and increment the version number. This cycle should be short, on a weekly basis (each week or two, depending on the development branch content). If this organization does not fit well the development progress, we will adapt to a more adequate one.

I cannot resist to show you the new unit testing output when running the complete tests suite (just run the %tests/run-all.r script):

Quick-Test v0.5.0
REBOL 2.7.6.3.1
Red/System Test Suite

ok - logic.............................76 / 76
ok - integer............................0 / 0
ok - byte..............................40 / 40
ok - c-string..........................20 / 20
ok - struct...........................116 / 116
ok - pointer...........................68 / 68
ok - cast..............................66 / 66
ok - alias.............................22 / 22
ok - length............................11 / 11
ok - null..............................11 / 11
ok - not...............................24 / 24
ok - size..............................36 / 36
ok - function...........................2 / 2
ok - exit...............................9 / 9
ok - return............................24 / 24
ok - modulo............................20 / 20
ok - infix..............................2 / 2
ok - conditional.......................13 / 13
ok - byte-auto.......................3492 / 3492
ok - integer-auto....................2106 / 2106
ok - alias-compile......................3 / 3
ok - cast-compile......................16 / 16
ok - comp-err...........................4 / 4
ok - exit-err...........................3 / 3
ok - int-literals-err...................1 / 1
ok - output.............................3 / 3
ok - return-err.........................5 / 5
ok - conditions-required-err...........23 / 23
ok - inference-err......................1 / 1
ok - callback-err.......................3 / 3
ok - infix-err..........................2 / 2
ok - not................................1 / 1

ok - Red/System Test Suite...........6230 / 6230
  in 0:00:16.3

A big thank to all the following contributors who are working hard each day to help me bring Red to life:

Andreas Bolka

Rudolf W.Meijer

Kaj de Vos

Peter W A Wood

Cheers!

EDIT: for those coming here for the first time and willing to test Red/System, please follow the link to Github's repository, you will find some instructions there for getting started.

Short look on last three months

Three months have passed since Red has gone public. It seems short, but a lot happened since then and I wanted to sum up the work accomplished for the people interested in Red that don't have the time to follow the progress daily on every channel.

Development

• Red/System is implemented at 98%
• ~2600 unit tests were created (thanks to Peter WA Wood)
• Red/System language specification draft added, now close to 95% of completion
• Code base almost doubled: 60KB to 110KB (~3200 LOC)
• 164 commits on Github, 44 by contributors
• 4600+ page views on Github's Red repository

Main new features

• New datatypes: byte!, logic!, pointer!
• Linux port (thanks to Andreas Bolka)
• Syllable port (thanks to Andreas and Kaj)
• MacOS X port in progress

Community

• 0MQ library support (Kaj de Vos)
• Quick-Test framework (Peter WA Wood)
• 3 contributors on Github, 22 followers
• 65 issues opened on Github, 53 closed (thanks to Rudolf Meijer)
• 8 blog posts, 10'000+ page views
• #red_lang: 130 tweets, 39 followers
• Mailing-list: 32 members, 45 topics, 150 posts
• IRC: 5 people permanently logged

Last but not least, we got an article on OSNews two days ago that boosted the page views on red-lang.org (+3'000) and Github's repo (+1'000).

Thanks for all people supporting Red, one way or another. This is a very good start and a big incentive for me to work even harder on the next steps.

Red meeting on 28/05 at Lille

I will present Red language and the Cheyenne Web Server at Lille (France) on May, 28 to a group of REBOL and Red users. The presentations will be done in french, but if some none-french speakers want to come, I will try to do it both in french and english.

Detailed information to get there are available in this thread from RebelBB forum. See you there on Saturday!

Red/System compiler overview

Source Navigation

As requested by several users, I am giving a little more insights on the Red/System compiler inner workings and a map for navigating in the source code.

Current Red/System source tree:

 red-system/
     %compiler.r        ; Main compiler code, loads everything else
     %emitter.r         ; Target code emitter abstract layer
     %linker.r          ; Format files loader
     %rsc.r             ; Compiler's front-end for standalone usage

 formats/               ; Contains all supported executable formats
     %PE.r              ; Windows PE/COFF file format emitter
     %ELF.r             ; UNIX ELF file format emitter

 library/               ; Third-party libraries

 runtime/               ; Contains all runtime definitions
     %common.reds       ; Cross-platform definitions
     %win32.r           ; Windows-specific bindings
     %linux.r           ; Linux-specific bindings

 targets/               ; Contains target execution unit code emitters
     %target-class.r    ; Base utility class for emitters
     %IA32.r            ; Intel IA32 code emitter

 tests/                 ; Unit tests

Once the compiler code is loaded in memory, the objects hierarchy looks like:

 system/words/          ; global REBOL context

 system-dialect/        ; main object
     loader/            ; preprocessor object
         process        ; preprocessor entry point function
     compiler/          ; compiler object
         compile        ; compiler entry point function

 emitter/               ; code emitter object
      compiler/         ; short-cut reference to compiler object
      target/           ; reference the target code emitter object
          compiler/     ; short-cut reference to compiler object

 linker/                ; executable file emitter
     PE/                ; Windows PE/COFF format emitter object
     ELF/               ; UNIX ELF format emitter object

Note: the linker file formats are currently statically loaded, this will be probably changed to a dynamic loading model.

Compilation Phases

The compilation is a process that goes through several phases to transform a textual source code to an executable file. Here is an overview of the process:

1) Source loading

This is a preparatory phase that would convert the text source code to its memory representation (close to an AST). This is achieved in 3 steps:

1. source is preprocessed in its text form to make the syntax REBOL-compatible
2. source is converted to a tree of nested blocks using the REBOL's LOAD function
3. source is postprocessed to interpret some compiler directives (like #include and #define)

2) Compilation

The compiler walks through the source tree using a recursive descent parser. It attempts to match keywords and values with its internal rules and emits:

• the corresponding machine code in the code buffer
• the global variables and complex data (c-string! and struct! literals) in the data buffer

The internal entry point function for the compilation is compiler/comp-dialect. All the compiler/comp-* functions are used to recursively analyze the source code and each one matches a specific semantic rule (or set of rules) from the Red/System language specification.

The production of native code is direct, there is no intermediary representation, machine code is generated as soon as a language statement or expression is matched. This is the simplest way to do it, but code cannot be efficiently optimised without a proper Intermediate Representation (IR). When Red/System will be rewritten in Red, a simple IR will be introduced to enable the full range of possible code optimisations.

As you know, a Red/System program entry point is at the beginning of the source code. During the compilation, the source code in the global context is compiled first and all functions are collected and compiled after all global code. So the generated code is always organised the same way:

• global code (including proper program finalization)
• function 1
• function 2
• ...

The results of the compilation process are:

• a global symbol table
• a machine code buffer
• a global data buffer
• a list of functions from external libraries (usually these are OS API mappings)

The compiler is able to process one or several source files this way before entering the linking phase.

3) Linking

The linking process goal is to create an executable file that could be loaded by the target platform. So the executable file needs to conform to the target ABI for that, like PE for Windows or ELF for Linux.

During the linking, the global symbol table is used to "patch" the code and data buffer (see linker/resolve-symbol-refs), inserting the final memory address for the pointed resources (variable, function, global data). The different parts to assemble are grouped into so-called "sections", that can be themselves be grouped into "segments" (as, e.g., in ELF).

Finally, some headers describing the file and its sections/segments are inserted to complete the file. The file is then written down on disk, marking the end of the whole process.

Static linking of external libraries (*.lib, *.a,...) will be added at some point in the future (when the need for such feature will appear).

I hope this short description gives you a better picture on how Red/System compiler works, even if it is probably obvious for the most experienced readers. Feel free to ask for more in the comments, or better, on the Google Groups mailing-list.

Update on Red/System progress

Here is a quick update on Red/System progress.

Specifications Draft

Draft has gone through several revisions. Several decisions mainly regarding pointers handling have been taken with simplification and disambiguation as main goals:

• Struct! values are now passed by reference (a syntax for passing by value will be added in future), simple arithmetic (+, -) can be used on struct references.

• String! was renamed to c-string! to avoid possible future collision with a Unicode-aware string! type. However, this case is not totally solved, there are still some possible issues, like the literal syntax. As c-string! values are passed by reference, simple arithmetic was added too.

• Pointer! datatype was restricted to point to integer! values only. It is more consistent with new c-string! and struct! which are both (implicit) pointers already, and that removes the need for verbose pointer syntax.

• Array! datatype: no decision taken for now, we will see if (both explicit and implicit) pointer arithmetic support will be sufficient during Red's runtime implementation.

• Logic! datatype was added as a first class type. This makes boolean literal values (true, false) and conditional expressions assignable to variables and can be passed as argument or returned by user functions. Logical NOT operator was also added as well.

• Byte! datatype (unsigned byte value) was added to ease string byte accesses.

• Limited type inference was added for function's local variable and return value types.

A couple of things to keep in mind about Red/System design:

• during the implementation of these specifications, some better design options might appear in light of the implementation decisions.

• at some point, when the bootstrapping phase will be over, Red/System will be rewritten entirely in Red, giving an opportunity to fix and improve the design (also helped by several months of feedback from Red/System usage).

Implementation Progress

Several items from the todo-list have been implemented in the last week, the most notable ones being:

• Logic! datatype support
• NOT function addition
• EXIT and RETURN support (for early exit from functions)

Among other changes:

• ELF emitter has been extended a bit to support sections table (thanks to Andreas).

• An experimental Mach-O (OS X) support is under work by Rebolek.

• Several unit test files have been added to the repository recently, using a temporary minimalistic approach, that will be soon superseded by a real dedicated framework.

• We are close to have a nice, small and lean unit test framework (Quick Test), created by Peter W A Wood for Red/System, covering all our needs (internal and external testing).

An interesting side note on how implementation can influence back the design:

During the implementation of Quick Test framework, Peter, while writing an integer to string conversion function, came up with a code pattern that is common in REBOL, but wasn't planned for Red/System: declaring private functions inside a function to hide them from global context. The pattern looked like this:

    prin-int: func [...][
        ...
        prin-digit: func [...][...]
        ...
        prin-digit ...
        ...
    ]

I was surprised when I tested the code he sent me, as I never did anything in the implementation to support such case explicitly. It was working because of a side-effect caused by the way functions are compiled in Red/System: the functions bodies are all gathered and compiled at the end of the global context code, regardless of where they are defined. The side-effect making the inner "prin-digit" function private is caused by a global function variable being declared inside a local scope, but without being part of the function specification block. So, once the outer function compiled, the "prin-digit" symbol is lost, making the function unreachable from global context and forever hidden.

I was thinking about adding a way for users to create private contexts much later in the future (relying on Red/System source header), but we have the opportunity to have an equivalent feature, right now and without adding anything to the code! So, I am giving it some time to see if the side-effect can be safely made permanent to keep that feature, and in the meantime, added it to the Possible Evolutions part of the specifications.

In the next days, I will work on:

• finishing implementing the features from the todo-list
• merging the unit tests framework once ready
• adding new unit tests

In my next blog article, I will give an overview of Red/System compiler internals as requested by some followers.

Red/System draft specifications released

I have published yesterday the first draft of Red/System's specifications. It is amazing how much time and energy writing such formal description can take. It is a working draft, so expect it to be updated often during next weeks. The document source is in MakeDoc format and stored on github, so feel free to fix typos and errors directly there.

As all features in the specifications are not yet implemented (I would say 85% is already done), I have added a todo-list on github's wiki to track the missing parts.

Also, all features are not yet set in stone, there are still a few important design decisions to take in the next weeks:

• Pointer! datatype: should we keep it or consider that struct! datatype can do the same job, so it's not worth the trouble of supporting an additional type for the syntactic sugar? Let's take an example with a pointer on integer and struct having a single integer member:
  

  p: &[integer! 0]
  p/value: 123
  
  p: struct [value [integer!]]
  p/value: 123
  

  Looks pretty much the same. Pointers have a shorter literal form, but once declared, structs could be used the same way and replace pointers in all use-cases.

• Array!: will it be necessary to add a standalone array! datatype or could its semantics be simply added to struct! or pointer! (if finally kept)?

• Booleans: there's currently no true/false support in Red/System, so that boolean values are not first class (can't be assigned nor passed as argument). This is quite limiting the expressibility of the language. Is using simple integer values (0,1 or 0,-1) enough or will it be better to support it as a separate datatype (logic!)?

The answers to these questions will come while working on unit tests and coding Red's runtime (written in Red/System). Feel free to share your thoughts about these features here, in comments, or on Google groups.

Comments relaxed

I wasn't aware that the comments on this blog required to have an account on Google/OpenID/... to be able to post comments. It was the default setting, but not my intention to make it hard or painful to post here, so I've relaxed this setting, anonymous posts are now allowed (you just need to pass a captcha).

Having fun with Red/System

A week after the first alpha release of Red/System compiler for Windows, we now have a working Linux/ELF support, thanks to Andreas help in deciphering the ELF format. Currently only syscalls are available, the dynamic library linking part is pending, it should be added before the end of March. The following Red/System "hello world" script:

Red/System [
   Purpose: "hello world script"
]

print "Hello World!"

compiles to a 162 bytes ELF binary, while a similar C code would produce a 5-6KB binary using Gcc...pretty good, no?

I must admit that supporting ELF wasn't in my short-term roadmap, but I changed my mind after seeing that a large part of people following Red project are Linux users. I think it was a good move as Linux users are also often hackers, so more inclined to contribute to an open source project like Red.

I was also impressed, yesterday, when I saw Kaj de Vos publishing a 0MQ mapping for Red/System (running only on Windows or Wine currently, as it needs dynamic linking). Even at this early stage, he managed to wrap 0MQ's main features. You can get his script from here (requires a libzmq.dll file that can be obtained from here, just put it in the same folder as the compiled binary). Here's the result after compiling and running the server and client versions:

The client sends 10 "Hello" messages to the server that replies to each of them with "World". Oh, did I mention that client.exe and server.exe files size is 3KB? :-)

I hope that people already playing with Red/System enjoy it as much as I did writing it.

Taking the Red/System pill

Here we go, this is the first release of Red/System dialect. Of course, it's an early alpha, don't expect too much from it for now (not crashing would be a success). I know that most of you are eager to see what this looks like, so, you can download right now an archive from github, or better clone the repository from:

https://github.com/red/red

See the README file for instructions on running the little demo script (tests/hello.reds).

Design & Implementation notes

Red/System is a low-level dialect (or DSL) of Red. Its main purpose is to allow implementing a small runtime for Red, basically covering three domains:

• Memory management (MAKE primitive)
• Lexical analyzer (LOAD primitive)
• Executable container

So, the feature set of Red/System has been chosen with regards to the goals to reach. Extending them beyond the main purpose is not in the current plan, but could become an interesting side-project for anyone interested in implementation of programming languages.

Red/System relies on a static compiler and a linker in order to produce working code. Here's a short list of currently supported features:

• types: integer!, string! (partial), struct!, pointer! (partial).
• literal strings only for now (can't build or change one at runtime)
• math ops, boolean ops, comparison ops (all as infix operators)
• control flow: if, either, until, while, any, all
• size? function (== sizeof( ) in C)
• length? function (get the size of a string at runtime)
• user functions (with local variables and return value support)
• OS functions importing

Main features not yet implemented:

• function calls and return value type-checking
• no duplicate variables name checking
• no reserved words protection
• int8! and int16! datatypes (maybe int64! too if required)
• pointer values arithmetic
• string's bytes read/write access
• arrays support
• proper minimal runtime for Red/System scripts

The compiler is a naïve one (non-optimizing) using a straightforward approach. It works in 2 passes:

• first pass: lexical analysis producing nested s-expressions (== LOAD)
• second pass: direct compilation from high-level to machine code

There's no intermediary representation, nor assembly mnemonics generation, just direct machine code output. I've used this compiler as an opportunity to prototype several ideas, mostly seeking the simpliest ways to reach to goals, with the best possible performance. Also, I expect to re-use as much code as possible from this compiler for Red's compiler itself. That's the main reason for not choosing C to build the runtime.

So far, the result is quite good, there are a few limited parts that I'm not very happy with, but will probably wait for the re-implementation of Red/System in Red before fixing them:

• Code generation for control flow function: that's the most complicated part in the compiler, mainly caused by the need to generate code in chunks with mutual dependencies to resolve (mixed forward branching). I managed to reduce/hide the complexity somehow, but I'm still not satisfied with the result (in terms of code readability).

• Machine code emitter is too verbose: the emitter (IA32 currently only) is too coarse-grained, this makes it easier to work with, but the CPU-dependent code is much larger than it could be, making the porting effort to other targets, bigger.

In the next days, I'll work on :

• adding missing features
• start writing unit tests and fixing bugs
• documenting Red/System specifications (on github's wiki?)

The goal is to be ready as soon as possible for implementing Red's runtime.

The Journey Begins!

I've announced yesterday at ReBorCon 2011 conference my intention to build a new programming language, strongly inspired by REBOL. See the slides presented during the conference below.

If you're experiencing issues seeing the slides in flash format, here's a PDF version.

More to come soon, stay tuned!