0.6.4 Simple GC and Pure Red GUI Console

It's been a long time coming.

While work on Red has progressed every day, work on the blockchain aspects of the project, and their associated logistics, came at a time when that workload temporarily prevented us from making new, formal/stable releases. There was a key feature that needed to be in place, and it wasn't a small one. Thanks for bearing with us!

Fixes and features kept rolling out on a regular basis, so most active community members were not affected by the lack of a fresh new "official" release. It did, however, become an issue for people new to Red, because as the stable download for 0.6.3 became more and more out of date, users would report bugs that had already been fixed! So now 0.6.4 should see us on the same page.

We'd like to promise we'll never go that long again between official releases, and that is our goal, but we hope you'll roll with the punches, as we do the same. The best way to keep up to date is to use the automated builds, so you're always current. Some people will want to stay with the stable version for various reasons, and that's fine too.

This new Red release contains 1'116 commits, closes 167 issues tagged for this milestone, and adds a lot of  small features. Looking at the list of bugs fixed, and features added, is a great reminder of all Red has to offer.

1. Simple GC

The main feature for 0.6.4 is what we call the Simple GC (Garbage Collector). A more advanced GC is planned for the future, but this release will make it easier to create long running processes, without having to be aware of all your allocations, or manually reusing series to conserve memory. Things just got a whole lot easier!

There are 2 functions related to this, that you can use:

1. You can use stats to see the amount of memory in use, and its /info refinement provides more details to aid deeper debugging if needed.

2. The second function is called recycle, which triggers a manual garbage collection pass, rather than waiting for internal heuristics and timing events to do so. Recycle has 2 refinements: /off and /on. They allow you to disable the garbage collector, and re-enable it. This gives you control in scenarios where you want to manage things more directly yourself.

It's also important to note that the GC operates at the level of Red values, and does not affect Red/System code. Red/System is a C level language, where you are responsible for allocating and freeing memory, and where you may need precise manual control over those actions. (It's possible to add an optional GC for Red/System in the future, but it's not currently on our roadmap.)

2. Simple Isn't Always Easy

Our main goal is to fight software complexity, but nothing can eliminate it completely.

Our small size belies the number of details that have to be considered, interactions between the large number of datatypes, and how to make each most useful with the standard library of functions. And then there's the reactive GUI system, with its draw dialect, both very flexible and powerful, which leads to more combinations that need to be handled, and edge cases to uncover. We accept the effort needed, because we are our own toolchain. A great deal of work went into the new GUI console, with a lot more planned.

Apple's talk of eliminating all 32-bit support could impact us. We are currently 32-bit only, and need to decide the best path for adding 64-bit support, should we elect to do so. libRed is also kept up-to-date in this release as well, for integration from other tools and languages. Additional improvements were made in the ELF format emitter.

We still face being flagged by a few big AV products, and these false positives hurt our image. It's frustrating to be marked clean one day, then have their heuristics change and flag you the next. Keep reporting the problem to them, and check VirusTotal to see how they compare. It's just one more challenge, because we're small, and use our own toolchain, so they don't know what to do with us.

Not all changes are big, and a lot of work also goes into documentation and tests on a regular basis. The code is only part of the project. A lot of our time this past year has been spent on logistics related to business, and making the Red project sustainable long term.

2.1: Operating System Support

Providing the maximum number of features across platforms is not easy, and we will need to freeze support for older OSs, like Windows XP, so we can focus efforts on the systems used by more people today.

Keeping up with mainline changes on all the operating systems we support is a challenge itself. We've kept ARM soft float working, and FreeBSD support has remained largely intact. We got a report about it breaking recently, and a fix went out the next day. FreeBSD isn't critical for us, but that fast response shows how a smaller and simpler code base can help.

2.2: A REPL In Time

Having an instant-on REPL is enormously helpful to newcomers and experts alike, and we'll build on this new foundation to add more features.

One change we expect to make for 0.6.5 is to change the download binary, so REPL use is even easier. Right now, when you download Red, you get the toolchain. The first time you run it, it builds the REPL console on your system locally. That takes some time, and isn't a great first user experience. So as we move forward, we're considering making the REPL the download, and then when you go to compile, if Red sees you haven't installed the toolchain, it will download it at that time. We'll keep you updated as we try our alternatives.

3. Rich Text

There's a new style in fashion this season: rich-text. 

Full documentation is forthcoming, but you can see the source code for the RTD (Rich Text Dialect) in %red/modules/view/RTD.red (only ~150 lines), and reference information in the wiki.

Eight attributes are currently supported. More can be added, but these gives you a great deal of control, and are by far the most commonly used: bold, italic, underline, strikethrough, text color, background color, font-name, and font-size. Border is planned but not yet implemented, as are some advanced options for the above attributes (e.g. wavy underline as is often used for spell checking markup). Attributes can be nested, of course.

@toomasv put together a small editor, which is about 400 lines of code, but since the rich-text style doesn't have built in keyboard handling yet, he had to roll his own, which is the majority of the code. 

Using the rich-text style directly in code is easy too. The data facet just needs to contain RTD commands and, internally rtd-layout is called to render those commands into the text and data facets of the face. You can also create a rich-text face, and later call rtd-layout/with to update commands and render them.

While Red prefers plain words for markup elements, RTD also supports HTML style tags, for convenience. (Note, RTD is not designed to be HTML compatible, and it's not a goal on our roadmap.) 

You can see different approaches to formatting, below: the first three examples all produce the same output, and show how attributes can be applied to blocks of commands, which is more idiomatic in Red. That is, when you have a formatting command followed by a block, everything in that block has the formatting command applied. It makes viewing the formatting structure much easier, and also eliminates attribute ordering issues. 

The last example below shows how a group of attributes can be expressed as a single path! value, applying all of them to the block that follows.

    view [
        rich-text data [
            <i> <b> "Hello" </b> <font> 24 red " Red " </font> blue "World!" </i>
        ]
        rich-text data [i b "Hello" /b font 24 red " Red " /font blue "World!" /i]
        rich-text data [i [b ["Hello"] red font 24 [" Red "] blue "World!"]]
        rich-text data [i/b/u/red ["Hello" font 32 " Red " /font blue "World!"]]
    ]

This is only the most basic use. See @toomasv's example gist for more uses, and consult the wiki doc to see how to query information like line height and caret position. Beyond that rich-text is already hardware optimized and supports multiple rendering areas in a single face, which makes it suitable for quite complex layouts of richly formatted text. You can also combine draw commands in a rich-text face, for even more power.

4. GUI Console: More Than Meets the Eye

While the GC took the spotlight in this release, the console also saw huge changes and improvements. The biggest change isn't visible to the naked eye: the new GUI console is now implemented in pure Red, using the rich-text face style. This sets the foundation for building more complete Red editors, tools, and IDEs. Basic features in place right now:

• History and Undo/Redo
• Copy/Paste support
• Hot-keys and shortcuts
• Syntax Coloring
• Auto-completion

4.1: Console Hot Keys

• Ctrl + Backspace  delete previous word
• Ctrl + Del        delete next word
• Ctrl + Up/Down    scroll up/down
• Ctrl + C          copy
• Ctrl + V          paste
• Shift + Insert    paste
• Ctrl + X          cut
• Ctrl + Z          undo
• Ctrl + Y          redo
• Ctrl + L          clear screen
• Ctrl + K          delete the whole line

4.2: Syntax Coloring

You can turn syntax highlighting on with the following command:

    gui-console-ctx/terminal/color?: yes

It is not dynamic, so you may find the colors don't work well with the colors you've selected in the Settings dialog for the console. There is no support for customizing themes yet. If you want to experiment with changing colors, you have to manually set values in gui-console-ctx/terminal/theme.

4.3: Auto-Completion

This feature can support completion for 3 types of values: words, files, and paths. 

For words, if you type a word and hit the Tab key, one of two things can happen. Either there's a single match for what you've typed so far, which will then be auto-completed for you, or all matches will be shown.

Try typing p <tab>. You'll get a list of all words in the system that start with the letter P.

Now, type pa <tab>. The help function works in a similar way, but with different output.

Filename completion works the same way. If you type a file value, starting with %, the letters you type are then prefix matched against the current directory in the file system. It works very much like a system shell's autocomplete feature, but built directly into the Red console. Snazzy!

Path completion is a little different:
If you type a path, e.g., system/, and then hit the Tab key, the console will display the list of words available for that path value.
If the value is an object, the words of that object will be displayed.
If it's a function, the possible refinements will be listed.

For objects, it makes it easy to drill down into nested structures. For functions, it offers basic help, but doesn't do anything advanced yet. For example, if you use more than one refinement, auto-completion will still list every refinement available, rather than removing those you've already used from the list.

5. Other Highlights & Changes in 0.6.4

Here are just a few more details:

5.1: View and VID

• DPI support added, which is great, but also raises new OS system scaling questions, currently being reviewed
• rich-text style added
• Password mode added to field style (not supported in VID yet); include 'password in face flags
• Make-face helper function added
• RTD (Rich Text Dialect) support (MacOS & Windows)
• Ctrl+A hot-key works in area now
• Text selection support improved for area and field

5.2: Core Language

• Added /deep and /case refinements to replace
• Simple I/O now supports all HTTP verbs
• Added decompress native, with support for gzip, zlib and deflate formats
• Adler32 support added to checksum
• Maps now support binary! values as keys
• Latin-1 strings can now be decoded
• Trim now supports binary! values. By default, only null values are removed. If you use the /all refinement, whitespace chars are trimmed as if it were a string
• Basic query action added for files, which returns their timestamp
• Added offset-to-char function. Given a face and a coordinate, it returns the corresponding character position
• Added os-info func. It returns detailed operating system version information, which can be helpful when debugging and submitting tickets (used by about/debug)
• Very simple alert function added; more are available as community experiments
• Enbase/Debase now support base 58
• Sum and average functions added

6. Looking Ahead

• Full I/O is our next major milestone, and some other key features, like async I/O and a networking port model, will come with that.
• Android: We've changed up the roadmap a bit, so Android will come after Full I/O, because it will benefit from having it, and save rework if done the other way around. This is a very important -- dare we say, critical? -- milestone, and one we will put all our efforts behind in the coming months.
• Blockchain Aspects: The RED Wallet is expected to get some new features, and possibly a new look. We need a dedicated product manager for it, and another team member who can focus on it. The C3 project is going full steam, and we'll be organizing resources to keep that moving as we redirect key players to Red's core early next year.
• We have an experimental money! datatype, based on Douglas Crockford's Dec64 design. It will have a limited value range initially, due to our 32-bit foundation, but it's a first test, and can be used to see how it works in action. We believe we're the first language to implement it.
• JSON and CSV codecs. These are works in progress, and their design had led to discussion of how best to do certain things outside them -- for example, how to represent JSON null values in Red maps, which isn't yet possible.

We're enormously proud of how Red's growth and community support have accelerated. For a project like ours -- ambitious, idealistic, open-source -- to have garnered interest in the way we have, has been greatly rewarding. We look forward to a 2019 rich with REDucing software complexity, yet celebrating the increasing power and impact of the language and the movement.