Happy New Year!

Hello and happy new year, friends of Red! We have some exciting projects we’ve been working on that will be available this year, including a new product. Let’s talk a little about what the team has been working on behind the scenes.

(TL;DR: A cool new product with Red in 2020...plus, a robust preliminary draft of Parse documentation can now be previewed...CLI library...fast-lexer to merge soon...GTK on the horizon...and a new native OS calendar widget!)





Documentation for Parse: Red’s Language Construction Tool
Our esteemed forerunner, Rebol, broke new ground with its Parse dialect, which Red has expanded on. Today, in Red, Parse has become an even more powerful built-in dialect (embedded domain-specific language) that processes input series with grammar rules in a clean and simple manner. Other language building tools exist, of course, such as Lexx and Yacc. ANTLR is a modern framework built in Java, and libraries exist for other languages, but the ease of use and power that Red’s Parse offers is unique.

This isn’t a new feature, by any means, with the first public introduction here. Parse is easy enough to use that those basic docs have been enough, for the most part. But over time, with new users joining the Parse chat room to discuss the dialect, and it was time for full reference documentation.

For the uninitiated, Parse can be used to for searching, to surface various patterns; validation, in order to confirm an input’s compliance to a specification; extraction, to sift through data and aggregate values; and modification--that is, changing the input stream itself (insertion of values, removing or transforming matched input). And Parse’s true power lies in language processing (compilers, interpreters, and lexical analyzers), particularly for DSLs; and encoding/decoding, to “translate” data formats from one to another.

What makes Red’s Parse dialect a killer feature is that you can do more than parse at the character level in strings of text. That's what every other tool out there does, at best. But with Parse, you can process blocks of Red values, based on their datatype or specific literal matches, letting Red do all the work applying these layered or nested levels of analysis. By parsing blocks of values rather than character strings, we take language construction to the meta level.

We’re happy to announce that @9214 has completed a robust preliminary draft of Parse documentation, which you can see here. Internal reviews by the core team are in process and public comment will be made possible after more analysis. In the meantime, you can view Parse itself here, and look at open issues there if you’re interested in contributing.

(As a final note, the first question people often ask is how Parse compares to regular expressions. Both are DSLs, to be sure, but regex is designed for flat pattern matching of text inputs. Formats with nesting, things like matching parentheses, etc. are difficult or impossible. Engines differ widely, as well (NFA vs DFA), so your code may have very different performance in different environments. Not all regex syntaxes are compatible either. But the real difference comes in readability. Parse is more verbose, but is very clear, readable, and composable. Regex patterns are not. For fun, look up the regex to match email addresses, written by Jeffrey Friedl, in his book Mastering Regular Expressions. It’s a 6.5Kib regex. Doesn’t seem to be online anymore, but is referenced in this chat. For more discussion on this specific area, https://emailregex.com/ points to https://www.regular-expressions.info/email.html about why there’s no perfect regex for email addresses. The point is not to criticize regular expressions. They are useful, especially in the more limited forms of wildcard matching and file globbing. But they can’t recognize all grammars, and quickly become impossible for a human to read and reason about.)

Fast-Lexer To Be Merged Soon
Over the last few months work on the fast-lexer branch of Red has accelerated, beginning with @dockimbel prototyping the next lexer in R/S, faster than Red’s predecessors, with much smaller and simpler code, and more speed and features to come. Issues with the current lexer had brought the need for a revision: this included the goal of making the lexer instrumentable. The requirements of a callback function with the current non-re-entrant lexer could have caused serious problems.

Another hangup? An analysis of all Red's repos showed that the current lexer code was duplicated twice, beyond the basic lexing needed by load: once in the console code, once again in the VSCode plugin. The redundancy had to be eliminated. A merge of the updated lexer into master should be expected early in 2020.

New Year, New Projects! 
In 2020, Red’s commercial arm will release its first product! It will be helpful to CS programs and their students, researchers, and anyone interested in analyzing the grammar and syntax of languages—which is all of you, since you're here. But it is meant to have much broader appeal as well.

Every developer has probably see json.org, which brought railroad syntax diagrams to everyone’s attention. That’s right, syntax is for data formats too. But why would Douglas Crockford go to all the trouble of creating those diagrams? Communication. Visualizations create a different experience for readers, facilitating new understanding and discussion options. This applies to those designing rich data structures (think GraphQL) and APIs, as well as languages. If you want to look professional, foster adoption, show that your design is formally specified, and help people understand it, this tool is for you.

It does a lot more than create fantastic diagrams, too, to help spec and test your grammar. It’s in invite-only beta test now, but we’ll need a limited number of testers soon, so contact us at pr@red-lang.org if you’re interested.

Some of you may ask: Why is it a commercial product? Because we’re a multifaceted organization, and Open Source doesn’t pay. ;^) The Red Foundation supports the open source work done on the Red language, as well as sponsoring other work that promotes Red’s use and adoption. They’ve been mainly focused on supporting Team Red and core language work, but will expand their reach as resources allow. The first step in that direction was to get behind proving Red’s commercial viability. Redlake Technologies is a new business entity that will use Red to build tools the world needs, and will pay for, with the goal of creating a self-sustaining ecosystem of applications, services, and tokenomics that benefit both the contributors to the language and the world at large, working toward a better model of sustainable open source development.

By Your Command (Line)
Also in the works: prolific community member @hiiamboris has been burning the midnight oil creating a fantastic CLI library, which will enable authors to accept and interpret arguments and define their types easily, in an expectable, standardized manner. Command lines are the API through which a program exposes its facilities to the outside world--to users and utilities. If you’d like to check it out, look at the repo and see how it works! By leveraging Red’s own function spec DSL we can, with this CLI library, turn any Red function into a command line utility with one or two words.

Suppose you have a `grep` function written in Red, that is to be used from the command line. Prefix it with `process-into` (the working CLI function name), add `print` for the output, compile it, and you have a standalone `grep` utility! Your feedback is important, because this design for a CLI library is quite unique.

As with Red itself, you may need to let go of old ideas about how CLI libraries like `get-opt` work (though you could recreate its model easily with what Boris has built), and think like a Reducer. Just write a function with parameters, refinements, types, and doc strings, and it does the rest.

GTK on the Horizon 
The GTK back end is nearing completion, with just GUI console support as the last big piece to be done. @bitbegin has done an enormous amount of work, with help, consultation, and testing from @loziniak and @rcqls. Once the GUI console is in place, GTK should have feature parity with MacOS.

Just In Time… 
The New Year will see one more surprise, in the form of a calendar widget with support for Windows and Mac. It’ll just show up one day--a native OS widget joining you in the latest automatic build--so keep your eyes peeled.