GTK, fast lexer, money, deep testing, and our first commercial product

It's been a busy start to the year for the team. Work has continued on many fronts, and we have some new team members helping us to keep the momentum going. We announced in January that GTK and the Fast Lexer were close, and they are even closer now. The hard part about making announcements is that some of this work is unpredictable and changes in scope after we do. Or the world steps in and a pandemic throws a wrench into your plans.

@bitbegin has done an enormous amount of work on the GTK branch. You can check out the GTK branch to see some of what goes into supporting a new GUI system. The more features we include in Red, the more that have to be ported and maintained. Unfortunately, most operating systems and UI systems have large, complicated sets of APIs and interactions.

Because GUI systems are so complex, and Red not only has to handle them, but also adds its own reactive framework, there are more places for bugs to hide. And users are involved, which is the worst part. They do nothing but cause problems. To that end, in addition to his normal deep diving and bug hunting, @hiiamboris has been working on an automated view test system, which is no small feat. @9214 has joined him on the hunt, and we will squash a good number of bugs for our next release.

The fast lexer was in near-final testing when we decided that it was worth delaying its merge in order to incorporate some new lexical forms that we had planned to include. Then we looked at some old tickets related to modulo and division operators, and a couple lexing questions came up related to tag!. Suddenly the fast lexer work was back in code mode. New lexical forms usually means new datatypes, and that's the case here.

New Datatypes

One we've expected for some time, and thanks to @9214 it's now a reality. Money! is coming. There is a branch for it, but no need to comment at this time. There are a few features, like round still to be completed, but the bulk of the work is done. @BeardPower did some great experimental work that we thought might be used for money, based on Douglas Crockford's Dec64 design, but until Red is fully 64-bit it was only Dec32, and the limited range was determined not to be enough. That work won't go to waste though, it's just waiting for its moment in the sun. The current version of money! is a BCD implementation, but that shouldn't concern anyone outside the core team. What you care about is that it can be used for accurate financial calculations that don't suffer from floating point errors. It will also support an optional currency identifier, e.g. USD or EUR, and automatic group separators when formed.

Another new type is called ref! and is still being designed. The basic concept is simple: @reference is a form most people are familiar with today, just as hashtags are known (though we use the historical name issue!, because the new name hadn't become part of our global lexicon back then). Issues, in Rebol 2, were a string type, but Red made them a word type instead. That has benefits (mainly efficiency), but also costs (symbol table space and lexical limitations). For instance, in R2 you could say #abc:123, but not so in Red. Life is compromise. Ref! will be a string type, making it quite flexible. While we most often think of them as referring to a person, they can refer to, for example, a location in a file. You can do that with strings too, of course, but that's the beauty of rich datatypes. By using a ref!, you can build rich dialects and make your intent clear in the data itself.

Finally, Red is going to add a new raw-string! form. It's a combination of the raw string literals some languages support, and heredoc. The goal is to make it easier to include content that would otherwise require escape sequences that sometimes clutter inline data and lead to errors. A lot of time and effort went into the (sometimes heated) discussion around the need, use cases, and syntax. Right now this is just a new literal form, rather than a whole new datatype. They will still be strings when loaded, until we see how they are used by others and if they deserve to be a separate type.

We don't add new datatypes lightly, and design choices have to keep the big picture in mind as well. Balancing the value of new types against their added complexity in the language is hard work, but satisfying if it makes everyone's life better. To that end, when these new types and features become available, we need your input on how they work, where you use them, and what gaps need to be filled.

Both the fast lexer and money datatype leverage some new features in Red/System, which we'll talk about in a future post.

All Aboard!

We're also very excited to announce the imminent release of our first commercial product. We alluded to it at the beginning of the year, and it's almost ready to leave the station. There will be plenty of train-related puns, because it's a Railroad syntax diagram generator. Here's what it looks like:

More details will come soon, but here's a short list of some of it's features:

• Live coded diagramming. You write your grammar and the diagram appears in real time.
• Red parse support (of course), but also support for foreign grammars like ABNF, McKeeman, and YACC. That's right, you can write using those grammars and get the same diagram. 
• When writing parse rules, you can use block level parsing, rather than character level.
• Test inputs. Not only can you test a single input, but also entire directories of test files.
• Test specific rules. Not only can you test from your top-level rule, but any rule in your grammar. You can also find where a particular rule matches part of your input.
• Custom diagram styles, including options for a cleaner, more abstract view, different charset rendering options, and whether actions (parens) are displayed.
• The ability to generate inputs that your grammar will recognize. This opens up many other use cases, including educational ones. Here's one of my favorites, that @toomasv created:

Look at the sentences in the input area. Those were created by clicking the generate button. It's like parsing in reverse. So if you're not sure what inputs your grammar might recognize, or want to see examples, this lets you view things in a whole new way.

Stay Tuned

It's an exciting time to be a Reducer, and we're rolling with changes just like everyone else right now. The best way to stay healthy during this pandemic is to stay away from other people and take the Red pill.

Until next time.