Overview of Red development history

This video shows a fast-paced timelapse of the Red source code changes in the main repository over last 6 years, 9000 commits. You can see the avatars of all contributors, flashing commits on the files over time.

Here are a few stats showing the changes between Jan 1st 2017 and Dec 31th 2017:

• Github stars: 1214 to 2970
• Github programming languages ranking: #34 to #22
• Page views on red-lang.org: 1.06M to 1.65M

As you can see, 2017 has been a great year, with interest in Red growing for some stats, faster than in all the previous years put together!

Short history of Red

Given that many newcomers arrived in Red community in the last two years, some might be confused about past Red history and timeline, so I would like to write it down now, and add some comments in retrospect on some of the choices made in the past.

The Red project was officially launched in March 2011, with the first public announcement and the release of the version 0.1.0, which was developed in 3 months. That early version provided a fully working Red/System toolchain, with a compiler and linker, all in 60KB of source code. That was just for Red/System, the low-level DSL of Red.

The Red language had different goals at that time, it was meant to be a statically typed sub-set of Rebol, with static scopes and no dynamic binding. That Red would have been only compiled, so no interpreter, and would eventually had a REPL after 1.0, once a JIT was available. Given my old work on an experimental Rebol clone in C a few years ago (implemented in 9 months), I estimated that a 1.0 for such Red could be doable in a year. Such a version would only support Windows and x86 architecture, would have most, but not all of Rebol datatypes, no DSL, no GUI nor floating point support.

During 2011, a community of users and followers started forming around this early Red version, the daily interactions with the community had some influence on the goals I had initially set for Red, and at some point during that year, I realized that what I really wanted was a more dynamic language, closer to Rebol, rather than a crippled static version. I decided then to trace a new longer road to a different design for Red, more in line with my personal desires and community expectations.

The scope extended as we kept adding new features (sometimes initiated by contributors), like ARM support, floating point support and Parse dialect. We then replaced the early low-level lexer with a much more capable version using Parse. The community was pressuring for a more "production-ready" version, which meant adding more features, like extra datatypes, postponing the 1.0 milestone further away up to 2014.

I have been working full-time on the project since 2011, and invested all my personal money to fund it, helped by individual donations from community members. Frankly, without those extra donations, I wouldn't have made it. In late 2013, I was contacted by a China-based VC (Innovation Works) who was interested by Red project. They offered me the chance to join them in Beijing. I initially politely refused, as I thought that doing so would consume too much time and accepting investments never has a guaranteed outcome. But thinking more about it, I also considered that raising a seed round would allow some talents in the community to work full time on Red, speeding it up. So, I accepted the offer and relocated to Beijing in 2014, settling for a new life there.

Red in its command-line form was not good enough to attract any investor, nor make any good business plan. So I made a major decision to incorporate the GUI module in the roadmap as soon as possible. The original plan was to only add it after Red had reached version 1.0. I also thought that Android support would be a key element for Red's future, so I also move it forward too. It took me about 7 months to raise a seed round, and 4 more months to effectively get the money into the bank account of the newly formed company in mid-2015. Work on Red during that time went more slowly. It never stopped though as we could hire our first developers (using a loan) and the number of contributors from the community was increasing.

A new roadmap was set and a 1.0 was planned for 2016 focusing on Android GUI support. Though, the early prototype on Android showed that developing a cross-platform GUI engine starting with Android (as we wanted it asap) along with the VID DSL was not a good way to achieve it, as the development cycle with Android was too long, and debugging at low-level there was very limited and painful. It was way more convenient to do it on a desktop OS like Windows, which was also the platform of roughly half of our users. So we switched the GUI development to Windows first, as it would be much faster to design and polish the cross-platform parts there, then make an Android back-end for it after.

The GUI engine with the VID dialect and back-end for Windows was released in early 2016. We initially planned only a minimal support and just a few (5-6) widgets for the Windows back-end, but the community wanted more (and still does). At the same time, we were also working in parallel on a Red IDE engine based on Scintilla. We needed more widget support, so we spent extra months doubling the initial number of widgets and polishing the features of VID, Draw dialect and View engine. The Scintilla IDE prototype was not as promising as we expected, some limitations with Scintilla were disappointing to say the least. They included awkward Unicode support and UI limitations. A Red GUI based IDE was a better option, allowing us to support any feature we could think of. We ran out of resources for such a side-project, so we put it to sleep until we can fund it (hint hint, the ICO would help there a lot too!).

After the initial GUI release, before we went back to Android support, we had to do some promotion about Red and our new shiny GUI system, in order to increase our userbase  (growth is a constant concern for VC-backed startups). Though, the competition was rough, as the new state of the art (on the web) was reactive frameworks which were providing a more powerful and simpler way to operate GUIs. Such reactive GUI was also something I had on my wish-list since 2010 for Red, and it seemed like the right time to add it, so that's what we did and released it in mid-2016. That enabled the production of some pretty impressive demos I think, for a cross-platform native GUI system.

We went back to Android development after that, starting by adding a preprocessor to Red lexer, in order to move a lot of costly work done at runtime to resolve Java method signatures, to compile-time. This work consumed around a couple of months, especially as we also took that opportunity to add an experimental macro system. The preprocessor is currently used by the Red runtime to reduce code size, though, we late figured out a better way to handle the Java method signatures  without relying on the preprocessor. So, we could have saved some time there, even if it's always easier to come to such conclusion when looking back in time.

As we continued on Android support, we stumbled upon new unexpected issues. All the features added in the last releases significantly increased the runtime size, resulting in very long compilation times. Moreover, the GUI system and newest object features broke the ability for the Red compiler to produce functioning shared library files for Red programs (Red/System shared libraries were fine, it was a limitation of the Red runtime library and compiler). We then decided to launch the "libRed" branch, in order to solve all those issues by enabling the compilation of the Red runtime as a shared library in two different forms: one is libRedRT, used only for speeding up the compilation process, the other is libRed, which provides an external API for embedding Red into external software (which was on our todo-list anyway and in the Red initial goals). That work took more time than expected, roughly 7 months, as I had to travel across the globe many times (for personal reasons) going offline for a total of 2 months during that period, which made it harder to properly focus on some of the complex issues that needed solving. That is where we lost the most time in the last couple of years, although the macOS GUI branch was still advancing at the same time in parallel.

After macOS release earlier this year, we could finally continue the work on Android, and guess what, we could develop a GUI backend on par with Windows one in about a month and half only! All the preparation work done before was paying off in the end! Still, that is not enough for acceptable Android support, as mobile devices have many features which need first-class support, and we are working on them for 0.6.5 release (delayed by the ICO preparations and process). We will provide more info about Android support very soon.

Another important consequence of adding GUI support for macOS, was the need to have a GUI console there too (identical to the Windows one), as we have great plans for the GUI console itself. As a side-project, we started working on a new GUI console engine in early 2017, replacing the low-level OS-specific Red/System bindings with the Red/View engine, enabling cross-platform support out of the box, and adding some key features we needed on the Red roadmap, like rich-text support. Those features are available in our 0.6.4 branch, and will be part of the next release shortly after the ICO (though probably after Chinese New Year in february).

One thing I have greatly underestimated during all those years (especially at the beginning), was the amount of time that would be spent interacting with the community, as I tried to be as present as possible online to answer questions, welcome newcomers and help them get started. My public posts are just the tip of the iceberg, as many people just contacted me on direct channels (email, private messages on AltME, Gitter, skype, ...), and that consumed a lot of time too. But, I think it was a necessity anyway, just unaccounted for. Nowadays it is much easier for me, as we have community managers and many other members who are doing a great job handling newcomers, wave after wave. Another thing I didn't consider at the beginning was also how much impact and influence the community would have on the features that we would put into Red (especially pre-1.0). As old-timers from Rebol community were used to have a great amount of feature available out of the box, they expected Red to be on par with that, often without much concern about the costs involved in supporting the desired features. I think I should have said "no" more often than I did, or just postponing them after 1.0.

All in all, despite the bumpy road sometimes, the current version of Red is better in many ways than what I hoped for in 2011. We still have the option to rename the 0.7.0 or 0.7.1 upcoming milestones to a 1.0 candidate, as they would have all the features we initially planned for a 1.0, though there are pros and cons about pushing the other planned features post-1.0, but I understand that Red users nowadays are more interested in reaching a 1.0, than having a more feature-complete one. We can discuss about that on our Gitter channels.

This is a (very) short version of the Red history, there is so much more to say about it, maybe someday when I will have some spare time, I'll write down the whole thing in a book, including my views on language design, and all the non-tech related side stories. ;-)

Cheers!