Long Time No Blog

 It's been almost a year since our last blog post. Sorry about that. It's one of those things that falls off our radar without a person dedicated to it, and we run lean so don't have anyone filling that role right now. We know it's important, even if we have many other channels where people can get information. So here we are.

Last year was a tough year all around, even for us. We were already a remote-only team, but the effect the pandemic had on the world, particularly travel, hit us too. We had some team changes, and also split our focus into product development alongside core Red Language development. This is necessary for sustainability, because people don't pay for programming languages, and they don't pay for Open Source software. There's no need to comment on the exceptions to these cases, because they are exceptions. The commercial goal, starting out, is to focus on our core strengths and knowledge, building developer-centric tools. Our first product, DiaGrammar for Windows, was released in December 2020, and we've issued a number of updates to it since then. Our thanks to Toomasv for his ingenuity and dedication in creating DiaGrammar. We are a team, but he really accepted ownership of the project and took it from an idea to a great product. Truly, there is nothing else like it on the market. 

We learned a lot from the process of creating a product, and will apply that experience moving forward. An important lesson is that the product itself is only half the work. As technologists, we're used to writing the code and maybe writing some docs to go with it. We don't think about outreach, marketing, payments, support, upgrade processes for users, web site issues, announcements, and more. The first time you do something is the hardest, and we're excited to improve and learn more as we update DiaGrammar and work on our next product. We'll probably announce what it will be in Q4. One thing we can say right now is that the work on DiaGrammar led to a huge amount of work on a more general diagramming subsystem for Red. It's really exciting, and we'll talk more about that in a future blog post.

So what have we been doing?

Since our last blog post we've logged over 400 fixes and 100 features into Red itself. Some of these are small, but important, others are headline-worthy; some are deep voodoo and some visible to every Reducer (what we call Red users). For example, most people use the console (the REPL), so the fixes and improvements there are easy to see. A prime feature being that the GUI console, but not the CLI console, didn't show output if the UI couldn't process events. This could happen if you printed output in a tight loop. The results would only show up at the termination of the loop, when the system could breathe again. That's been addressed, but wasn't easy and still isn't perfect. Red is still single threaded, so there's no separate UI thread (pros and cons there). We make these tradeoffs every day, and need feedback from users and real world scenarios to help find the right balance. Less obvious are things like improvements to parse, which not everyone uses. Or how fmod works across platforms, and edge cases for lexical forms (e.g. is -1.#NaN valid?). The latter is particularly important, because Red is a data language first.

JSON is widely used, but people may not notice that the JSON decoder is 20x faster now, unless they're dealing with extremely large JSON datasets. JSON is so widely used that we felt the time spent, and the tradeoffs made, were worth it. It also nicely shows one of Red's strengths. Profiling showed that the codec spent a lot of time in its unescape function. @hiiamboris rewrote that as a Red/System routine, tweaked it, and got a massive speedup. No external compiler needed, no need to use C, and the code is inlined so it's all in context. Should your JSON be malformed, you'll also get nicer error information now. As always, Red gives you options. Use high level Red as much as possible, for the most concise and flexible code, but drop into Red/System when it makes sense.

Some features cross the boundary of what's visible. A huge amount of work went into D2D support on Windows. D2D is Direct2D, the hardware-accelerated backend for vector graphics on Windows. For users, nothing should change as all the details are hidden. But the rendering behavior is not exactly the same. We try to work around that, but sometimes users have to make adjustments as well; we know because DiaGrammar is written in Red and uses the draw dialect heavily. It's an important step forward, but comes at a cost. GDI+ is now a legacy graphics back end, and won't see regular updates. Time marches on and we need to look forward. As if @qtxie wasn't busy enough with that, he and @dockimbel also pushed Full I/O forward in a big way. It hasn't been merged into the main branch yet, but we expect that to happen soon. @rebolek has been banging on it, and has a full working HTTP protocol ready to go, which is great. TLS/SSL support gets an A+ rating, which is also a testament to the design and implementation. It's important to note that the new I/O system is a low level interface. The higher level API is still being designed. At the highest level, these details will all be hidden from users. You'll continue to use read, write, save, load exactly as you do today, unless you need async I/O. 

Another big "feature" came from @vazub: NetBSD support. The core team has to focus on what stands to help the project overall, with regard to users and visibility. Community support for lesser known platforms is key. If you're on one of those platforms, be (or find) a champion. We'll help all we can, but that's what Open Source is for. Thanks for this contribution @vazub!

We also have some new Python primers up, thanks to @GalenIvanov. Start at Coming-to-Red-from-Python. Information like this is enormously important. Red is quite different from other languages, and learning any new language can be hard. We're used to a set of functionality and behaviors, which sometimes makes the syntax the easiest part to learn. Just knowing what things are called is a learning curve. Red doesn't use the same names, because we (and Carl when he designed Rebol) took a more holistic view. That's a hard sell though. We feel the pain. A user who found Red posted a video as they tried to do some basic things. We learned a lot from watching it. Where other languages required you to import a networking library, it's already built into Red. When they were looking for request or http.get, and expecting strings to be used for URLs, they couldn't find answers. In Red you just read http://.... It's obvious to us, but not to the rest of the world. So these new primers are very exciting. We have reference docs, and Red by Example, but still haven't written a User's Guide for Red. We'll get there though. 

Why do things take so long?

Even with that many fixes and features logged, and huge amounts of R&D, it can still feel like progress is slow. The world moves fast, and software projects are often judged by their velocity. We even judge ourselves that way, and have to be reminded to stay the course, our course, rather than imitating others. Red's flexibility also comes into play. Where other languages may limit how you can express solutions, we don't. It's so flexible that people can do crazy things or perform advanced tricks which end up being logged as bugs and wishes. Sometimes we say No (a lot of times in fact), but we also try to keep an open mind. We have to ask "Should that be allowed?", "Why would you want to do that (even though I never have)?", and "What are the long term consequences?" We have to acknowledge that Red is a data format first, and we never want to break that. It has to evolve, but not breaking the format is fundamental. And while code is expected to change, once people depend on a function or library it causes them pain if we break compatibility. We don't want to do that, though sometimes we will for the greater good and the long view. There are technical bandages we can patch over things, but it's a big issue that doesn't have a single solution. Not just for us, but for all software development. We'll talk more about this in the future as well.

I'll note some internal projects related to our "slow and steady" process:

• Composite is a simple function that does for strings what compose does for blocks. It's a basic interpolator. But the design has taken many turns. Not just in the possible notations, but whether it should be a mezzanine function, a macro, or both. Each has pros and cons (Side note: we don't often think about "cons" being an abbreviation for "consequences"). This simple design and discussion is stalled again, because another option would be a new literal form for interpolated strings. That's what other languages do, but is it a good fit for Red? We belabor the point of how tight lexical space is already, so have to weigh that against the value of a concise notation.
• Non-native GUI. Red's native GUI system was chosen in response to Rebol's choice to go non-native. Unfortunately it's another case of needing both. Being cross platform is great for Red users, but Hell for us. Throw in mobile and it's even worse. Don't even talk about running in the browser. But every platform has native widget limitations. Once you move beyond static text, editable fields, buttons, and simple lists, you're in the realm of "never the twain shall meet". How do you define and interact with grids and tables or collapsible trees? Red already has its own rich-text widget, so you don't have to embed (even if you could) an entire web browser and then write in HTML and CSS. To address all this, with much research and extensive use case outlines, @hiiamboris has spent a lot of time and effort on Red Spaces. Show me native widgets that can do editable spiral text, put any layout inside a rotator, or define recursive UIs. I didn't think so. Oh, and the wiggling you see in the GIFs there are not mistakes or artifacts, they are tests to show that any piece of the UI can be animated.
• Other projects include format, split, HOFs, and modules, each with a great deal of design work and thought put into them. As an example, look at Boris' HOF analysis. They are large and important pieces, based on historical and contemporary research, but not something we will just drop into Red, though we could. A simple map function is a no-brainer, and could have been there day one. But that's not how we work. It's not a contest to see how many features we can add, or how fast; but how we can move software forward, make things easier, and push the state of the art. Not just in technical features (the engineering part), but in the design of a language and its ecosystem.

Not Everyone Has These Problems

An important aspect of Red is being self-contained. We talk about this a lot. Yes, we're considering LLVM as a target, but that has a big cost, not just benefits. Using our own compiler for everything also has costs, like slowing the move to 64-bit which is an issue for Mac users now. Workarounds like VMs and Docker containers are just that. We want things to be easy for you, but that doesn't mean they're easy for us. Here's an example.

Boris found a bug related to printing time values in Red/System. @dockimbel finally tracked it down, and posted this investigative report:

@hiiamboris It was a (R/S) compiler issue afterall. ;-)size? a was the guilty part. The compiler was wrongly generating code for loading a even though size? is statically evaluated by the compiler and replaced by a static integer value. Given that a was a float type, its value was pushed onto the x87 FPU stack, but never popped. That stack has a 7 slots limit. Running the loop 5 times was enough to leave only 2 slots free. When the big float expression is encountered in dtoa library, it requires 3 free slots on the FPU stack, which fails and results in producing a NaN value, which wreaks havoc in the rest of the code.

The fix in the compiler was trivial (fetch-expression/final vs fetch-expression) but getting to that point was not. Understanding machine architectures at the lowest levels isn't for everyone, but even though our compiler code will be rewritten in the future, it's small and maintainable today. If we rely on GCC, Clang, or other compilers, hitting a bug may mean hitting a wall. So while there are costs to using our own compiler, there are also costs to depending on others. Robert Heinlein popularized  TANSTAAFL, but the concept is not science fiction. As a side note, just as we moved from GDI+ to D2D,  x87 for float support was an early choice meant to support older platforms and we are planning to switch to SSE.

If compilers are your thing, or you like system level programming, join our community and get to know Red/System. See how our toolchain works, and consider joining us.

The Big Picture

I just read 101 things I learned in architecture school (which I heard about via Kevlin Henney, (though it may not have been that specific talk) and what struck me the most about it is how we've commandeered the word "architecture" for software but completely removed the human aspect. An architect does so much more than we do. Software architects are really structural engineers. If a single developer builds a complete app, they have to do the UI. They engineer a building and then slap on whatever sheathing is at hand, cutting doors and windows without concern for their location. And the app is viewed in isolation, as if it's the only thing a user has on their system, without consideration for its site, context, or relationships. What makes real architecture hard (and why the author notes that many architects hit their stride when they are older), is that you have to know so much. So many considerations, disciplines, and constraints are involved, and you have to unify them. It's both creative and scientific. What makes great architecture great is that it makes your experience better. Maybe even wonderful. If we only think about the mechanical aspects, our software will never be beautiful.

We haven't articulated this view for what we do, I think because we didn't realize it. At least I didn't. We talk about the whole being greater than the sum of its parts, and not just making everything libraries so it feels more natural and less mechanical. How a REPL and single exe make it easier to get started, and not having to use many tools is better. But we haven't explicitly said "Here's how it's laid out, and why. Here's how it's put together; these are the critical elements. Here's what it looks like from a distance, and when you enter its space." Implicitly we do that every day, through the work, but we don't talk about it. Or only once a year.