Dynamic Refinements and Function Application

It's Time to Apply Yourself to Red

Red has never had an apply function, though we knew it would come someday. In the meantime, some of us rolled our own. Gregg's was simple, neither flexible nor efficient, and just a couple lines of code. Boris made a much more capable version, but it could still only be so fast as a mezzanine. R2 had a mezz version, which suffered the same problem. All that changes now. Apply is dead! Long live Apply! It required deep work, and a lot of design effort, but we think you'll like the results, whether you're a high-level Reducer, or anxious to see how much leverage you can, um, apply, from a functional perspective. Everybody wins.

If you don't know what apply is, in terms of functional languages, take a moment and read up. If you can get through the introduction there without getting dizzy, great. If your head is spinning, feel free to stop after the first section of this article and ignore the deep dive. You still get 90% of the value for most high level use cases. Gregg got so dizzy that he fell down, but was still able to help with this article.

Function application is largely about composition. How you can combine functions in a concise way for maximum leverage and minimum code. The problem with its design in many languages is that it makes things harder to understand. Rather than concrete functions names, there is indirection and abstraction. It can be tricky to get right, especially in a flexible language like Red, while also maintaining as much safety as possible. You can drive fast, but still wear your seat belt.

Dynamic Refinements

This subtle feature is likely to see wide use, because it will reduce code and let people build more flexible functions. It's also easy to explain. Here's an example. First, how you would write it today:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    either only [
        append/only series reduce :value
    ][
        append series reduce :value
    ]
]

With dynamic refinements, it can be done like this.

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    append/:only series reduce :value
]

In case you missed the subtlety, it's :only being a get-word in the path. That's right, it's evaluated, rather than being treated literally, just like you use in selector paths. The value for the dynamic refinement is taken from its context, which can be a refinement in the function, or a local value. It can be any truthy value to use the refinement, and it is only retrieved not evaluated. That means you can't use a computed value directly, which makes it safer. Other than that, paths work just as they always have. If a refinement is used, fixed (literal) or dynamic, which can be mixed however you want, any arguments it requires will be fetched and used. Otherwise they are silently ignored, so you don't have to clutter your code or worry about what a dynamic path expression will consume.

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
    /dup  "Duplicate the value"
        count [integer!]
][
    append/:only/:dup series reduce :value count
]

>> only: false  dup: false  repend/:only/:dup [] [1] 3
== [1]
>> only: true  dup: false  repend/:only/:dup [] [1] 3
== [[1]]
>> only: false  dup: true  repend/:only/:dup [] [1] 3
== [1 1 1]
>> only: true  dup: true  repend/:only/:dup [] [1] 3
== [[1] [1] [1]]

This is an incredibly exciting and powerful feature. It's a shame there isn't more to explain. ;^)

Function Application

Functional Programming has never yet become mainstream, though it has periodic rises in popularity and a devoted following in many language camps. Even Red. Yes, Red is a functional language. It's not a pure functional language, because functions can have side effects, but functions are first class values and can be passed around like any other. It lets you do things like this:

>> do-math-op: func [
    fn    [any-function!]
    arg-1 [number!]
    arg-2 [number!]
][
    fn arg-1 arg-2
]

== func [fn [any-function!] arg-1 [number!] arg-2 [number!]][fn arg-1 arg-2]
>> do-math-op :add 1 2
== 3
>> do-math-op :subtract 1 2
== -1

That's called a Higher Order Function, or HOF. It also means you can return a function as a result.

>> make-multiplier: func [
    arg-1 [number!]
][
    ;; We're making a function and returning it here.
    func [n [number!]] compose [(arg-1) * n]
]

== func [arg-1 [number!]][func [n [number!]] compose [(arg-1) * n]]
>> fn-m: make-multiplier 4
== func [n [number!]][4 * n]
>> fn-m 3
== 12

That's all well and good, but what if you want to call different functions that take a different number or type of arguments? Now it gets tricky, and inefficient. Because Red uses free-ranging evaluation (function args are not contained in a paren or marked as ending in any way at the call site), how do you handle different arities (number of arguments)? Here's a very simple apply mezzanine:

apply: func [
    "Apply a function to a block of arguments." 
    fn [any-function!] "Function to apply" 
    args [block!] "Arguments for function" 
    /only "Use arg values as-is, do not reduce the block"
][
    args: either only [copy args] [reduce args] 
    do head insert args :fn
]

So easy! The power of Red. But there is a cost. It's a mezzanine function, so it's slower than a native function, the args are copied or reduced, and then do is used to evaluate it. We can live with this kind of overhead for a great deal of Red code, but apply is a building block, and may be used in deep code where performance is important. You may also have noticed that the fn argument is any-function!, that means two things: 1) If you want to know the name of the function, the word that refers to it, too bad. You'd have to pass another arg for that. 2) Refinements. You can't use them with this model. And that limitation is a killer. For example, you could pass :append but not :append/:only. And there's no way you could have an /only refinement in your function and just pass that along. Until now. 

The Real Apply

Here's the new apply native that is now available in Red:

USAGE:
    APPLY func args

DESCRIPTION: 
    Apply a function to a reduced block of arguments. 
    APPLY is a native! value.

ARGUMENTS:
    func    [word! path! any-function!] "Function to apply, with eventual refinements."
    args    [block!] "Block of args, reduced first."

REFINEMENTS:
    /all    => Provides a continuous list of arguments, tail-completed with false/none.
    /safer  => Forces single refinement arguments, skip them when inactive instead of evaluating.

Notice that the func arg can now be a word! or path!, so you can use the name, or a path including refinements. That's right, the Dynamic Refinements feature explained above works with apply too. And having access to the name being used to call the function is enormously valuable when it comes to tracing and debugging. It's a huge win. 

One big difference is that all the arguments for apply are in a single block. Another is that you MUST include the on/off values for each dynamic refinement in the arg block, they DO NOT come from the environment (context). Compare this version to the one in the Dynamic Refinements section. Really, paste them into an editor and look at them side by side. 

>> only: false  dup: false  apply 'append/:only/:dup [[] [1] only dup 3] 
== [1]
>> only: true  dup: false  apply 'append/:only/:dup [[] [1] only dup 3] 
== [[1]]
>> only: false  dup: true  apply 'append/:only/:dup [[] [1] only dup 3] 
== [1 1 1]
>> only: true  dup: true  apply 'append/:only/:dup [[] [1] only dup 3] 
== [[1] [1] [1]]

; Refinement names in the arg block don't have to match the spec.
; You can use other names, or literal values. For example:

apply 'append/:only/:dup [[] [1] false false 3] 

a: b: false  apply 'append/:only/:dup [[] [1] a b 3]

It means you have to be more careful in lining things up with the function spec, in a different way than you're used to, but here's where you could use a computed refinement value, which may be useful for generative scenarios like testing. You can also see that both refinements are dynamic, so both need an associated on/off value in the args block.

>> only: does [random true]
== func [][random true]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/:dup [blk [1] only dup none]]
== [1 [1] 1 [1] [1] 1 [1] [1] 1 1]

But if you use a fixed refinement, it does not need the extra on/off value. In this example, /dup is always used, so there is no on/off value for it in the args, block, but its associated count arg has to be there, and is type-checked normally.

>> only: does [random true]
== func [][random true]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/dup [blk [1] only 2]]
== [[1] [1] 1 1 1 1 [1] [1] [1] [1] 1 1 [1] [1] [1] [1] 1 1 [1] [1]]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/dup [blk [1] only none]]
*** Script Error: append does not allow none! for its count argument
*** Where: append
*** Near : apply 'append/:only/dup [blk [1] only none]
*** Stack: 

But those aren't your only options. During the design of apply there was a lot of discussion about the interface(s) to it, and different use cases benefit from different models. For example, programmatically constructed calls means you need to build the path, and keep the args in sync. It may be easier to build a single block with everything in it, which you can do.

>> only: does [random true]
== func [][random true]
>> blk: copy []  loop 5 [apply :append [blk [1] /only only /dup no none]]
== [1 1 [1] [1] 1]
>> blk: copy []  loop 5 [apply :append [blk [1] /only only /dup yes 2]]
== [[1] [1] 1 1 1 1 [1] [1] 1 1]
>> blk: copy []  loop 5 [apply 'append [blk [1] /only only /dup no none]]
== [1 1 [1] [1] 1]
>> blk: copy []  loop 5 [apply 'append [blk [1] /only only /dup yes 2]]
== [[1] [1] [1] [1] 1 1 1 1 [1] [1]]

This interface is used if the first argument to apply is a function or lit-word, and /all is not used.

Apply/all

This is the most direct model, and what the others map to internally. In those models, you get friendly refinements, which may be optional, and those may have their own optional args. It's great for humans, and one of the best things about Redbol languages. But look at it from the view of a function spec.

>> print mold spec-of :append
[
    {Inserts value(s) at series tail; returns series head} 
    series [series! bitset! port!] 
    value [any-type!] 
    /part "Limit the number of values inserted" 
    length [number! series!] 
    /only {Insert block types as single values (overrides /part)} 
    /dup "Duplicate the inserted values" 
    count [integer!] 
    return: [series! port! bitset!]
]

The doc string isn't used when calling functions, so we can ignore that for this discussion. We can also ignore return: here. What's left is all the parameters (we often interchange arg and parameter, but there's a technical difference. Parameters are the named slots in a function spec, and arguments are the actual values passed in those slots). There are seven of those. Some are required args, some are refinements, and some are optional args. But there are seven slots, and when a function is invoked it expects there to be seven values on the stack that it could use if needed, or ignored if not.

When you use /all, you're telling apply that you are going to provide all those values in a single block, in the order of the function spec, like a stack frame (don't worry about the terminology too much if it's unfamiliar). Apply/all calls look like this:

1.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none false true 2]]
== [1 1 1 1 1 1 1 1 1 1]

2.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true true 2]]
== [[1] [1] [1] [1] [1] [1] [1] [1] [1] [1]]

3.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true false 2]]
== [[1] [1] [1] [1] [1]]

4.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none only false none]]
== [1 [1] 1 1 [1]]

5.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none 1 false none]]
*** Script Error: append does not allow integer! for its only argument
*** Where: append
*** Near : apply/all 'append [blk [1] false none 1 ]
*** Stack:  

6.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true true none]]
*** Script Error: append does not allow none! for its count argument
*** Where: append
*** Near : apply/all 'append [blk [1] false none true ]
*** Stack:  

7.
>> blk: copy []  loop 5 [apply/all 'append [blk [1]]]
== [1 1 1 1 1]

You can see that the refinement slots are now anonymous logic values in examples 1-3, but 4 uses only, our func from earlier examples, which randomly returns true or false. You can use anything that evaluates to logic! for a refinement slot. 5 shows that it has to be logic!, not just truthy, because types are checked (and logic refinement values then stand out against none argument values). And 6 shows that if you use /dup (second from the last arg), the count arg is also type checked, where 4 didn't complain because /dup was false. Confused yet? Look at 7. How can that work? I thought we had to fill all the slots! Yes and No. Apply "tail completes" the block with false/none values for you, if you don't provide enough arguments. Think of find, which has 16 slots. You only have to include enough args up to the last one you need. That may help, but if you need to use /match, the last refinement in find, you will have to provide all 16 args. Before you think this is unacceptable, consider our first repend example:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    append/:only series reduce :value
]

It would look like this:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /as-one "Appends a block value as a block"
][
    apply/all 'append [series reduce :value false none :as-one]
]

The point here is not to show that apply/all is longer, but that we can use a different name, where the first example must use :only in the path (make your own version to try it). Not all refinements will propagate using the same name. With /all it cares only about the logic value in the slot.

Apply/safer

/Safer is a form of short-circuit logic. Its purpose is to avoid the evaluation of unused args. Without it, everything in the args block is evaluated, but may be discarded if an associated refinement isn't active. The easiest way to explain this is with an example. 

; This is the function we're going to apply
applied: func [i [integer!] b /ref c1 /ref2 /ref3 c3 c4][
    reduce ['i i 'b b 'ref ref 'c1 c1 'ref2 ref2 'ref3 ref3 'c3 c3 'c4 c4]
]
; And some passive and active arg values
c: 0
bar40: does [4 * 2]
baz40: does [c: c + 1 456]

; Refinement args are evaluated
apply       'applied [10 "hi" /ref on bar40  /ref3 on  baz40            "ok"]

; No /safer difference because all refinement args are single values.
apply       'applied [10 "hi" /ref no bar40  /ref3 no  baz40            "ok"]
apply       'applied [10 "hi" /ref no bar40  /ref3 no  (c: c + 1 4 * 2) "ok"]
apply/safer 'applied [10 "hi" /ref no bar40  /ref3 no  (c: c + 1 4 * 2) "ok"]


apply       'applied [10 "hi" /ref no 4 * 2  /ref3 no  baz40            "ok"]
apply       'applied [10 "hi" /ref no bar40  /ref3 no  c: c + 1 4 * 2   "ok"]
apply/safer 'applied [10 "hi" /ref no 4 * 2  /ref3 no  baz40            "ok"]
apply/safer 'applied [10 "hi" /ref no bar40  /ref3 no  c: c + 1 4 * 2   "ok"]

In Real Life

Here are some examples of these new features being applied in the Red code base. The parse-trace example is jaw-dropping, not because it turns 9 lines into 1 (though, wow!), but because it makes the intent so much clearer and eliminates so much redundant code and the errors they can lead to. Not only that, it adds a capability! Before now you couldn't use both refinements together, i.e. parse-trace/case/part, but now you can.

Things we left out

The design of apply took many turns, with long and vigorous discussion and analysis. Many views and preferences were expressed, and which ultimately led to dynamic refinements as what we called straight sugar. That is, syntactic sugar at its sweetest. We knew /all had to be there, as that's what the others build on, but it was originally the default. We all eventually agreed that it shouldn't be, as it's the lowest level and likely the least directly used, though still vital for some use cases. Our problem was striking a balance between what would be most useful, with minimal overlap in use cases, and making the rules too complex to remember and get right. So a couple models didn't make the cut.

If Dynamic Refinements are straight sugar, the candy-wrapper version might be something like this, where you can still use a path, but only the args are in the block.

apply-args: function [
    "Make straight sugar call from semi-sweet model."
    fn [word! path!] "Get-word refinements come from context."
    args [block!]	 "Args only, no refinement values."
][
    ; Use temp result so we don't return any extra args
    do compose [res: (fn) (args)]
    res
]

>> apply-args 'append [[] [a]]
== [a]
>> apply-args 'append/only [[] [a]]
== [[a]]
>> only: false
== false
>> apply-arg 'append/:only [[] [a]]
== [a]
>> only: true
== true
>> apply-args 'append/:only [[] [a]]
== [[a]]

It's easy, but has quite a bit of overhead, because of compose and do. Remember, this amount of overhead only matters in loops running thousands of times at the very least, or in a real-time interactive interface. When in doubt, clock it. Write for humans to understand, and only optimize as needed (and after you know what's slow)

Another model is name-value args. That is, you provide a structure of arg names and their values, which is applied. It can make some code much clearer, but you also have to make sure the names match, so any refactoring of names will break code, which doesn't happen if args are positional. This is a bit involved, but it shows the power of Red. We'll use objects in our example, for a particular reason. That reason is values-of. The idea being that apply/all wants all the args, in order, and every slot filled. If your object matches the function spec, it's a perfect match. But making objects manually that way is error prone. So we'll use reflection for an object tailor-made for a given function.

Step 1: Find all the words in the spec. Remember it could have doc strings and arg types as well.

func-spec-words: function [
    "Get all the word-type values from a func spec."
    fn [any-function!]
    /as type [datatype!] "Cast results to this type."
][
    arg-types: make typeset! [word! lit-word! get-word! refinement!]
    parse spec-of :fn [
        ; If we want an apply-specific version of objects, we could
        ; denote refinements with a sigil for added clarity.
        collect [
            any [set w arg-types keep (either type [to type w][w]) | skip]
        ]
    ]
]

Step 2:  Make an object from that

func-spec-to-obj-proto: function [
    "Returns an object whose words match a function's spec."
    fn [any-function!]
    ; The idea here is that you can both preset values that are in the spec,
    ; and also extend the object with extra words, which will be ignored.
    /with args [block!] "APPLY/ALL ignores extra values."
][
    obj: construct/with any [args []]
    construct append func-spec-words/as :fn set-word! none
    ; Refinement values in APPLY/ALL calls MUST be logic!, not none!.
    foreach word func-spec-words :fn [
        if refinement? word [set in obj to word! word false]
    ]
    obj
]

Step Aside: Here's another approach, which combines steps 1 and 2, and lets you use a path for the function arg.

; Alt approach to func-spec-to-obj-proto, does NOT allow extending the spec.
make-apply-obj-proto: function [
    "Returns an object whose words match a function's spec."
    fn [any-function! word! path!]
    /with args [block!] "TBD: If APPLY doesn't ignore extra values, keys must be in spec."
][
    if path? :fn [refs: next fn   fn: first fn]        ; split path
    if word? :fn [
        name: fn                                       ; hold on to this for error report
        set/any 'fn get/any fn                         ; get func value
        if not any-function? :fn [
            do make error! rejoin ["FN argument (" name ") does not refer to a function."]
        ]
    ]                                                  ; get func value
    obj: construct append func-spec-words/as :fn set-word! none	; make object
    ; Refinement values in apply calls MUST be logic!, not none!.
    foreach word func-spec-words :fn [
        if refinement? word [set in obj to word! word false]
    ]
    if refs [foreach ref refs [obj/:ref: true]]        ; set refinements
    ; can't use obj/:key: if key is a set-word!
    if args [foreach [key val] args [set in obj key :val]]  ; set arg values
    obj
]

o-1: make-apply-obj-proto/with 'find/case/part/tail/skip/with [wild: "*?+" length: 10 size: 2]

Step 3. Using the object with APPLY

; We finally get here, and it's anticlimactic.
apply-object: func [
    "Call APPLY using an object's values as args."
    fn  [any-function!]
    obj [object!]
][
    apply/all :fn values-of obj
]

And an example call:

>> o-fctm: make-apply-obj-proto/with 'find/case/tail/match [series: [a b c] value: 'a]
== make object! [
    series: [a b c]
    value: 'a
    part: false
    length: none
    only: false
    case: true
    same: fal...
>> apply-object :find o-fctm
== [b c]

But you may see that this is verbose and inefficient, making a whole object just for a call like this. And you'd be right. It's just an example.

You don't want to recreate objects like this, especially in a loop. But you don't have to. You can reuse the object and just change the important values in it. This blog is getting long already, so we'll leave that as an exercise for the reader, or a question in community chat. And if you reuse the same object, the overhead is minimal.

We even talked about an idea whose time has not come, and is not guaranteed to work in the future. Here's the idea:

dyna-ref: func [p [path!]][
    res: make path! collect [
        keep p/1
        foreach val next p [
            case [
                get-word? val [if get :val [keep to word! val]]
                all [paren? val get-word? val/1] [if do next val [keep to word! val/1]]
                paren? val [do make error! "Sorry, has to be (get-word expr) for use with dyna-path."]
                'else [keep val]
            ]
        ]
    ]
    res
]

c: true
print mold dyna-ref 'a/b/:c/(:d true)
print mold dyna-ref 'a/b/:c/(:d false)
c: false
print mold dyna-ref 'a/b/:c/(:d true)
print mold dyna-ref 'a/b/:c/(:d false)

That's right, it's a dialected path! that builds a dynamic path. Crazy, right? You may know that while paths can currently contain parens, for Rebol compatibility, that feature may go away. It has deep design implications, but is also very handy at times. And while this isn't part of Red, it's another example of how Red lets us think off the beaten path.

Interpreter improvements

Dynamic refinements and function application are supported at interpreter level, for maximum efficiency and code reuse (mostly arguments fetching and type-checking). In addition to that, long standing issues and needed simplifications have been made in the interpreter code. Here is the changelog:

Function arguments cache entirely reimplemented:

• Massively reduces the amount of code needed to manage the caches.
• Simpler and faster cache design, O(1) lookup time for refinements in paths.
• Adds a context! to native!, action! and routine!, to speed up word lookups.
• Fixes long standing issue #4854 (CRASH & CORRUPTION in "dynamic" function calls with refinements)

Simpler reimplementation of op! datatype and its evaluation:

• red-op! structure redesigned: it is now a shallow copy of the underlying function (nodes are not copied), the sub-type is stored in the op! header.
• is infix function has been deprecated and replaced by a prefix version: relate.
• Massive code reduction and simplification compared to previous version. Now op! maximally reuses the interpreter code for evaluating other functions.

Those changes lead to a general interpreter speed-up of 3-5% and up to 20% in some cases.

Additional language changes:

• The in native now accepts any-function! as its first argument and refinements as the second argument. Refinements, if matched, will be converted to word values. This makes in a fast way to check if a symbol is part of an object or function spec and return a word bound to that context.

Conclusion

The most important thing you should do now is try it. It's a new design, and we want to hear what people like, see what they try, and where it falls short.

Happy Reducing!

-Red Team