# CLOSURES IN RUBY Paul Cantrell http://innig.net # Email: username "cantrell", domain name "pobox.com" # I recommend executing this file, then reading it alongside its output. # # Alteratively, you can give yourself a sort of Ruby test by deleting all the comments, # then trying to guess the output of the code! # A closure is a block of code which meets three criteria: # # * It can be passed around as a value and # # * executed on demand by anyone who has that value, at which time # # * it can refer to variables from the context in which it was created # (i.e. it is closed with respect to variable access, in the # mathematical sense of the word "closed"). # # (The word "closure" actually has an imprecise meaning, and some people don't # think that criterion #1 is part of the definition. I think it is.) # # Closures are a mainstay of functional languages, but are present in many other # languages as well (e.g. Java's anonymous inner classes). You can do cool stuff # with them: they allow deferred execution, and some elegant tricks of style. # # Ruby is based on the "principle of least surprise," but I had a really nasty # surprise in my learning process. When I understood what methods like "each" # were doing, I thought, "Aha! Ruby has closures!" But then I found out that a # function can't accept multiple blocks -- violating the principle that closures # can be passed around freely as values. # # This document details what I learned in my quest to figure out what the deal is. def example(num) puts puts "------ Example #{num} ------" end # ---------------------------- Section 1: Blocks ---------------------------- # Blocks are like closures, because they can refer to variables from their defining context: example 1 def thrice yield yield yield end x = 5 puts "value of x before: #{x}" thrice { x += 1 } puts "value of x after: #{x}" # A block refers to variables in the context it was defined, not the context in which it is called: example 2 def thrice_with_local_x x = 100 yield yield yield puts "value of x at end of thrice_with_local_x: #{x}" end x = 5 thrice_with_local_x { x += 1 } puts "value of outer x after: #{x}" # A block only refers to *existing* variables in the outer context; if they don't exist in the outer, a # block won't create them there: example 3 thrice do # note that {...} and do...end are completely equivalent y = 10 puts "Is y defined inside the block where it is first set?" puts "Yes." if defined? y end puts "Is y defined in the outer context after being set in the block?" puts "No!" unless defined? y # OK, so blocks seem to be like closures: they are closed with respect to variables defined in the context # where they were created, regardless of the context in which they're called. # # But they're not quite closures as we've been using them, because we have no way to pass them around: # "yield" can *only* refer to the block passed to the method it's in. # # We can pass a block on down the chain, however, using &: example 4 def six_times(&block) thrice(&block) thrice(&block) end x = 4 six_times { x += 10 } puts "value of x after: #{x}" # So do we have closures? Not quite! We can't hold on to a &block and call it later at an arbitrary # time; it doesn't work. This, for example, will not compile: # # def save_block_for_later(&block) # saved = █ # end # # But we *can* pass it around if we use drop the &, and use block.call(...) instead of yield: example 5 def save_for_later(&b) @saved = b # Note: no ampersand! This turns a block into a closure of sorts. end save_for_later { puts "Hello!" } puts "Deferred execution of a block:" @saved.call @saved.call # But wait! We can't pass multiple blocks to a function! As it turns out, there can be only zero # or one &block_params to a function, and the ¶m *must* be the last in the list. # # None of these will compile: # # def f(&block1, &block2) ... # def f(&block1, arg_after_block) ... # f { puts "block1" } { puts "block2" } # # What the heck? # # I claim this single-block limitation violates the "principle of least surprise." The reasons for # it have to do with ease of C implementation, not semantics. # # So: are we screwed for ever doing anything robust and interesting with closures? # ---------------------------- Section 2: Closure-Like Ruby Constructs ---------------------------- # Actually, no. When we pass a block ¶m, then refer to that param without the ampersand, that # is secretly a synonym for Proc.new(¶m): example 6 def save_for_later(&b) @saved = Proc.new(&b) # same as: @saved = b end save_for_later { puts "Hello again!" } puts "Deferred execution of a Proc works just the same with Proc.new:" @saved.call # We can define a Proc on the spot, no need for the ¶m: example 7 @saved_proc_new = Proc.new { puts "I'm declared on the spot with Proc.new." } puts "Deferred execution of a Proc works just the same with ad-hoc Proc.new:" @saved_proc_new.call # Behold! A true closure! # # But wait, there's more.... Ruby has a whole bunch of things that seem to behave like closures, # and can be called with .call: example 8 @saved_proc_new = Proc.new { puts "I'm declared with Proc.new." } @saved_proc = proc { puts "I'm declared with proc." } @saved_lambda = lambda { puts "I'm declared with lambda." } def some_method puts "I'm declared as a method." end @method_as_closure = method(:some_method) puts "Here are four superficially identical forms of deferred execution:" @saved_proc_new.call @saved_proc.call @saved_lambda.call @method_as_closure.call # So in fact, there are no less than seven -- count 'em, SEVEN -- different closure-like constructs in Ruby: # # 1. block (implicitly passed, called with yield) # 2. block (&b => f(&b) => yield) # 3. block (&b => b.call) # 4. Proc.new # 5. proc # 6. lambda # 7. method # # Though they all look different, some of these are secretly identical, as we'll see shortly. # # We already know that (1) and (2) are not really closures -- and they are, in fact, exactly the same thing. # Numbers 3-7 all seem to be identical. Are they just different syntaxes for identical semantics? # ---------------------------- Section 3: Closures and Control Flow ---------------------------- # No, they aren't! One of the distinguishing features has to do with what "return" does. # # Consider first this example of several different closure-like things *without* a return statement. # They all behave identically: example 9 def f(closure) puts puts "About to call closure" result = closure.call puts "Closure returned: #{result}" "Value from f" end puts "f returned: " + f(Proc.new { "Value from Proc.new" }) puts "f returned: " + f(proc { "Value from proc" }) puts "f returned: " + f(lambda { "Value from lambda" }) def another_method "Value from method" end puts "f returned: " + f(method(:another_method)) # But put in a "return," and all hell breaks loose! example 10 begin f(Proc.new { return "Value from Proc.new" }) rescue Exception => e puts "Failed with #{e.class}: #{e}" end # The call fails because that "return" needs to be inside a function, and a Proc isn't really # quite a full-fledged function: example 11 def g result = f(Proc.new { return "Value from Proc.new" }) puts "f returned: " + result #never executed "Value from g" #never executed end puts "g returned: #{g}" # Note that the return inside the "Proc.new" didn't just return from the Proc -- it returned # all the way out of g, bypassing not only the rest of g but the rest of f as well! It worked # almost like an exception. # # This means that it's not possible to call a Proc containing a "return" when the creating # context no longer exists: example 12 def make_proc_new begin Proc.new { return "Value from Proc.new" } # this "return" will return from make_proc_new ensure puts "make_proc_new exited" end end begin puts make_proc_new.call rescue Exception => e puts "Failed with #{e.class}: #{e}" end # (Note that this makes it unsafe to pass Procs across threads.) # A Proc.new, then, is not quite truly closed: it depends in the creating context still existing, # because the "return" is tied to that context. # # Not so for lambda: example 13 def g result = f(lambda { return "Value from lambda" }) puts "f returned: " + result "Value from g" end puts "g returned: #{g}" # And yes, you can call a lambda even when the creating context is gone: example 14 def make_lambda begin lambda { return "Value from lambda" } ensure puts "make_lambda exited" end end puts make_lambda.call # Inside a lambda, a return statement only returns from the lambda, and flow continues normally. # So a lambda is like a function unto itself, whereas a Proc remains dependent on the control # flow of its caller. # # A lambda, therefore, is Ruby's true closure. # # As it turns out, "proc" is a synonym for either "Proc.new" or "lambda." # Anybody want to guess which one? (Hint: "Proc" in lowercase is "proc.") example 15 def g result = f(proc { return "Value from proc" }) puts "f returned: " + result "Value from g" end puts "g returned: #{g}" # Hah. Fooled you. # # The answer: Ruby changed its mind. If you're using Ruby 1.8, it's a synonym for "lambda." # That's surprising (and also ridiculous); somebody figured this out, so in 1.9, it's a synonym for # Proc.new. Go figure. # I'll spare you the rest of the experiments, and give you the behavior of all 7 cases: # # "return" returns from caller: # 1. block (called with yield) # 2. block (&b => f(&b) => yield) # 3. block (&b => b.call) # 4. Proc.new # 5. proc in 1.9 # # "return" only returns from closure: # 5. proc in 1.8 # 6. lambda # 7. method # ---------------------------- Section 4: Closures and Arity ---------------------------- # The other major distinguishing of different kinds of Ruby closures is how they handle mismatched # arity -- in other words, the wrong number of arguments. # # In addition to "call," every closure has an "arity" method which returns the number of expected # arguments: example 16 puts "One-arg lambda:" puts (lambda {|x|}.arity) puts "Three-arg lambda:" puts (lambda {|x,y,z|}.arity) # ...well, sort of: puts "No-args lambda: " puts (lambda {}.arity) # This behavior is also subject to change in 1.9. puts "Varargs lambda: " puts (lambda {|*args|}.arity) # Watch what happens when we call these with the wrong number of arguments: example 17 def call_with_too_many_args(closure) begin puts "closure arity: #{closure.arity}" closure.call(1,2,3,4,5,6) puts "Too many args worked" rescue Exception => e puts "Too many args threw exception #{e.class}: #{e}" end end def two_arg_method(x,y) end puts; puts "Proc.new:"; call_with_too_many_args(Proc.new {|x,y|}) puts; puts "proc:" ; call_with_too_many_args(proc {|x,y|}) puts; puts "lambda:" ; call_with_too_many_args(lambda {|x,y|}) puts; puts "Method:" ; call_with_too_many_args(method(:two_arg_method)) def call_with_too_few_args(closure) begin puts "closure arity: #{closure.arity}" closure.call() puts "Too few args worked" rescue Exception => e puts "Too few args threw exception #{e.class}: #{e}" end end puts; puts "Proc.new:"; call_with_too_few_args(Proc.new {|x,y|}) puts; puts "proc:" ; call_with_too_few_args(proc {|x,y|}) puts; puts "lambda:" ; call_with_too_few_args(lambda {|x,y|}) puts; puts "Method:" ; call_with_too_few_args(method(:two_arg_method)) # Yet oddly, the behavior for one-argument closures is different.... example 18 def one_arg_method(x) end puts; puts "Proc.new:"; call_with_too_many_args(Proc.new {|x|}) puts; puts "proc:" ; call_with_too_many_args(proc {|x|}) puts; puts "lambda:" ; call_with_too_many_args(lambda {|x|}) puts; puts "Method:" ; call_with_too_many_args(method(:one_arg_method)) puts; puts "Proc.new:"; call_with_too_few_args(Proc.new {|x|}) puts; puts "proc:" ; call_with_too_few_args(proc {|x|}) puts; puts "lambda:" ; call_with_too_few_args(lambda {|x|}) puts; puts "Method:" ; call_with_too_few_args(method(:one_arg_method)) # Yet when there are no args... example 19 def no_arg_method end puts; puts "Proc.new:"; call_with_too_many_args(Proc.new {||}) puts; puts "proc:" ; call_with_too_many_args(proc {||}) puts; puts "lambda:" ; call_with_too_many_args(lambda {||}) puts; puts "Method:" ; call_with_too_many_args(method(:no_arg_method)) # For no good reason that I can see, Proc.new, proc and lambda treat a single argument as a special # case; only a method enforces arity in all cases. Principle of least surprise my ass. # ---------------------------- Section 5: Rant ---------------------------- # # This is quite a dizzing array of syntactic options, with subtle semantics differences that are not # at all obvious, and riddled with minor special cases. It's like a big bear trap from programmers who # expect the language to just work. # # Why are things this way? Because Ruby is: # # (1) designed by implementation, and # (2) defined by implementation. # # The language grows because the Ruby team tacks on cool ideas, without maintaining a real spec apart # from CRuby. A spec would make clear the logical structure of the language, and thus help highlight # inconsistencies like the ones we've just seen. Instead, these inconsinstencies creep into the language, # confuse the crap out of poor souls like me who are trying to learn it, and then get submitted as bug # reports. Something as fundamental as the semantics of proc should not get so screwed up that they have # to backtrack between releases, for heaven's sake! Yes, I know, language design is hard -- but something # like this proc/lambda issue or the arity problem wasn't so hard to get right the first time. # Yammer yammer. # ---------------------------- Section 6: Summary ---------------------------- # # So, what's the final verdict on those 7 closure-like entities? # # "return" returns from closure # True closure? or declaring context...? Arity check? # --------------- ----------------------------- ------------------- # 1. block (called with yield) N declaring no # 2. block (&b => f(&b) => yield) N declaring no # 3. block (&b => b.call) Y except return declaring warn on too few # 4. Proc.new Y except return declaring warn on too few # 5. proc <<< alias for lambda in 1.8, Proc.new in 1.9 >>> # 6. lambda Y closure yes, except arity 1 # 7. method Y closure yes # # The things within each of these groups are all semantically identical -- that is, they're different # syntaxes for the same thing: # # 1. block (called with yield) # 2. block (&b => f(&b) => yield) # ------- # 3. block (&b => b.call) # 4. Proc.new # 5. proc in 1.9 # ------- # 5. proc in 1.8 # 6. lambda # ------- # 7. method (may be identical to lambda with changes to arity checking in 1.9) # # Or at least, this is how I *think* it is, based on experiment. There's no authoritative answer other # than testing the CRuby implementation, because there's no real spec -- so there may be other differences # I haven't discovered. # # The final verdict: Ruby has four types of closures and near-closures, expressible in seven syntactic # variants. Not pretty. But you sure sure do cool stuff with them! That's up next.... # # This concludes the "Ruby sucks" portion of our broadcast; from here on, it will be the "Ruby is # awesome" portion. # ---------------------------- Section 7: Doing Something Cool with Closures ---------------------------- # Let's make a data structure containing all of the Fibonacci numbers. Yes, I said *all* of them. # How is this possible? We'll use closures to do lazy evaluation, so that the computer only calculates # as much of the list as we ask for. # To make this work, we're going to use Lisp-style lists: a list is a recursive data structure with # two parts: "car," the next element of the list, and "cdr," the remainder of the list. # # For example, the list of the first three positive integers is [1,[2,[3]]]. Why? Because: # # [1,[2,[3]]] <--- car=1, cdr=[2,[3]] # [2,[3]] <--- car=2, cdr=[3] # [3] <--- car=3, cdr=nil # # Here's a class for traversing such lists: example 20 class LispyEnumerable include Enumerable def initialize(tree) @tree = tree end def each while @tree car,cdr = @tree yield car @tree = cdr end end end list = [1,[2,[3]]] LispyEnumerable.new(list).each do |x| puts x end # So how to make an infinite list? Instead of making each node in the list a fully built # data structure, we'll make it a closure -- and then we won't call that closure # until we actually need the value. This applies recursively: the top of the tree is a closure, # and its cdr is a closure, and the cdr's cdr is a closure.... example 21 class LazyLispyEnumerable include Enumerable def initialize(tree) @tree = tree end def each while @tree car,cdr = @tree.call # <--- @tree is a closure yield car @tree = cdr end end end list = lambda{[1, lambda {[2, lambda {[3]}]}]} # same as above, except we wrap each level in a lambda LazyLispyEnumerable.new(list).each do |x| puts x end example 22 # Let's see when each of those blocks gets called: list = lambda do puts "first lambda called" [1, lambda do puts "second lambda called" [2, lambda do puts "third lambda called" [3] end] end] end puts "List created; about to iterate:" LazyLispyEnumerable.new(list).each do |x| puts x end # Now, because the lambda defers evaluation, we can make an infinite list: example 23 def fibo(a,b) lambda { [a, fibo(b,a+b)] } # <---- this would go into infinite recursion if it weren't in a lambda end LazyLispyEnumerable.new(fibo(1,1)).each do |x| puts x break if x > 100 # we don't actually want to print all of the Fibonaccis! end # This kind of deferred execution is called "lazy evaluation" -- as opposed to the "eager # evaluation" we're used to, where we evaluate an expression before passing its value on. # (Most languages, including Ruby, use eager evaluation, but there are languages (like Haskell) # which use lazy evaluation for everything, by default! Not always performant, but ever so very cool.) # # This way of implementing lazy evaluation is terribly clunky! We had to write a separate # LazyLispyEnumerable that *knows* we're passing it a special lazy data structure. How unsatisfying! # Wouldn't it be nice of the lazy evaluation were invisible to callers of the lazy object? # # As it turns out, we can do this. We'll define a class called "Lazy," which takes a block, turns it # into a closure, and holds onto it without immediately calling it. The first time somebody calls a # method, we evaluate the closure and then forward the method call on to the closure's result. class Lazy def initialize(&generator) @generator = generator end def method_missing(method, *args, &block) evaluate.send(method, *args, &block) end def evaluate @value = @generator.call unless @value @value end end def lazy(&b) Lazy.new &b end # This basically allows us to say: # # lazy {value} # # ...and get an object that *looks* exactly like value -- except that value won't be created until the # first method call that touches it. It creates a transparent lazy proxy object. Observe: example 24 x = lazy do puts "<<< Evaluating lazy value >>>" "lazy value" end puts "x has now been assigned" puts "About to call one of x's methods:" puts "x.size: #{x.size}" # <--- .size triggers lazy evaluation puts "x.swapcase: #{x.swapcase}" # So now, if we define fibo using lazy instead of lambda, it should magically work with our # original LispyEnumerable -- which has no idea it's dealing with a lazy value! Right? example 25 def fibo(a,b) lazy { [a, fibo(b,a+b)] } end LispyEnumerable.new(fibo(1,1)).each do |x| puts x end # Oops! That didn't work. What went wrong? # # The failure started in this line of LispyEnumerable (though Ruby didn't report the error there): # # car,cdr = @tree # # Let's zoom in on that result, and see what happened: example 26 car,cdr = fibo(1,1) puts "car=#{car} cdr=#{cdr}" # Here's the problem. When we do this: # # x,y = z # # ...Ruby calls z.respond_to?(to_a) to see if z is an array. If it is, it will do the multiple # assignment; if not, it will just assign x=z and set y=nil. # # We want our Lazy to forward the respond_to? call to our fibo list. But it doesn't forward it, # because we used the method_missing to do the proxying -- and every object implements respond_to? # by default, so the method isn't missing! The respond_to? doesn't get forwarded; instead, out Lazy # says "No, I don't respond to to_a; thanks for asking." The immediate solution is to forward # respond_to? manually: class Lazy def initialize(&generator) @generator = generator end def method_missing(method, *args, &block) evaluate.send(method, *args, &block) end def respond_to?(method) evaluate.respond_to?(method) end def evaluate @value = @generator.call unless @value @value end end # And *now* our original Lispy enum can work: example 27 LispyEnumerable.new(fibo(1,1)).each do |x| puts x break if x > 200 end # Of course, this only fixes the problem for respond_to?, and we have the same problem for every other # method of Object. There is a more robust solution -- frightening, but it works -- which is to undefine # all the methods of the Lazy when it's created, so that everything gets forwarded. # # And guess what? There's already a slick little gem that will do it: # # http://moonbase.rydia.net/software/lazy.rb/ # # Read the source. It's fascinating. # ---------------------------- Section 8: Wrap-Up ---------------------------- # So sure, this was all entertaining -- but is it good for anything? # # Well, suppose you have an object which requires a network or database call to be created, or will # use a lot of memory once it exists. And suppose that it may or may not be used, but you don't know # at the time it's created whether it will be. Making it lazy will prevent it from consuming resources # unless it needs to. Hibernate does this to prevent unnecessary DB queries, and it does it with more or # less arbitrary Java objects (i.e. unlike ActiveRecord, it doesn't depend on a base class to do its # lazy loading). Ruby can do the same thing, but with a lot less code! # # That's just an example. Use your imagination. # # If you're a functional langauge geek, and enjoyed seeing Ruby play with these ideas from Lisp and # Haskell, you may enjoy this thread: # # http://redhanded.hobix.com/inspect/curryingWithArity.html # # OK, I'll stop making your brain hurt now. Hope this has been a bit enlightening! The experience # of working it out certainly was for me. # # Paul