Lisp in Parallel : lparallel

• exported types: kernel, channel, ptree
• added ptree-computed-p — query the computed state of a ptree node
• make-kernel now aborts cleanly when a worker fails to initialize, e.g. when make-thread fails or when a :context function aborts
• check-kernel now returns a kernel instance
• added a front-end lock to some ptree functions — removes the requirement that some calls be exclusive
• improved performance of functions defined by defpun

• added readers kernel-name and kernel-context
• added restart kill-errors to workers — removes debugger popups
• attempting to submit a task to an ended kernel now signals an error
• suicidal calls to kill-tasks inside workers are now permitted

Below is another example of Concurrent Hello World. The macros receive and run are just 9 lines each, given below the fold.

(defpackage :example (:use :cl :node))
(in-package :example)

(defun hello (hello-queue world-queue)
  (receive hello-queue
    (:say-hello (n)
      (cond ((plusp n)
             (format t "Hello ")
             (send world-queue :say-world n))
            (t
             (send world-queue :quit)
             (return))))))

(defun world (world-queue hello-queue)
  (receive world-queue
    (:say-world (n)
      (format t "World!~%")
      (send hello-queue :say-hello (- n 1)))
    (:quit ()
      (return))))

(defun main (n)
  (let ((hello-queue (make-queue))
        (world-queue (make-queue)))
    (run
      (make-node 'hello hello-queue world-queue)
      (make-node 'world world-queue hello-queue)
      (send hello-queue :say-hello n))))

Continue reading →

The major version bump is for some incompatible changes, though they are unlikely to cause trouble.

• keyword arguments to psort besides :key have been replaced with a single :granularity argument; the old arguments are now ignored
• removed deprecated aliases from 1.2.0 and 1.3.0 (you may not be aware of them since they haven’t been listed in the documentation)
• A function defined with defpun is now optimized for N worker threads where N is the number of cores. The old behavior is available with defpun*, which defines a function that is optimized for N-1 workers (and has less overhead).
• added psort* — like psort but targets N-1 workers
• improved performance of psort
• task categories are now compared with eql; same for ptree node ids

The benchmarks page has been updated to reflect recent optimizations, which includes optimizations in SBCL itself.

• The latest CLISP and ECL appear close to passing all tests, and may indeed pass on some platforms.
• I am not aware of any outstanding CL implementation bugs affecting lparallel besides those reported in CLISP and ECL. I am also not aware of any bugs in lparallel itself.
• If you happen to be using bignums on 32-bit x86 CCL 1.8 or earlier, you should get latest from the CCL repository and build a new image.
• lparallel does not seem complex enough to warrant a mailing list, yet some things may not be entirely simple either. Feel free to ask questions or offer feedback on this thread, or send me email.
• I plan to remove some old deprecated aliases in version 2.0 (they are not shown in the documentation here). Now is the time to suggest incompatible changes, before the 2.0 bump.
• I have been hesitant to add a nickname to the lparallel package. The separation of the lparallel API into a handful of packages was meant to encourage people to use a subset of the API, e.g. (:use :lparallel.cognate). However some people always write package-qualified symbols, and for them an lp or ll nickname would be convenient. I am not exactly against this, but it does entail a bit of peril in the form of increased likelihood of conflict.
• I have noticed this pattern being used: (let ((*kernel* (make-kernel ...))) ...). This is not recommended for three reasons. First, it makes the kernel object inaccessible to other (non-worker) threads, preventing the use of kill-tasks in the REPL for example. Second, end-kernel is likely to be forgotten, resulting in a kernel that is not garbage collected. Third, even if we properly abstract this pattern by writing a with-temp-kernel macro that calls end-kernel, such a macro lends itself to suboptimal code because multiple uses of it would defeat the benefits of a thread pool. These issues are avoided by calling (setf *kernel* ...) or by binding to an existing kernel, for example (let ((*kernel* *io-kernel*)) ...).
• A with-temp-kernel macro may still be convenient in non-critical cases such as testing, yet I hesitate to include it in the lparallel API for the reasons mentioned above.

• added pdotimes
• optimized cognate functions and macros when they are called inside worker threads; e.g. pmap in (future (pmap ...)) no longer blocks a worker