lparallel-2.1.0 released

  • added readers kernel-name and kernel-context
  • added restart kill-errors to workers — removes debugger popups
  • attempting to submit a task to an ended kernel now signals an error
  • suicidal calls to kill-tasks inside workers are now permitted
Posted in Uncategorized | Leave a comment

Concurrent Hello World

Below is another example of Concurrent Hello World. The macros receive and run are just 9 lines each, given below the fold.

(defpackage :example (:use :cl :node))
(in-package :example)

(defun hello (hello-queue world-queue)
  (receive hello-queue
    (:say-hello (n)
      (cond ((plusp n)
             (format t "Hello ")
             (send world-queue :say-world n))
            (t
             (send world-queue :quit)
             (return))))))

(defun world (world-queue hello-queue)
  (receive world-queue
    (:say-world (n)
      (format t "World!~%")
      (send hello-queue :say-hello (- n 1)))
    (:quit ()
      (return))))

(defun main (n)
  (let ((hello-queue (make-queue))
        (world-queue (make-queue)))
    (run
      (make-node 'hello hello-queue world-queue)
      (make-node 'world world-queue hello-queue)
      (send hello-queue :say-hello n))))

Continue reading

Posted in Uncategorized | 3 Comments

lparallel-2.0.0 released

The major version bump is for some incompatible changes, though they are unlikely to cause trouble.

  • keyword arguments to psort besides :key have been replaced with a single :granularity argument; the old arguments are now ignored
  • removed deprecated aliases from 1.2.0 and 1.3.0 (you may not be aware of them since they haven’t been listed in the documentation)
  • A function defined with defpun is now optimized for N worker threads where N is the number of cores. The old behavior is available with defpun*, which defines a function that is optimized for N-1 workers (and has less overhead).
  • added psort* — like psort but targets N-1 workers
  • improved performance of psort
  • task categories are now compared with eql; same for ptree node ids

The benchmarks page has been updated to reflect recent optimizations, which includes optimizations in SBCL itself.

Posted in Uncategorized | Leave a comment

Status update

  • The latest CLISP and ECL appear close to passing all tests, and may indeed pass on some platforms.
  • I am not aware of any outstanding CL implementation bugs affecting lparallel besides those reported in CLISP and ECL. I am also not aware of any bugs in lparallel itself.
  • If you happen to be using bignums on 32-bit x86 CCL 1.8 or earlier, you should get latest from the CCL repository and build a new image.
  • lparallel does not seem complex enough to warrant a mailing list, yet some things may not be entirely simple either. Feel free to ask questions or offer feedback on this thread, or send me email.
  • I plan to remove some old deprecated aliases in version 2.0 (they are not shown in the documentation here). Now is the time to suggest incompatible changes, before the 2.0 bump.
  • I have been hesitant to add a nickname to the lparallel package. The separation of the lparallel API into a handful of packages was meant to encourage people to use a subset of the API, e.g. (:use :lparallel.cognate). However some people always write package-qualified symbols, and for them an lp or ll nickname would be convenient. I am not exactly against this, but it does entail a bit of peril in the form of increased likelihood of conflict.
  • I have noticed this pattern being used: (let ((*kernel* (make-kernel ...))) ...). This is not recommended for three reasons. First, it makes the kernel object inaccessible to other (non-worker) threads, preventing the use of kill-tasks in the REPL for example. Second, end-kernel is likely to be forgotten, resulting in a kernel that is not garbage collected. Third, even if we properly abstract this pattern by writing a with-temp-kernel macro that calls end-kernel, such a macro lends itself to suboptimal code because multiple uses of it would defeat the benefits of a thread pool. These issues are avoided by calling (setf *kernel* ...) or by binding to an existing kernel, for example (let ((*kernel* *io-kernel*)) ...).
  • A with-temp-kernel macro may still be convenient in non-critical cases such as testing, yet I hesitate to include it in the lparallel API for the reasons mentioned above.
Posted in Uncategorized | Leave a comment

lparallel-1.7.0 released

  • added pdotimes
  • optimized cognate functions and macros when they are called inside worker threads; e.g. pmap in (future (pmap ...)) no longer blocks a worker
Posted in Uncategorized | Leave a comment

lparallel-1.6.0 released

  • added clear-ptree-errors — for resuming after an error
  • added clear-ptree — for recomputing from scratch
  • improved task handling for ptrees
  • :lparallel now in *features* after load
  • defpun no longer transforms pfuncall forms
Posted in Uncategorized | Leave a comment

Auto-kill

When an evaluation fails or is interrupted, it may be convenient to automatically kill tasks created during the evaluation. One use for this might be for debugging a set of long-running tasks. Here is a solution using alexandria’s unwind-protect-case.

(defpackage :example (:use :cl :lparallel :alexandria))
(in-package :example)

(defun call-with-kill-on-abort (fn task-category)
  (let ((*task-category* task-category))
    (unwind-protect-case ()
        (funcall fn)
      (:abort (kill-tasks task-category)))))

(defmacro with-kill-on-abort ((&key (task-category '*task-category*))
                              &body body)
  `(call-with-kill-on-abort (lambda () ,@body) ,task-category))

(defun foo ()
  (with-kill-on-abort (:task-category 'foo-stuff)
    (pmap nil #'sleep '(9999 9999))))

Example run in SLIME:

CL-USER> (example::foo) ; ... then hit C-c-c
WARNING: lparallel: Replacing lost or dead worker.
WARNING: lparallel: Replacing lost or dead worker.
; Evaluation aborted on NIL.

As always, worker threads are regenerated after being killed.

Posted in Uncategorized | Leave a comment