Parsl
Brief for “Parallel Scripting Library,” Parsl enables you to take computing jobs and break up them throughout a number of techniques utilizing roughly the identical syntax as Python’s present Pool objects. It additionally enables you to sew collectively totally different computing duties into multi-step workflows, which may run in parallel, in sequence, or through map/cut back operations.
Parsl enables you to execute native Python functions, but in addition run another exterior software by the use of instructions to the shell. Your Python code is written like regular Python code, save for a particular operate decorator that marks the entry level to your work. The job-submission system additionally offers you fine-grained management over how issues run on the targets—for instance, the variety of cores per employee, how a lot reminiscence per employee, CPU affinity controls, how typically to ballot for timeouts, and so forth.
One wonderful characteristic Parsl presents is a set of prebuilt templates to dispatch work to quite a lot of high-end computing assets. This not solely consists of staples like AWS or Kubernetes clusters, however supercomputing assets (assuming you’ve got entry) like Blue Waters, ASPIRE 1, Frontera, and so forth. (Parsl was co-developed with assistance from most of the establishments that constructed such {hardware}.)
