Copying and Process Management

This page documents the process-copy and bounded-process-management utilities implemented in src/Copy.jl and src/ProcessManager.jl.

At the moment these files are standalone utilities in the repository and are not yet included from Processes.jl, so the examples here assume those files have been loaded in the same way as the docs and tests do.

Why Copy Instead of `deepcopy`

Process contexts are often built from Input(...) values that point at external storage, buffers, or data views. A raw deepcopy of the live context can therefore copy the wrong thing or preserve sharing that should be rebuilt per process.

The copying helpers work from TaskData and the normal init pipeline instead:

copy the task description,
replace selected inputs and overrides,
initialize a fresh context for each copy.

Copy APIs

Main.ProcessesExtensionsDocs.copyinputs — Function

copyinputs(td::TaskData)
copyinputs(p::Process)

Return the input description stored on a task or process as fresh wrapper objects.

This copies the input descriptors, not the runtime context. It is meant for rebuilding new processes from the same process description while allowing new per-copy inputs to be merged in later.

source

Main.ProcessesExtensionsDocs.copyoverrides — Function

copyoverrides(td::TaskData)
copyoverrides(p::Process)

Return the override description stored on a task or process as fresh wrapper objects.

As with copyinputs, this copies the constructor-time description rather than the materialized runtime context.

source

Main.ProcessesExtensionsDocs.copytaskdata — Function

copytaskdata(td::TaskData, inputs_overrides...; keep_inputs = true, keep_overrides = true,
             lifetime = getlifetime(td), func = getalgo(td))
copytaskdata(p::Process, inputs_overrides...; kwargs...)

Create a fresh TaskData from an existing task or process description.

This is the safe copy primitive for process duplication. Instead of cloning a fully materialized context, it reconstructs the constructor-time description and optionally replaces inputs, overrides, lifetime, or the algorithm. This is useful when contexts contain external buffers or other values that should be re-initialized per copy.

source

Main.ProcessesExtensionsDocs.copyprocess — Function

copyprocess(td::TaskData, inputs_overrides...; keep_inputs = true, keep_overrides = true,
            lifetime = getlifetime(td), timeout = 1.0, context = nothing,
            context_builder = nothing, func = getalgo(td))
copyprocess(p::Process, inputs_overrides...; keep_inputs = true, keep_overrides = true,
            lifetime = getlifetime(taskdata(p)), timeout = p.timeout, context = nothing,
            context_builder = nothing, func = getalgo(taskdata(p)))

Build a fresh Process from an existing task description or process.

copyprocess deliberately avoids deepcopy of the live runtime context. Instead it reconstructs the process from TaskData, then initializes a fresh context unless you pass either:

context: an already-prepared context to install directly.
context_builder: a function receiving (taskdata) or (taskdata, original_process).

Additional Input(...) and Override(...) arguments are merged into the copied task description before initialization.

source

Typical Copy Pattern

template = Process(
    MyAlgo,
    Input(MyAlgo, :start => 0, :buffer => Int[]),
    Override(MyAlgo, :delta => 2);
    lifetime = 10,
)

p = copyprocess(
    template,
    Input(MyAlgo, :start => 100, :buffer => Int[]),
)

run(p)
wait(p)
close(p)

If the context needs custom rebuilding logic, pass context_builder = (taskdata, original) -> ... or provide a fully prepared context = ... directly.

Managed Execution

ProcessManager and manageprocesses are for running many processes while limiting how many are active at once.

This is useful when you have a list of job properties and want to:

build one process per property,
keep at most N running concurrently,
save large results to disk as soon as each process finishes.

Main.ProcessesExtensionsDocs.ManagedProcessResult — Type

ManagedProcessResult

Result record returned for each property handled by manageprocesses.

Fields:

idx: original index in the property list.
property: the property value used to create the process.
process: the process that was launched, or nothing if creation failed early.
context: final context if retained in memory, otherwise nothing.
savefile: saved .jld2 path when persistence was requested.
error: captured exception, or nothing on success.

source

Main.ProcessesExtensionsDocs.ProcessManager — Type

ProcessManager(makeprocess, properties; max_running = Threads.nthreads(),
               poll_interval = 0.01, savefolder = nothing,
               filename = default_manager_filename, onfinish = nothing,
               throw = true)

Bounded launcher for a collection of processes.

makeprocess is called with (property) or (property, idx) and must return either:

a Process, or
a named tuple containing at least process = ... and optionally savefile = ....

The manager launches at most max_running processes at a time, waits for finished ones, optionally saves their final contexts, and then continues launching the remaining jobs.

source

Main.ProcessesExtensionsDocs.manageprocesses — Function

manageprocesses(makeprocess, properties; kwargs...)

Run a bounded set of processes produced from properties.

This is a convenience wrapper around run(ProcessManager(...)).

source

manageprocesses(template::Process, properties, mapper = nothing; kwargs...)

Run a bounded set of copied processes starting from a template process.

mapper may return:

nothing: copy the template as-is,
a Process: use that process directly,
a named tuple with inputs, overrides, inputs_overrides, copy keywords, and optional savefile,
any other value, which is treated as positional input/override data for copyprocess.

source

Main.ProcessesExtensionsDocs.savecontext — Function

savecontext(context, filename)

Save a final context directly to a JLD2 file.

This method complements the package-level savecontext(::Process, ...) helper by working on already-materialized contexts, which is what ProcessManager has after finalization.

source

Property-Driven Manager Example

jobs = [
    (; start = 1, buffer = Int[]),
    (; start = 10, buffer = Int[]),
    (; start = 100, buffer = Int[]),
]

results = manageprocesses(jobs; max_running = 2) do job, idx
    process = Process(
        MyAlgo,
        Input(MyAlgo, :start => job.start, :buffer => job.buffer),
        Override(MyAlgo, :delta => 3);
        lifetime = 20,
    )

    (; process, savefile = "job_$idx.jld2")
end

The result vector stays in the same order as jobs, even when completion order differs.

Template-Based Manager Example

template = Process(
    MyAlgo,
    Input(MyAlgo, :start => 0, :buffer => Int[]),
    Override(MyAlgo, :delta => 3);
    lifetime = 20,
)

results = manageprocesses(template, jobs,
    job -> (;
        inputs = Input(MyAlgo, :start => job.start, :buffer => job.buffer),
        savefile = "job_$(job.start).jld2",
    );
    max_running = 2,
    savefolder = "saved_contexts",
)

When savefolder is provided, finished contexts are written to JLD2 files and omitted from the in-memory result payload.

Copying and Process Management

Why Copy Instead of deepcopy

Copy APIs

Typical Copy Pattern

Managed Execution

Property-Driven Manager Example

Template-Based Manager Example

Why Copy Instead of `deepcopy`