ledger¶
The unit ledger: workers of a split task claim units through atomic renames.
A directory with pending/<i>, claimed/<i>@<worker> and done/<i>.
Workers touch heartbeats/<worker>, so that the claims of a worker which died
without releasing them are found by the age of its heartbeat (release_stale).
Every transition is one os.rename, atomic on a local file system and on NFS v4,
so a unit is held by at most one worker. On NFS the mount must be NFS >= 4.1
(sessions give renames exactly-once semantics; claim also tolerates a resent
rename that answers ENOENT although it happened).
Open the ledger at root of n_units units (see create).
classmethod
¶
Create the ledger, or open an existing one of the same size.
The ledger is built in a temporary directory and renamed, so it appears complete or not at all, also when several processes create it at once or one of them crashes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
Path
|
The ledger directory. |
required |
n_units
|
int
|
The number of units, indices |
required |
Returns:
| Type | Description |
|---|---|
Self
|
The ledger. |
Raises:
| Type | Description |
|---|---|
ValueError
|
An existing ledger has another size. |
Free every unit claimed by worker, as a restarted worker does.
Returns:
| Type | Description |
|---|---|
list[int]
|
The indices released, sorted. |
Touch the heartbeat file of worker, created with its directory lazily.
Its modification time is set by the file server, so the heartbeats of workers on different servers compare without their clocks.
Raises:
| Type | Description |
|---|---|
OSError
|
The time of the file was not set. |
Return the claims of workers without a recent heartbeat to pending.
The heartbeat of reference is touched and its time is "now"; a claim of
another worker which has no heartbeat or one older than now - stale_s is
released. The claims of reference are never released.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stale_s
|
float
|
Seconds after which a heartbeat is stale. |
required |
reference
|
str
|
The worker which releases, e.g. the coordinator. |
required |
Returns:
| Type | Description |
|---|---|
list[int]
|
The indices released, sorted. |
Raises:
| Type | Description |
|---|---|
OSError
|
The heartbeat of |