Topologies
[training topology]
@beam/dilocoSynchronous DiLoCo
H local steps per round on workers with optimizer state, an outer aggregation on the lead and a broadcast of the merged model.
Static membershipBuiltinActivedistributed run
[topology identity]
Details
What version 1.0.1 declares.
- Package
@beam/diloco- Membership
- Static
- Engines
- distributed run
- Tags
- diloco, rounds, stateful, synthetic
- Step kinds
- broadcast, local, tree
- Latest tag
- latest -> 1.0.1
- Publisher
- registry
- Visibility
- Public
- Checksum
sha256:aca3df8a00b3d93495dccd3c94a94e140c9dc836366778481d3012cb97919401
[roles]
Roles
Who takes part, and how many members each role holds.
The member that initializes, merges and broadcasts the global model.
Every eligible room member runs the inner steps.
[slots]
Slots
The actions a run binds: what each slot consumes and produces.
model?, neighbor? -> model, report
Initializes the global model (a training run without compute or failures).
update* -> model, report?
The outer step; associative, commutative and closed (a tree on the lead above 15 workers).
model?, neighbor? -> update, report
H inner steps from the received global model; keeps its inner optimizer state.
[round]
Step kinds
The steps of one round, in order.
- broadcastfrom lead · to workers · object model
- localrole workers · slot train
- treefrom workers · to lead · object update · slot merge