EiBSSKrSSKrSSKJr SSKJrJrJrJr SSK J r J r J r J r SSKJr \(aSSKJr \"SS55rg) N) dataclass) TYPE_CHECKINGLiteralOptionalUnion)DeepSpeedSequenceParallelConfigDistributedTypeTorchContextParallelConfigTorchTensorParallelConfig)is_torch_version) AcceleratorcR\rSrSr%SrSr\\\S'Sr \\\S'Sr \\\S'Sr \\\S'Sr \ S\S 'Sr\\\S 'Sr\ S \S 'Sr\S\4\S 'Sr\S\4\S'Sr\S\4\S'SrSrSr\S5r\S5r\S5r\S5r\S5r\S5r \S5r!\S5r"\S5r#\S5r$\S5r%\S5r&\S5r'\S5r(S \)4S!jr*S,S \\)4S"jjr+S#\,\,\S$4\,\)S$444S%jr-S&r.S'\)S(\4S)jr/S-S*jr0S+r1g).ParallelismConfig!aM A dataclass to configure parallelisms applied to the model. Inspired by torchtitan's `ParallelDims` https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/parallel_dims.py Args: dp_replicate_size (`int`, defaults to `1`): The size of the data parallel group. If `dp_replicate_size` is set to 1, the data parallel replication group will not be used. dp_shard_size (`int`, defaults to `1`): The size of the model shard group. If `dp_replicate_size > 1` and `tp_size > 1`, `dp_shard_size` must also be greater than 1, as composing DDP + TP is currently not supported. tp_size (`int`, defaults to `1`): The size of the tensor parallel group. If `tp_size` is set to `1`, the tensor parallel group will not be used. tp_handler (`~utils.TorchTensorParallelConfig`, defaults to `None`): The handler for the tensor parallel group. cp_size (`int`, defaults to `1`): The size of the context parallel group. Currently not supported, but reserved for future use and enabled for downstream libraries. cp_backend (`str`, defaults to `torch`): Which CP backend to use: `torch` (FSDP2) sp_size (`int`, defaults to `1`): The size of the sequence parallel group. sp_backend (`str`, defaults to `deepspeed`): Which SP backend to use:`deepspeed` (ALST/Ulysses) You may obtain different distributed data parallel paradigms by configuring `dp_replicate_size` and `dp_shard_size` together: - `dp_replicate_size == 1` and `dp_shard_size > 1`, we obtain Fully Sharded Data Parallel (FSDP). - `dp_replicate_size > 1` and `dp_shard_size > 1`, we obtain Hybrid Sharded Data Parallel (HSDP). - `dp_replicate_size > 1` and `dp_shard_size == 1` is an invalid configuration, to use pure DP, use `DistributedDataParallelKwargs` instead. Ndp_replicate_size dp_shard_sizetp_sizecp_sizetorch cp_backendsp_size deepspeed sp_backend tp_handler cp_handler sp_handlerc SURSURSURSURSURSUR SUR SURS URS URS 3$) Nz'ParallelismConfig( dp_replicate_size=z, dp_shard_size=z , tp_size=z , cp_size=z, cp_backend=z , sp_size=z, sp_backend=z, total_size=z tp_handler=z, cp_handler=z) ) rrrrrrr total_sizerrselfs Z/home/mande/repo/quber/.venv/lib/python3.13/site-packages/accelerate/parallelism_config.py__repr__ParallelismConfig.__repr__Us ##'#9#9":;#1123'' OO,-' OO,- OO,- OO,- OO,C 1 c SSKnS/nURURR5VVs0sH;up4X2;dM U[ US5(aURUR5OU_M= snn5 gs snnf)Nr device_mesh__dict__)copydeepcopyr'itemshasattr)r r(_non_serializable_fieldskvs r!to_jsonParallelismConfig.to_jsondss$1?  !MM//1 1DA4N:0F0F4==,AM1   s A; 3A; cf/nUR(aUS/- nUR(aUS/- nU$)zENames of enabled dimensions across which data parallelism is applied. dp_replicatedp_shard)dp_replicate_enableddp_shard_enabledr dimss r! dp_dim_namesParallelismConfig.dp_dim_namesqs9  $ $ ^$ $D  ZL D r$c/nUR(aUS/- nUR(aUS/- nUR(aUS/- nU$)z]Names of enabled dimensions which will receive the same batch (non-data parallel dimensions).tpcpsp) tp_enabled cp_enabled sp_enabledr6s r!non_dp_dim_names"ParallelismConfig.non_dp_dim_names{sD ?? TFND ?? TFND ?? TFND r$cf/nUR(aUS/- nUR(aUS/- nU$)zlNames of enabled dimensions which will be flattened into a joint mesh across which is model sharded in FSDP.r3r<)r5r?r6s r!dp_shard_cp_dim_names'ParallelismConfig.dp_shard_cp_dim_namess5  ZL D ?? TFND r$c/nUR(aUS/- nUR(aUS/- nUR(aUS/- nU$)z@Names of enabled dimensions across which loss should be averagedr2r3r<)r4r5r?r6s r!dp_cp_dim_names!ParallelismConfig.dp_cp_dim_namessK  $ $ ^$ $D  ZL D ?? TFND r$cD/nUR(aUS/- nUS/- nU$)z^Names of enabled dimensions across which FSDP is applied, including data parallel replication.r2 dp_shard_cp)r4r6s r!fsdp_dim_names ParallelismConfig.fsdp_dim_namess0  $ $ ^$ $D  r$cURUR-UR-UR-UR-$)zSThe total size of the parallelism configuration, which is the product of all sizes.)rrrrrrs r!rParallelismConfig.total_sizes9%%(:(::T\\IDLLX[_[g[gggr$cNURUR-UR-$)zhThe size of the non-data parallel dimensions, which is the product of tensor and context parallel sizes.)rrrrs r!non_data_parallel_size(ParallelismConfig.non_data_parallel_sizes ||dll*T\\99r$c4URUR-$)z_The size of the data parallel dimensions, which is the product of data parallel replication and)rrrs r!data_parallel_size$ParallelismConfig.data_parallel_sizes%%(:(:::r$c URS:$)zKTrue if data parallel replication is enabled, i.e. `dp_replicate_size > 1`.)rrs r!r4&ParallelismConfig.dp_replicate_enableds%%))r$c URS:$)zDTrue if data parallel sharding is enabled, i.e. `dp_shard_size > 1`.rV)rrs r!r5"ParallelismConfig.dp_shard_enableds!!A%%r$c URS:$)z:True if tensor parallelism is enabled, i.e. `tp_size > 1`.rV)rrs r!r>ParallelismConfig.tp_enabled||ar$c URS:$)z;True if context parallelism is enabled, i.e. `cp_size > 1`.rV)rrs r!r?ParallelismConfig.cp_enabledr\r$c URS:$)z;True if context parallelism is enabled, i.e. `sp_size > 1`.rV)rrs r!r@ParallelismConfig.sp_enabledr\r$c4URUR-$)z$Names of all active mesh dimensions.)r8rArs r!active_mesh_dims"ParallelismConfig.active_mesh_dimss  4#8#888r$ device_typecURS:XaURS:ag[SS5(aSSKJn O [ S5eUR 5n[U5S:XagUupEU"UUUS 9nUR(aX`RRS 5 UR(aX`RRS 5 UR(aX`RRS 5 U$) aBuilds a device mesh for the given device type based on the parallelism configuration. This method will also create required joint meshes (e.g. `dp_shard_cp`, `dp_cp`, `dp`). Args: device_type (`str`): The type of device for which to build the mesh, e rrVNz>=z2.2.0r)init_device_meshz4Building a device_mesh requires to have torch>=2.2.0)mesh_dim_namesdprJdp_cp) rrr torch.distributed.device_meshrf RuntimeError _get_meshlenr8_flattenrDrG)r rdrfmeshrg mesh_shaper&s r!build_device_mesh#ParallelismConfig.build_device_meshs ??k )dllQ.> D' * * FUV V~~ t9>%)"&  )    )) * 3 3D 9  % % 22 3 < <] K    ,, - 6 6w ?r$cURc'Ub"URU5UlUR$SeUb@URRU:wa&[SURRSUS35eUR$)Nz@You need to pass a device_type e.g cuda to build the device meshz4The device_mesh is already created with device type z@. However, you are trying to get a device mesh with device_type z<. Please check if you correctly initialized your device_mesh)r&rqrd ValueError)r rds r!get_device_mesh!ParallelismConfig.get_device_meshs    #&#'#9#9+#F ZZ&##//;>$NtO_O_OkOkNlmmnymzzvwr$return.c^URVs0sHoURU_M nn/SQm[UR5U4SjS9n[ [ U65$s snf)zQGenerate mesh shape and dimension names for torch.distributed.init_device_mesh().)r2r3r<r=r;c,>TRUS5$)Nr)index)x mesh_orders r!-ParallelismConfig._get_mesh..s:++AaD1r$)key)rb_sizessortedr*tuplezip)r parallelism mesh_dims sorted_itemsr|s @r!rlParallelismConfig._get_meshsfOSNcNcdNc{$++k"::Nc dD  OO 2 S,'((esAc URc.[[RR SS55UlUR c.[[RR SS55UlUR c.[[RR SS55UlURc.[[RR SS55UlURc%[RR SS5UlURc.[[RR SS55Ul URc%[RR S S 5Ul UR S :aURc[5Ul URS :aURc[5Ul Oq[[S 9n[!URXR5(d=[#S URSXRS[%UR535eURS :aUR&c[)5UlURS :a[#SUR35eUR S :a[#SUR 35eUR S :a[#SUR 35eURS :a[#SUR35eS/nURU;a[#SUSUR35eURS :a[#SUR35eS /nURU;a[#SUSUR35eURS :a6URS :a&[#SURSURS35eUR S :dURS :a+URS :aUR S :Xa [#S5eURUR UR URURS.Ulg)N$PARALLELISM_CONFIG_DP_REPLICATE_SIZE1 PARALLELISM_CONFIG_DP_SHARD_SIZEPARALLELISM_CONFIG_TP_SIZEPARALLELISM_CONFIG_CP_SIZEPARALLELISM_CONFIG_CP_BACKENDrPARALLELISM_CONFIG_SP_SIZEPARALLELISM_CONFIG_SP_BACKENDrrV)rzParallelismConfig's cp_backend=z requires z, but cp_handler was set to z.dp_replicate_size must be at least 1, but got z*dp_shard_size must be at least 1, but got z$tp_size must be at least 1, but got z$cp_size must be at least 1, but got zcp_backend must be one of z , but got z$sp_size must be at least 1, but got zsp_backend must be one of z[Context Parallelism (CP) and Sequence Parallelism (SP) are mutually exclusive. Got cp_size=z and sp_size=z+. Please set either cp_size=1 or sp_size=1.aCTensor/Context parallelism (tp/cp_size > 1) cannot be used with pure data parallelism (dp_replicate_size > 1 and dp_shard_size == 1). Please set dp_shard_size > 1 and dp_replicate_size == 1 to compose FSDP + TP/CP for 2D parallel, or set dp_replicate_size == 1 and dp_shard_size > 1 to compose HSDP + TP/CP for 3D parallel.)r2r3r;r<r=)rintosenvirongetrrrrrrrr rr dict isinstancerttyperrr)r cp_backends_config_mapvalid_cp_backendsvalid_sp_backendss r! __post_init__ParallelismConfig.__post_init__s  ! ! )%(8^`c)d%eD "    %!$RZZ^^4VX[%\!]D  << rzz~~.JCPQDL << rzz~~.JCPQDL ?? " jjnn-LgVDO << rzz~~.JCPQDL ?? " jjnn-LkZDO <)-4*&"$//3I//3Z[[$9$//9J*Ukl{l{U|T}~Z[_`d`o`o[pZqr <RS S <RS S$I ??"3 39:K9LJW[WfWfVghi i <RS S(M ??"3 39:K9LJW[WfWfVghi i <;+D+D E ??k )dllQ.>  __ 9 9 900AB""-";";!<=9:  ??Q   ''++/H/HHbcnccb@@AB "&!2!2!4 KqyWT](+CTJV ( 5J;-XHI"5 44 MMADIIiDXX 59r$) rrrrr&rrrrrrr)N)rr )2__name__ __module__ __qualname____firstlineno____doc__rrr__annotations__rrrrrrrrrr rr rrr&r"r/propertyr8rArDrGrKrrPrSr4r5r>r?r@rbstrrqrurrlrrr__static_attributes__r$r!rr!s;!F(,x}+#'M8C='!GXc]!!GXc]!#'J '!GXc]!'+J $+:>Jd556=:>Jd667>?CJd;;<CK        hh::;;**&&      99!S!F  8C=   )5sCx%S/!AB )J X3S33 +r$r)rr dataclassesrtypingrrrraccelerate.utils.dataclassesrr r r accelerate.utils.versionsr accelerater rrr$r!rsI !:: 7& ll lr$