]j_SrSSKJr SSKrSSKrSSKrSSKJrJrJ r SSK J r J r SSK Jr SSKJr SSKJrJr SS KJr SS KJr SS KJrJr SS KJr SS KJrJrJ r J!r!J"r" \(aSSK#J$r$J%r%J&r&J'r'J(r(J)r) SSjr*SSSjjr+"SS5r,"SS\,5r-"SS\,5r.SSSjjr/\"S5SSS\R`SSS4S Sjj5r1g)!zparquet compat) annotationsN) TYPE_CHECKINGAnyLiteral)catch_warningsfilterwarnings)lib)import_optional_dependency)AbstractMethodErrorPandas4Warning) set_module)check_dtype_backend) DataFrame get_option)arrow_table_to_pandas) IOHandles get_handle is_fsspec_urlis_urlstringify_path) DtypeBackendFilePathParquetCompressionOptions ReadBufferStorageOptions WriteBufferBaseImplc2US:Xa [S5nUS:Xa-[[/nSnUH nU"5s $ [SU35eUS:Xa [5$US:Xa [5$[ S 5e![anUS[ U5-- nSnAMiSnAff=f) zreturn our implementationautozio.parquet.enginez - NzUnable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. A suitable version of pyarrow or fastparquet is required for parquet support. Trying to import the above resulted in these errors:pyarrow fastparquetz.engine must be one of 'pyarrow', 'fastparquet')r PyArrowImplFastParquetImpl ImportErrorstr ValueError)engineengine_classes error_msgs engine_classerrs N/home/mande/repo/quber/.venv/lib/python3.13/site-packages/pandas/io/parquet.py get_enginer.4s /0 %7 *L 1#~%+  C l    } =   E FF% 1gC00  1sA00 B:BBc[U5nUb[SSS9n[SSS9nUb-[XR5(aU(a [ S5eOIUb%[XR R 5(aO![S[U5R35e[U5(aqUcnUc4[S5n[S5nURRU5upUc3[S5nURR"U40U=(d 0D6upO(U(a![!U5(aUS :wa [S 5eSn U(dYU(dR[U["5(a=[$R&R)U5(d[+XSS US 9n SnU R,nXYU4$![UR4a Nf=f) zFile handling for PyArrow.Nz pyarrow.fsignore)errorsfsspecz8storage_options not supported with a pyarrow FileSystem.z9filesystem must be a pyarrow or fsspec FileSystem, not a r!rbz8storage_options passed with buffer, or non-supported URLFis_textstorage_options)rr isinstance FileSystemNotImplementedErrorspecAbstractFileSystemr'type__name__rfrom_uri TypeError ArrowInvalidcore url_to_fsrr&ospathisdirrhandle) rDfsr6modeis_dirpath_or_handlepa_fsr2pahandless r-_get_path_or_handlerNVs$D)N ~*<I+HXF  B0@0@!A!A)N Jr;;3Q3Q$R$R b**+- ^$$  "+I6B.|%>t%D" :/9F!'!6!6"#2#8b" B &"8"8DDLSTTG  ~s + + n--  %   B &&7r/  s F++GGc@\rSrSr\SSj5rSSjrSS SjjrSrg) rcD[U[5(d [S5eg)Nz+to_parquet only supports IO with DataFrames)r7rr')dfs r-validate_dataframeBaseImpl.validate_dataframes "i((JK K)c [U5eNr )selfrRrD compressionkwargss r-writeBaseImpl.write !$''rUNc [U5erWrX)rYrDcolumnsr[s r-read BaseImpl.readr^rU)rRrreturnNonerW)rdr) r= __module__ __qualname____firstlineno__ staticmethodrSr\ra__static_attributes__rcrUr-rrs%LL(((rUc\rSrSrSSjrSS SjjrSS\RSSS4S SjjrSr g) r#c4[SSS9 SSKnSSKnXlg)Nr!z(pyarrow is required for parquet support.extrar)r pyarrow.parquet(pandas.core.arrays.arrow.extension_typesapi)rYr!pandass r-__init__PyArrowImpl.__init__s" G   8rUNc URU5 SURSS50n UbXIS'URRR"U40U D6n UR (aOS[ R"UR 50n U RRn 0U EU En U RU 5n [UUUSUSLS9upn[U[R5(a|[US5(ak[UR ["[$45(aF[UR [$5(aUR R'5nO UR nUb-URR(R*"U U4UUUS.UD6 O+URR(R,"U U4UUS.UD6 UbUR/5 gg!UbUR/5 ff=f) Nschemapreserve_index PANDAS_ATTRSwb)r6rHrIname)rZpartition_cols filesystem)rZr})rSpoprrTable from_pandasattrsjsondumpsrwmetadatareplace_schema_metadatarNr7ioBufferedWriterhasattrr{r&bytesdecodeparquetwrite_to_dataset write_tableclose)rYrRrDrZindexr6r|r}r[from_pandas_kwargstable df_metadataexisting_metadatamerged_metadatarJrMs r-r\PyArrowImpl.writes #.6 8T8R-S  38/ 0**2D1CD 88)4::bhh+?@K % 5 5 B!2BkBO11/BE.A  +!- / + ~r'8'8 9 9//>..e ==.--u55!/!4!4!;!;!=!/!4!4 )  11"!,#1)     ,,"!,)   " #w" #s "AGG(c 2SUS'[UUUSS9upnURRR"U 4UUUS.UD6n [ 5 [ SS[ 5 [U UUS9n SSS5 U RR(aNS U RR;a4U RRS n [R"U 5W l W U bU R5 $$!,(df  N=f!U bU R5 ff=f) NTuse_pandas_metadatar3)r6rH)r`r}filtersr0make_block is deprecated) dtype_backendto_pandas_kwargss PANDAS_ATTRS)rNrrr read_tablerrr rrwrrloadsrr)rYrDr`rrr6r}rr[rJrMpa_tableresultrs r-raPyArrowImpl.reads)-$%.A  + / +  xx''22%   H !." /"/%5 "''"hoo&>&>>"*//":":?"KK#'::k#:FL" #%"!$" #s$5D C/(A2D/ C=9DDrrrdresnappyNNNN)rRrrDzFilePath | WriteBuffer[bytes]rZrr bool | Noner6StorageOptions | Noner|list[str] | Nonerdre)rDtypeBackend | lib.NoDefaultr6rrzdict[str, Any] | Nonerdr) r=rfrgrhrtr\r no_defaultrarjrcrUr-r#r#s 2:!15+/@ @ ,@ / @  @ / @ )@  @ J69nn1526. 4 . / . 0.  . . rUr#cf\rSrSrSSjrSS SjjrS S SjjrSrg) r$i!c$[SSS9nXlg)Nr"z,fastparquet is required for parquet support.rn)r rr)rYr"s r-rtFastParquetImpl.__init__"s1 !O rUNc ^^ URU5 SU;aUb [S5eSU;aURS5nUbSUS'Ub [S5e[ U5n[ U5(a[ S5m U U4SjUS'OT(a [S 5e[S S 9 URR"UU4UUUS .UD6 SSS5 g!,(df  g=f) N partition_onzYCannot use both partition_on and partition_cols. Use partition_cols for partitioning datahive file_scheme9filesystem is not implemented for the fastparquet engine.r2cZ>TR"US40T=(d 0D6R5$)Nrz)open)rD_r2r6s r-'FastParquetImpl.write..Ms,&++d3.4"3df3rU open_withz?storage_options passed with file object or non-fsspec file pathT)record)rZ write_indexr) rSr'r~r9rrr rrrr\) rYrRrDrZrr|r6r}r[r2s ` @r-r\FastParquetImpl.write*s # V #(BK  V ##ZZ7N  %$*F= !  !%K  d#   /9F#F; Q 4 ( HHNN (!+     ) ( (s !#C  Cc b0nURS[R5n SUS'U [RLa [S5eUb [ S5eUb [ S5e[ U5nSn [ U5(a6[S5n U R"US40U=(d 0D6RUS 'OQ[U[5(a<[RRU5(d[USSUS 9n U R nUR"R$"U40UD6n ['5 [)S S [*5 U R,"SX#S .UD6sSSS5 U bU R/5 $$!,(df  O=fU bU R/5 gg!U bU R/5 ff=f)NrF pandas_nullszHThe 'dtype_backend' argument is not supported for the fastparquet enginerz?to_pandas_kwargs is not implemented for the fastparquet engine.r2r3rGr4r0r)r`rrc)r~r rr'r9rrr rrGr7r&rCrDrErrFrr ParquetFilerrr to_pandasr) rYrDr`rr6r}rr[parquet_kwargsrrMr2 parquet_files r-raFastParquetImpl.read_s*, ?CNNC ).~&  .%   !%K   '%Q d#   /9F#);;tT#Uo>SQS#U#X#XN4 c " "277==+>+>!dE?G>>D 88//GGL!." $--#8> "!" #"!!" #w" #s$'F.%E1 F1 E?;FF.rrr)rRrrZz*Literal['snappy', 'gzip', 'brotli'] | Noner6rrdre)NNNNN)r6rr dict | Nonerdr)r=rfrgrhrtr\rarjrcrUr-r$r$!sCK153 3@ 3/3 3p15(,7 / 7 &7  7 7 rUr$rc [U[5(aU/n[U5n Uc[R"5OUn U R "UU 4UUUUUS.UD6 Uc1[U [R5(deU R 5$g)a Write a DataFrame to the parquet format. Parameters ---------- df : DataFrame path : str, path object, file-like object, or None, default None String, path object (implementing ``os.PathLike[str]``), or file-like object implementing a binary ``write()`` function. If None, the result is returned as bytes. If a string, it will be used as Root Directory path when writing a partitioned dataset. The engine fastparquet does not accept file-like objects. engine : {'auto', 'pyarrow', 'fastparquet'}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. When using the ``'pyarrow'`` engine and no storage options are provided and a filesystem is implemented by both ``pyarrow.fs`` and ``fsspec`` (e.g. "s3://"), then the ``pyarrow.fs`` filesystem is attempted first. Use the filesystem keyword with an instantiated fsspec filesystem if you wish to use its implementation. compression : {'snappy', 'gzip', 'brotli', 'lz4', 'zstd', None}, default 'snappy'. Name of the compression to use. Use ``None`` for no compression. index : bool, default None If ``True``, include the dataframe's index(es) in the file output. If ``False``, they will not be written to the file. If ``None``, similar to ``True`` the dataframe's index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn't require much space and is faster. Other indexes will be included as columns in the file output. partition_cols : str or list, optional, default None Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. storage_options : dict, optional Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to ``urllib.request.Request`` as header options. For other URLs (e.g. starting with "s3://", and "gcs://") the key-value pairs are forwarded to ``fsspec.open``. Please see ``fsspec`` and ``urllib`` for more details, and for more examples on storage options refer `here `_. filesystem : fsspec or pyarrow filesystem, default None Filesystem object to use when reading the parquet file. Only implemented for ``engine="pyarrow"``. .. versionadded:: 2.1.0 **kwargs Additional keyword arguments passed to the engine: * For ``engine="pyarrow"``: passed to :func:`pyarrow.parquet.write_table` or :func:`pyarrow.parquet.write_to_dataset` (when using partition_cols) * For ``engine="fastparquet"``: passed to :func:`fastparquet.write` Returns ------- bytes if no path argument is provided else None N)rZrr|r6r})r7r&r.rBytesIOr\getvalue) rRrDr(rZrr6r|r}r[impl path_or_bufs r- to_parquetrsV.#&&() f DAESWKJJ   %'    |+rzz2222##%%rUrsc b[U5n [U5 U R"U4UUUUUUS.UD6$)aM Load a parquet object from the file path, returning a DataFrame. The function automatically handles reading the data from a parquet file and creates a DataFrame with the appropriate structure. Parameters ---------- path : str, path object or file-like object String, path object (implementing ``os.PathLike[str]``), or file-like object implementing a binary ``read()`` function. The string could be a URL. Valid URL schemes include http, ftp, s3, gs, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.parquet``. A file URL can also be a path to a directory that contains multiple partitioned parquet files. Both pyarrow and fastparquet support paths to directories as well as file URLs. A directory path could be: ``file://localhost/path/to/tables`` or ``s3://bucket/partition_dir``. engine : {'auto', 'pyarrow', 'fastparquet'}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. When using the ``'pyarrow'`` engine and no storage options are provided and a filesystem is implemented by both ``pyarrow.fs`` and ``fsspec`` (e.g. "s3://"), then the ``pyarrow.fs`` filesystem is attempted first. Use the filesystem keyword with an instantiated fsspec filesystem if you wish to use its implementation. columns : list, default=None If not None, only these columns will be read from the file. storage_options : dict, optional Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to ``urllib.request.Request`` as header options. For other URLs (e.g. starting with "s3://", and "gcs://") the key-value pairs are forwarded to ``fsspec.open``. Please see ``fsspec`` and ``urllib`` for more details, and for more examples on storage options refer `here `_. dtype_backend : {'numpy_nullable', 'pyarrow'} Back-end data type applied to the resultant :class:`DataFrame` (still experimental). If not specified, the default behavior is to not use nullable data types. If specified, the behavior is as follows: * ``"numpy_nullable"``: returns nullable-dtype-backed :class:`DataFrame` * ``"pyarrow"``: returns pyarrow-backed nullable :class:`ArrowDtype` :class:`DataFrame` .. versionadded:: 2.0 filesystem : fsspec or pyarrow filesystem, default None Filesystem object to use when reading the parquet file. Only implemented for ``engine="pyarrow"``. .. versionadded:: 2.1.0 filters : List[Tuple] or List[List[Tuple]], default None To filter out data. Filter syntax: [[(column, op, val), ...],...] where op is [==, =, >, >=, <, <=, !=, in, not in] The innermost tuples are transposed into a set of filters applied through an `AND` operation. The outer list combines these sets of filters through an `OR` operation. A single list of tuples can also be used, meaning that no `OR` operation between set of filters is to be conducted. Using this argument will NOT result in row-wise filtering of the final partitions unless ``engine="pyarrow"`` is also specified. For other engines, filtering is only performed at the partition level, that is, to prevent the loading of some row-groups and/or files. .. versionadded:: 2.1.0 to_pandas_kwargs : dict | None, default None Keyword arguments to pass through to :func:`pyarrow.Table.to_pandas` when ``engine="pyarrow"``. .. versionadded:: 3.0.0 **kwargs Additional keyword arguments passed to the engine: * For ``engine="pyarrow"``: passed to :func:`pyarrow.parquet.read_table` * For ``engine="fastparquet"``: passed to :meth:`fastparquet.ParquetFile.to_pandas` Returns ------- DataFrame DataFrame based on parquet file. See Also -------- DataFrame.to_parquet : Create a parquet object that serializes a DataFrame. Examples -------- >>> original_df = pd.DataFrame({"foo": range(5), "bar": range(5, 10)}) >>> original_df foo bar 0 0 5 1 1 6 2 2 7 3 3 8 4 4 9 >>> df_parquet_bytes = original_df.to_parquet() >>> from io import BytesIO >>> restored_df = pd.read_parquet(BytesIO(df_parquet_bytes)) >>> restored_df foo bar 0 0 5 1 1 6 2 2 7 3 3 8 4 4 9 >>> restored_df.equals(original_df) True >>> restored_bar = pd.read_parquet(BytesIO(df_parquet_bytes), columns=["bar"]) >>> restored_bar bar 0 5 1 6 2 7 3 8 4 9 >>> restored_bar.equals(original_df[["bar"]]) True The function uses `kwargs` that are passed directly to the engine. In the following example, we use the `filters` argument of the pyarrow engine to filter the rows of the DataFrame. Since `pyarrow` is the default engine, we can omit the `engine` argument. Note that the `filters` argument is implemented by the `pyarrow` engine, which can benefit from multithreading and also potentially be more economical in terms of memory. >>> sel = [("foo", ">", 2)] >>> restored_part = pd.read_parquet(BytesIO(df_parquet_bytes), filters=sel) >>> restored_part foo bar 0 3 8 1 4 9 )r`rr6rr}r)r.rra) rDr(r`r6rr}rrr[rs r- read_parquetrsL@ f D & 99  '#)    rU)r(r&rdr)Nr3F) rDz1FilePath | ReadBuffer[bytes] | WriteBuffer[bytes]rGrr6rrHr&rIboolrdzVtuple[FilePath | ReadBuffer[bytes] | WriteBuffer[bytes], IOHandles[bytes] | None, Any])NrrNNNN)rRrrDz$FilePath | WriteBuffer[bytes] | Noner(r&rZrrrr6rr|rr}rrdz bytes | None)rDzFilePath | ReadBuffer[bytes]r(r&r`rr6rrrr}rrz&list[tuple] | list[list[tuple]] | Nonerrrdr)2__doc__ __future__rrrrCtypingrrrwarningsrr pandas._libsr pandas.compat._optionalr pandas.errorsr r pandas.util._decoratorsr pandas.util._validatorsrrsrrpandas.io._utilrpandas.io.commonrrrrrpandas._typingrrrrrrr.rNrr#r$rrrrcrUr-rs"   >/7 2GJ.2 <' ;<' <'+<'  <'  <'  <'~ ( (| (| ~u hu t26-5-1'+`` .` `+ `  ` + `%```F H $-125..6:$(k &k kk+ k 0 k  k4k"kkkrU