Upload pyarrow/PYARROW_USER_GUIDE.txt with huggingface_hub

pyarrow/PYARROW_USER_GUIDE.txt

@@ -0,0 +1,402 @@
+================================================================================
+  PYARROW - USER GUIDE (Android Python STB) - Generated by RIMI
+================================================================================
+  Covers: what pyarrow is, install/verify, arrays, tables, types, IPC,
+          CSV, JSON, Feather, compute, filesystem, and Android-specific notes.
+
+  Written for: Python 3.12 (RIMI build) on Android
+  Version:     pyarrow 25.0.1 (Arrow C++ 25.0.1)
+  Scripts dir: /storage/emulated/0/PythonSTB/Scripts/
+  Installed:   /data/user/0/com.pythonstb.rimi/files/python/lib/python3.12/site-packages/
+================================================================================
+
+
+--------------------------------------------------------------------------------
+1) WHAT IS PYARROW?
+--------------------------------------------------------------------------------
+
+PyArrow is the Python bindings for Apache Arrow — a cross-language development
+platform for in-memory columnar data. It provides:
+
+  - Apache Arrow arrays and tables (columnar memory format)
+  - Zero-copy data interchange between Python, pandas, and other languages
+  - IPC (Inter-Process Communication) for fast serialization
+  - CSV, JSON, and Feather file readers/writers
+  - Compute kernels (sum, mean, cast, filter, sort, etc.)
+  - A local filesystem abstraction
+
+Arrow is the foundation behind many modern data tools:
+  pandas 2.x uses Arrow as its default backend
+  Polars, DuckDB, and Spark all speak Arrow natively
+
+    import pyarrow as pa
+    print(pa.__version__)          # 25.0.1
+    print(pa.cpp_version)          # 25.0.1
+
+
+--------------------------------------------------------------------------------
+2) INSTALL / VERIFY
+--------------------------------------------------------------------------------
+
+Install:
+    pip install pyarrow-25.0.1-cp312-cp312-android_24_x86_64.whl
+        or:  pyarrow-25.0.1-cp312-cp312-android_24_arm64_v8a.whl
+
+Quick smoke test:
+    import pyarrow as pa
+    a = pa.array([1, 2, 3])
+    print(a)                       # [<pyarrow.Int64Scalar: 1>, ...]
+    print("version:", pa.__version__)
+    print("cpp:", pa.cpp_version)
+
+Expected output:
+    [1, 2, 3]
+    version: 25.0.1
+    cpp: 25.0.1
+
+
+--------------------------------------------------------------------------------
+3) ARRAYS - CREATION BASICS
+--------------------------------------------------------------------------------
+
+    import pyarrow as pa
+
+    # from Python lists
+    a = pa.array([1, 2, 3])                      # int64
+    b = pa.array([1.0, 2.5, 3.7])               # float64
+    c = pa.array(["hello", "world"])             # string
+    d = pa.array([True, False, True])            # bool
+
+    # explicit type
+    e = pa.array([1, 2, 3], type=pa.int32())
+    f = pa.array([1, 2, 3], type=pa.float64())
+
+    # with nulls
+    g = pa.array([1, None, 3], type=pa.int64())
+    g.null_count                # 1
+
+    # from numpy
+    import numpy as np
+    h = pa.array(np.arange(10))   # numpy -> Arrow (zero-copy for numeric types)
+
+    # important attributes
+    a.type          # DataType: int64
+    a.dtype         # same as .type
+    len(a)          # 3
+    a.to_pylist()   # [1, 2, 3]
+    a.as_py()       # [1, 2, 3] (same for scalar)
+
+
+--------------------------------------------------------------------------------
+4) TYPES
+--------------------------------------------------------------------------------
+
+    pa.int8()       pa.int16()      pa.int32()      pa.int64()
+    pa.uint8()      pa.uint16()     pa.uint32()     pa.uint64()
+    pa.float16()    pa.float32()    pa.float64()
+    pa.bool_()
+    pa.string()     pa.large_string()
+    pa.binary()     pa.large_binary()
+    pa.date32()     pa.date64()
+    pa.timestamp("ns")             # nanosecond timestamp
+    pa.duration("ms")              # millisecond duration
+    pa.null()                      # all-null type
+    pa.struct([pa.field("x", pa.int64()), pa.field("y", pa.string())])
+
+    # check type
+    a = pa.array([1, 2])
+    a.type == pa.int64()           # True
+    isinstance(a.type, pa.DataType) # True
+
+    # convert between types
+    a = pa.array([1, 2, 3], type=pa.int64())
+    b = a.cast(pa.float64())       # explicit cast
+    c = a.cast(pa.int32())         # downcast
+
+
+--------------------------------------------------------------------------------
+5) TABLES
+--------------------------------------------------------------------------------
+
+    import pyarrow as pa
+
+    # from dict
+    t = pa.table({
+        "id":    [1, 2, 3],
+        "name":  ["alice", "bob", "charlie"],
+        "score": [95.5, 87.0, 92.3],
+    })
+
+    t.num_rows              # 3
+    t.num_columns           # 3
+    t.column_names          # ["id", "name", "score"]
+    t.schema                # id: int64, name: string, score: float64
+
+    # access columns
+    t.column("id")          # <pyarrow.lib.ChunkedArray ...>
+    t.column("id").to_pylist()  # [1, 2, 3]
+
+    # convert to pandas
+    df = t.to_pandas()
+    print(df)
+    #    id     name  score
+    # 0   1    alice   95.5
+    # 1   2      bob   87.0
+    # 2   3  charlie   92.3
+
+    # from pandas
+    t2 = pa.Table.from_pandas(df)
+
+    # slice
+    t.slice(0, 2)           # first 2 rows
+
+    # combine tables
+    t3 = pa.concat_tables([t, t])
+
+    # sort
+    t.sort_by("score", descending=True)
+
+
+--------------------------------------------------------------------------------
+6) IPC (Inter-Process Communication)
+--------------------------------------------------------------------------------
+
+IPC is Arrow's fast binary serialization format — much faster than CSV or JSON.
+
+    import pyarrow as pa
+    import pyarrow.ipc as ipc
+
+    t = pa.table({"x": [1, 2, 3], "y": ["a", "b", "c"]})
+
+    # --- File format (random access) ---
+    path = "/storage/emulated/0/Download/data.arrow"
+    writer = ipc.new_file(path, t.schema)
+    writer.write_table(t)
+    writer.close()
+
+    reader = ipc.open_file(path)
+    t2 = reader.read_all()
+    assert t.equals(t2)
+
+    # --- Stream format (sequential access, smaller header) ---
+    path2 = "/storage/emulated/0/Download/data_stream.arrow"
+    writer = ipc.new_stream(path2, t.schema)
+    writer.write_table(t)
+    writer.close()
+
+    reader = ipc.open_stream(path2)
+    t3 = reader.read_all()
+    assert t.equals(t3)
+
+    # --- In-memory IPC (for zero-copy between processes) ---
+    sink = ipc.BufferOutputStream()
+    writer = ipc.new_stream(sink, t.schema)
+    writer.write_table(t)
+    writer.close()
+    buf = sink.getvalue().to_pybytes()   # bytes
+    # send buf over socket / pipe, then:
+    reader = ipc.open_stream(pa.BufferReader(buf))
+    t4 = reader.read_all()
+
+
+--------------------------------------------------------------------------------
+7) CSV
+--------------------------------------------------------------------------------
+
+    import pyarrow as pa
+    import pyarrow.csv as pcsv
+
+    t = pa.table({"a": [1, 2], "b": [3.0, 4.0]})
+
+    # write
+    pcsv.write_csv(t, "/storage/emulated/0/Download/data.csv")
+
+    # read (basic)
+    t2 = pcsv.read_csv("/storage/emulated/0/Download/data.csv")
+
+    # read with options
+    read_opts = pcsv.ReadOptions(column_names=["x", "y"])
+    convert_opts = pcsv.ConvertOptions(
+        column_types={"x": pa.int64(), "y": pa.float64()},
+        null_values=["NA", "NULL"],
+    )
+    t3 = pcsv.read_csv(
+        "/storage/emulated/0/Download/data.csv",
+        read_options=read_opts,
+        convert_options=convert_opts,
+    )
+
+    # read as pandas directly
+    df = pcsv.read_csv("/storage/emulated/0/Download/data.csv").to_pandas()
+
+
+--------------------------------------------------------------------------------
+8) JSON
+--------------------------------------------------------------------------------
+
+    import pyarrow as pa
+    import pyarrow.json as pjson
+
+    # JSON Lines (one JSON object per line) — preferred for tabular data
+    # file content:
+    # {"a": 1, "b": "hello"}
+    # {"a": 2, "b": "world"}
+
+    t = pjson.read_json("/storage/emulated/0/Download/data.json")
+    # t: 2 rows, columns ["a", "b"]
+    # types inferred automatically (a: int64, b: string)
+
+
+--------------------------------------------------------------------------------
+9) FEATHER (fast columnar format)
+--------------------------------------------------------------------------------
+
+Feather is Arrow's columnar format optimized for pandas read/write.
+Faster than CSV, smaller than IPC for single-table files.
+
+    import pyarrow as pa
+    import pyarrow.feather as pf
+
+    t = pa.table({"x": [1, 2, 3], "y": [4.0, 5.0, 6.0]})
+
+    # write
+    pf.write_feather(t, "/storage/emulated/0/Download/data.feather")
+
+    # read
+    t2 = pf.read_feather("/storage/emulated/0/Download/data.feather")
+    assert t.equals(t2)
+
+    # read as pandas
+    df = pf.read_feather("/storage/emulated/0/Download/data.feather")
+
+    # older Feather v1 format also supported (pandas read_feather兼容)
+
+
+--------------------------------------------------------------------------------
+10) COMPUTE FUNCTIONS
+--------------------------------------------------------------------------------
+
+    import pyarrow as pa
+    import pyarrow.compute as pc
+
+    a = pa.array([1, 2, 3, 4, 5])
+
+    # aggregations
+    pc.sum(a)            # 15
+    pc.mean(a)           # 3.0
+    pc.min(a)            # 1
+    pc.max(a)            # 5
+    pc.count(a)          # 5
+
+    # element-wise
+    pc.add(a, pa.array([10, 10, 10, 10, 10]))
+    pc.multiply(a, pa.scalar(2))
+    pc.sqrt(a.cast(pa.float64()))
+    pc.abs(pa.array([-1, 2, -3]))
+
+    # comparison / filtering
+    mask = pc.greater(a, 3)               # [False, False, False, True, True]
+    pc.filter(a, mask)                    # [4, 5]
+    pc.sum(mask.as_py()) if hasattr(mask, 'as_py') else pc.sum(mask)  # 2
+
+    # casting
+    b = pc.cast(a, to=pa.float64())
+    c = pc.cast(pa.array(["1", "2", "3"]), to=pa.int64())
+
+    # sort
+    pc.sort(pa.array([3, 1, 4, 1, 5]))   # [1, 1, 3, 4, 5]
+
+    # unique / value_counts
+    pc.unique(pa.array([1, 2, 1, 3, 2]))  # [1, 2, 3]
+    pc.value_counts(pa.array([1, 1, 2]))  # [{values: [1,2], counts: [2,1]}]
+
+    # string ops (via compute)
+    names = pa.array(["alice", "BOB", "charlie"])
+    pc.upper(names)                        # ["ALICE", "BOB", "CHARLIE"]
+    pc.utf8_length(names)                  # [5, 3, 7]
+    pc.utf8_lower(names)                   # ["alice", "bob", "charlie"]
+
+
+--------------------------------------------------------------------------------
+11) FILESYSTEM
+--------------------------------------------------------------------------------
+
+    import pyarrow.fs as pfs
+
+    # local filesystem
+    local = pfs.LocalFileSystem()
+
+    # list directory
+    local.get_file_info(pfs.FileSelector("/storage/emulated/0/Download"))
+
+    # file info
+    meta = local.get_file_info("/storage/emulated/0/Download/data.csv")
+    meta.type    # FileType.File
+    meta.size    # bytes
+    meta.mtime   # modification time
+
+    # read / write
+    with local.open_output_stream("/storage/emulated/0/Download/test.txt") as f:
+        f.write(b"hello arrow")
+    with local.open_input_stream("/storage/emulated/0/Download/test.txt") as f:
+        content = f.read()
+    # content == b"hello arrow"
+
+    # create / delete
+    local.create_dir("/storage/emulated/0/Download/testdir")
+    local.delete_dir("/storage/emulated/0/Download/testdir")
+
+
+--------------------------------------------------------------------------------
+12) ANDROID-SPECIFIC NOTES
+--------------------------------------------------------------------------------
+
+  - libarrow_python.so preload:
+    The __init__.py is patched to call ctypes.CDLL("libarrow_python.so") before
+    importing any Cython modules. This is required because Android's linker
+    cannot find shared libraries the same way as Linux desktop. The preload
+    ensures symbols from libarrow_python.so are available when the .so extension
+    modules load.
+
+  - Static Arrow C++:
+    Arrow C++ (libarrow.a, libarrow_compute.a) is linked statically into
+    pyarrow's .so files. Only libarrow_python.so is a separate shared library
+    that needs preloading.
+
+  - Dependencies:
+    Only numpy is required. Install numpy first before using pyarrow.
+
+  - Wheels are tagged android_24_arm64_v8a and android_24_x86_64.
+    Install the wheel matching your device ABI:
+      arm64 phone -> android_24_arm64_v8a
+      x86_64 emulator -> android_24_x86_64
+
+  - Temp files for IPC/CSV/Feather:
+    Write to /storage/emulated/0/Download/ or use tempfile.gettempdir().
+
+
+--------------------------------------------------------------------------------
+13) TESTED FEATURES
+--------------------------------------------------------------------------------
+
+  v  import pyarrow 25.0.1
+  v  import numpy (dependency)
+  v  import submodules: compute, csv, json, feather, fs, ipc
+  v  array creation: int64, float64, string
+  v  null handling
+  v  type system: int64, float64, string, bool
+  v  table creation, schema inspection
+  v  table column access + to_pandas
+  v  IPC file round-trip
+  v  IPC stream round-trip
+  v  CSV write / read
+  v  CSV custom options (column_names, column_types)
+  v  JSON read
+  v  Feather round-trip
+  v  compute: sum, cast, filter, add, multiply
+  v  LocalFileSystem get_file_info
+
+
+================================================================================
+  Generated by RIMI
+================================================================================