Feature proposal: Stacking, potentially heterogeneous, datasets

### Introduction

Hello there,
I noticed that there are two ways to combine multiple datasets: Either through `datasets.concatenate_datasets` or `datasets.interleave_datasets`. However, to my knowledge (please correct me if I am wrong) both approaches require the datasets that are combined to have the same features.

I think it would be a great idea to add support for combining multiple datasets that might not follow the same schema (i.e. have different features), for example an image and text dataset. That is why I propose a third function of the `datasets.combine` module called `stack_datasets`, which can be used to combine a list of datasets with (potentially) different features. This would look as follows:

```python
>>> from datasets import stack_datasets
>>> image_dataset = ...
>>> next(iter(image_dataset))
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=555x416 at 0x313E79CD0> }
>>> text_dataset = ...
>>> next(iter(text_dataset))
{'text': "This is a test."}
>>> stacked = stack_datasets(datasets={'i_ds': image_dataset, 't_ds': text_dataset}, stopping_strategy='all_exhausted')
>>> next(iter(stacked))
{
'i_ds': {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=555x416 at 0x313E79CD0> }
't_ds': {'text': "This is a test."}
}
```

### Motivation

I motivate this by:

**A**: The fact that Pytorch offers a similar functionality under `torch.utils.data.StackDataset` ([link](https://pytorch.org/docs/stable/data.html#torch.utils.data.StackDataset)).

**B**: In settings where one would like to e.g. train a Vision-Language model using an image-text dataset, an image dataset, and a text dataset, this functionality would offer a clean and intuitive solution to create multimodal datasets. I am aware that the aforementioned is also feasible without my proposed function, but I believe this offers a nice approach that aligns with existing functionality and is directly provided within the `datasets` package.

### API
`stack_datasets` has two arguments: `datasets` and `stopping_strategy `.

`datasets` is a dictionary of either type `Dict[str, Dataset]` or `Dict[str, IterableDatasets]`, a mixture is not allowed. It contains the names of the datasets (the keys) and the datasets themselves (the values) that should be stacked. Each item returned is a dictionary with one key-value pair for each dataset. The keys are the names of the datasets as provided in the argument `datasets`, and the values are the respective examples from the datasets.

`stopping_strategy` is the same as for `interleave_datasets`. If it is `first_exhausted` we stop if the smallest dataset runs out of examples, if it is `all_exhausted` we stop if all datasets ran out of examples at least once. For `all_exhausted` that means that we may visit examples from datasets multiple times.

### Docs
I saw that there are multiple documentations and guides on the HuggingFace website that introduce `concatenate_datasets` and `interleave_datasets`, for example [here](https://huggingface.co/docs/datasets/process#concatenate). If this request is merged I would be willing to add the new functionality at the appropriate points in the documentation (if desired).

### Tests
I also added some tests to ensure correctness. Some tests I wrote in [tests/test_iterable_dataset.py](https://github.com/TimCares/datasets/blob/fadc1159debf2a65d44e40cbf7758f2bd2cc8b08/tests/test_iterable_dataset.py#L2169)
run for both `Dataset` and `IterableDataset` even though tests for `Dataset` technically do not belong in this script, but I found that this was a nice way to cover more cases with mostly the same code.

### Additional information
I tried to write the code in a way so that it is similar to that of `concatenate_datasets` and `interleave_datasets`.
I’m open to feedback and willing to make adjustments based on your suggestions, so feel free to give me your take. :)


Let soundfile directly read local audio files

- [x] Fixes #7276


Add link to video dataset

This PR updates https://huggingface.co/docs/datasets/loading to also link to the new video loading docs.

cc @mfarre


Accessing audio dataset value throws Format not recognised error

### Describe the bug

Accessing audio dataset value throws `Format not recognised error`

### Steps to reproduce the bug

**code:**
```py
from datasets import load_dataset

dataset = load_dataset("fawazahmed0/bug-audio")

for data in dataset["train"]:
    print(data)

```

**output:**
```bash
(mypy) C:\Users\Nawaz-Server\Documents\ml>python myest.py
[C:\vcpkg\buildtrees\mpg123\src\0d8db63f9b-3db975bc05.clean\src\libmpg123\layer3.c:INT123_do_layer3():1801] error: dequantization failed!
{'audio': {'path': 'C:\\Users\\Nawaz-Server\\.cache\\huggingface\\hub\\datasets--fawazahmed0--bug-audio\\snapshots\\fab1398431fed1c0a2a7bff0945465bab8b5daef\\data\\Ghamadi\\037135.mp3', 'array': array([ 0.00000000e+00, -2.86519935e-22, -2.56504911e-21, ...,
       -1.94239747e-02, -2.42924765e-02, -2.99104657e-02]), 'sampling_rate': 22050}, 'reciter': 'Ghamadi', 'transcription': 'الا عجوز ا في الغبرين', 'line': 3923, 'chapter': 37, 'verse': 135, 'text': 'إِلَّا عَجُوزࣰ ا فِي ٱلۡغَٰبِرِينَ'}
Traceback (most recent call last):
  File "C:\Users\Nawaz-Server\Documents\ml\myest.py", line 5, in <module>
    for data in dataset["train"]:
                ~~~~~~~^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\datasets\arrow_dataset.py", line 2372, in __iter__
    formatted_output = format_table(
                       ^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\datasets\formatting\formatting.py", line 639, in format_table
    return formatter(pa_table, query_type=query_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\datasets\formatting\formatting.py", line 403, in __call__
    return self.format_row(pa_table)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\datasets\formatting\formatting.py", line 444, in format_row
    row = self.python_features_decoder.decode_row(row)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\datasets\formatting\formatting.py", line 222, in decode_row
    return self.features.decode_example(row) if self.features else row
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\datasets\features\features.py", line 2042, in decode_example
    column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\datasets\features\features.py", line 1403, in decode_nested_example
    return schema.decode_example(obj, token_per_repo_id=token_per_repo_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\datasets\features\audio.py", line 184, in decode_example
    array, sampling_rate = sf.read(f)
                           ^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\soundfile.py", line 285, in read
    with SoundFile(file, 'r', samplerate, channels,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\soundfile.py", line 658, in __init__
    self._file = self._open(file, mode_int, closefd)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nawaz-Server\.conda\envs\mypy\Lib\site-packages\soundfile.py", line 1216, in _open
    raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BufferedReader name='C:\\Users\\Nawaz-Server\\.cache\\huggingface\\hub\\datasets--fawazahmed0--bug-audio\\snapshots\\fab1398431fed1c0a2a7bff0945465bab8b5daef\\data\\Ghamadi\\037136.mp3'>: Format not recognised.
```

### Expected behavior

Everything should work fine, as loading the problematic audio file directly with soundfile package works fine

**code:**
```
import soundfile as sf

print(sf.read('C:\\Users\\Nawaz-Server\\.cache\\huggingface\\hub\\datasets--fawazahmed0--bug-audio\\snapshots\\fab1398431fed1c0a2a7bff0945465bab8b5daef\\data\\Ghamadi\\037136.mp3'))
```

**output:**
```bash
(mypy) C:\Users\Nawaz-Server\Documents\ml>python myest.py
[C:\vcpkg\buildtrees\mpg123\src\0d8db63f9b-3db975bc05.clean\src\libmpg123\layer3.c:INT123_do_layer3():1801] error: dequantization failed!
(array([ 0.00000000e+00, -8.43723821e-22, -2.45370628e-22, ...,
       -7.71464454e-03, -6.90496899e-03, -8.63333419e-03]), 22050)
```

### Environment info

- `datasets` version: 3.0.2
- Platform: Windows-11-10.0.22621-SP0
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.2
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.10.0
- soundfile: 0.12.1


load_dataset

### Describe the bug

I am performing two operations I see on a hugging face tutorial (Fine-tune a language model), and I am defining every aspect inside the mapped functions, also some imports of the library because it doesnt identify anything not defined outside that function where the dataset elements are being mapped:

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

`- lm_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    batch_size=batch_size,
    num_proc=4,
)

- tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])
def tokenize_function(examples):
    model_checkpoint = 'gpt2'
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    return tokenizer(examples["text"])`

### Steps to reproduce the bug

Currently handle all the imports inside the function

### Expected behavior

The code must work es expected in the notebook, but currently this is not happening.

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

### Environment info

print(transformers.__version__)

4.46.1


[MINOR:TYPO] Fix typo in exception text


Raise error for incorrect JSON serialization

Raise error when `lines = False` and `batch_size < Dataset.num_rows` in `Dataset.to_json()`.

Issue: #7037

Related PRs:
#7039 #7181


fix conda release worlflow


Set dev version


Release: 3.1.0


Memory leak when streaming

### Describe the bug

I try to use a dataset with streaming=True, the issue I have is that the RAM usage becomes higher and higher until it is no longer sustainable.

I understand that huggingface store data in ram during the streaming, and more worker in dataloader there are, more a lot of shard will be stored in ram, but the issue I have is that the ram usage is not constant. So after each new shard loaded, the ram usage will be higher and higher.

### Steps to reproduce the bug

You can run this code and see you ram usage, after each shard of 255 examples, your ram usage will be extended.
```py
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("WaveGenAI/dataset", streaming=True)

dataloader = DataLoader(dataset["train"], num_workers=3)

for i, data in enumerate(dataloader):
    print(i, end="\r")
```

### Expected behavior

The Ram usage should be always the same (just 3 shards loaded in the ram).

### Environment info

- `datasets` version: 3.0.1
- Platform: Linux-6.10.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1


load_from_disk

### Describe the bug

I have data saved with save_to_disk. The data is big (700Gb). When I try loading it, the only option is load_from_disk, and this function copies the data to a tmp directory, causing me to run out of disk space. Is there an alternative solution to that?

### Steps to reproduce the bug

when trying to load data using load_From_disk after being saved using save_to_disk

### Expected behavior

run out of disk space

### Environment info

lateest version


Source installation fails on Macintosh with python 3.10

### Describe the bug

Hi,

Decord is a dev dependency not maintained since couple years.
It does not have an ARM package available rendering it uninstallable on non-intel based macs

Suggestion is to move to eva-decord (https://github.com/georgia-tech-db/eva-decord)  which doesnt have this problem.

Happy to raise a PR

### Steps to reproduce the bug

Source installation as mentioned in contributinog.md

### Expected behavior

Installation without decord failing to be installed.

### Environment info

python=3.10, M3 Mac


### Describe the bug

After waiting for 2 hours, it still presents ``The dataset viewer should be available soon. Please retry later.''

### Steps to reproduce the bug

dataset link: https://huggingface.co/datasets/BryanW/HI_EDIT

### Expected behavior

Present the dataset viewer.

### Environment info

NA


Disallow video push_to_hub


fix docs relative links


Small addition to video docs


Allow video with disabeld decoding without decord

for the viewer, this way it can use Video(decode=False) and doesn't need decord (which causes segfaults)


Cannot load the cache when mapping the dataset

### Describe the bug

I'm training the flux controlnet. The train_dataset.map() takes long time to finish. However, when I killed one training process and want to restart a new training with the same dataset. I can't reuse the mapped result even I defined the cache dir for the dataset.

with accelerator.main_process_first():
        from datasets.fingerprint import Hasher

        # fingerprint used by the cache for the other processes to load the result
        # details: https://github.com/huggingface/diffusers/pull/4038#discussion_r1266078401
        new_fingerprint = Hasher.hash(args)
        train_dataset = train_dataset.map(
            compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint, batch_size=10,
        )

### Steps to reproduce the bug

train flux controlnet and start again

### Expected behavior

will not map again

### Environment info

latest diffusers


cache can't cleaned or disabled

### Describe the bug

I tried following ways, the cache can't be disabled.

I got 2T data, but I also got more than 2T cache file. I got pressure on storage. I need to diable cache or cleaned immediately after processed. Following ways are all not working, please give some help!

```python
from datasets import disable_caching
from transformers import AutoTokenizer
disable_caching()

tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path)
def tokenization_fn(examples):
    column_name = 'text' if 'text' in examples else 'data'
    tokenized_inputs = tokenizer(
        examples[column_name], return_special_tokens_mask=True, truncation=False,
        max_length=tokenizer.model_max_length
    )
    return tokenized_inputs

data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=None)
data.cleanup_cache_files()
updated_dataset = data.map(tokenization_fn, load_from_cache_file=False)
updated_dataset .cleanup_cache_files()
```

### Expected behavior

no cache file generated

### Environment info

Ubuntu 20.04.6 LTS
datasets  3.0.2


Don't embed videos

don't include video bytes when running download_and_prepare(format="parquet")

this also affects push_to_hub which will just upload the local paths of the videos though


Always set non-null writer batch size

bug introduced in #7230, it was preventing the Viewer limit writes to work


fix ci for pyarrow 18


Retry all requests timeouts

as reported in https://github.com/huggingface/datasets/issues/6843


fix decord import

delay the import until Video() is instantiated + also import duckdb first (otherwise importing duckdb later causes a segfault)


mismatch for datatypes when providing `Features` with `Array2D` and user specified `dtype` and using with_format("numpy")

### Describe the bug

If the user provides a `Features` type value to `datasets.Dataset` with members having `Array2D` with a value for `dtype`, it is not respected during `with_format("numpy")` which should return a `np.array` with `dtype` that the user provided for `Array2D`. It seems for floats, it will be set to `float32` and for ints it will be set to `int64`

### Steps to reproduce the bug

```python
import numpy as np
import datasets
from datasets import Dataset, Features, Array2D

print(f"datasets version: {datasets.__version__}")

data_info = {
    "arr_float" : "float64",
    "arr_int" : "int32"
}

sample = {key : [np.zeros([4, 5], dtype=dtype)] for key, dtype in data_info.items()}

features = {key : Array2D(shape=(None, 5), dtype=dtype) for key, dtype in data_info.items()}
features = Features(features)

dataset = Dataset.from_dict(sample, features=features)

ds = dataset.with_format("numpy")
for key in features:
    print(f"{key} feature dtype: ", ds.features[key].dtype)
    print(f"{key} dtype:", ds[key].dtype)
```

Output:
```bash
datasets version: 3.0.2
arr_float feature dtype:  float64
arr_float dtype: float32
arr_int feature dtype:  int32
arr_int dtype: int64
```

### Expected behavior

It should return a `np.array` with `dtype` that the user provided for the corresponding member in the `Features` type value

### Environment info

- `datasets` version: 3.0.2
- Platform: Linux-6.11.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


Unable to upload a large dataset zip either from command line or UI

### Describe the bug

Unable to upload a large dataset zip from command line or UI. UI simply says error. I am trying to a upload a tar.gz file of 17GB.

### Steps to reproduce the bug

Upload a large file

### Expected behavior

The file should upload without any issue.

### Environment info

None


Add IterableDataset.shard()

Will be useful to distribute a dataset across workers (other than pytorch) like spark

I also renamed `.n_shards` -> `.num_shards` for consistency and kept the old name for backward compatibility. And a few changes in internal functions for consistency as well (rank, world_size -> num_shards, index)

Breaking change: the new default for `contiguous` in `Dataset.shard()` is `True`, but imo not a big deal since I couldn't find any usage of `contiguous=False` internally (we always do contiguous=True for map-style datasets since its more optimized) or in the wild


Missing video docs


Basic XML support (mostly copy pasted from text)

enable the viewer for datasets like https://huggingface.co/datasets/FrancophonIA/e-calm (there will be more and more apparently)


How to debugging

### Describe the bug

I wanted to use my own script to handle the processing, and followed the tutorial documentation by rewriting the MyDatasetConfig and MyDatasetBuilder (which contains the _info,_split_generators and _generate_examples methods) classes. Testing with simple data was able to output the results of the processing, but when I wished to do more complex processing, I found that I was unable to debug (even the simple samples were inaccessible). There are no errors reported, and I am able to print the _info,_split_generators and _generate_examples messages, but I am unable to access the breakpoints.

### Steps to reproduce the bug

# my_dataset.py
import json
import datasets

class MyDatasetConfig(datasets.BuilderConfig):
    def __init__(self, **kwargs):
        super(MyDatasetConfig, self).__init__(**kwargs)

class MyDataset(datasets.GeneratorBasedBuilder):
    VERSION = datasets.Version("1.0.0")

    BUILDER_CONFIGS = [
        MyDatasetConfig(
            name="default",
            version=VERSION,
            description="myDATASET"
        ),
    ]

    def _info(self):
        print("info")  # breakpoints
        return datasets.DatasetInfo(
            description="myDATASET",
            features=datasets.Features(
                {
                    "id": datasets.Value("int32"),
                    "text": datasets.Value("string"),
                    "label": datasets.ClassLabel(names=["negative", "positive"]),
                }
            ),
            supervised_keys=("text", "label"),
        )

    def _split_generators(self, dl_manager):

        print("generate")  # breakpoints
        data_file = "data.json"

        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"filepath": data_file}
            ),
        ]

    def _generate_examples(self, filepath):
        print("example")  # breakpoints
        with open(filepath, encoding="utf-8") as f:
            data = json.load(f)
            for idx, sample in enumerate(data):
                yield idx, {
                    "id": sample["id"],
                    "text": sample["text"],
                    "label": sample["label"],
                }

#main.py
import os
os.environ["TRANSFORMERS_NO_MULTIPROCESSING"] = "1"

from datasets import load_dataset

dataset = load_dataset("my_dataset.py", split="train", cache_dir=None)

print(dataset[:5])

### Expected behavior

Pause at breakpoints while running debugging

### Environment info

pycharm


### Describe the bug

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-9-13b5f31bd391>](https://bcb6shpazyu-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241022-060119_RC00_688494744#) in <cell line: 1>()
----> 1 dataset = load_dataset('knowledgator/events_classification_biotech')

11 frames
[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://bcb6shpazyu-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241022-060119_RC00_688494744#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, keep_in_memory, save_infos, revision, token, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   2130
   2131     # Create a dataset builder
-> 2132     builder_instance = load_dataset_builder(
   2133         path=path,
   2134         name=name,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://bcb6shpazyu-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241022-060119_RC00_688494744#) in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, storage_options, trust_remote_code, _require_default_config_name, **config_kwargs)
   1886         raise ValueError(error_msg)
   1887
-> 1888     builder_cls = get_dataset_builder_class(dataset_module, dataset_name=dataset_name)
   1889     # Instantiate the dataset builder
   1890     builder_instance: DatasetBuilder = builder_cls(

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://bcb6shpazyu-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241022-060119_RC00_688494744#) in get_dataset_builder_class(dataset_module, dataset_name)
    246         dataset_module.importable_file_path
    247     ) if dataset_module.importable_file_path else nullcontext():
--> 248         builder_cls = import_main_class(dataset_module.module_path)
    249     if dataset_module.builder_configs_parameters.builder_configs:
    250         dataset_name = dataset_name or dataset_module.builder_kwargs.get("dataset_name")

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://bcb6shpazyu-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241022-060119_RC00_688494744#) in import_main_class(module_path)
    167 def import_main_class(module_path) -> Optional[Type[DatasetBuilder]]:
    168     """Import a module at module_path and return its main class: a DatasetBuilder"""
--> 169     module = importlib.import_module(module_path)
    170     # Find the main class in our imported module
    171     module_main_cls = None

[/usr/lib/python3.10/importlib/__init__.py](https://bcb6shpazyu-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241022-060119_RC00_688494744#) in import_module(name, package)
    124                 break
    125             level += 1
--> 126     return _bootstrap._gcd_import(name[level:], package, level)
    127
    128

/usr/lib/python3.10/importlib/_bootstrap.py in _gcd_import(name, package, level)

/usr/lib/python3.10/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib/python3.10/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib/python3.10/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib/python3.10/importlib/_bootstrap_external.py in exec_module(self, module)

/usr/lib/python3.10/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

[~/.cache/huggingface/modules/datasets_modules/datasets/knowledgator--events_classification_biotech/9c8086d498c3104de3a3c5b6640837e18ccd829dcaca49f1cdffe3eb5c4a6361/events_classification_biotech.py](https://bcb6shpazyu-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20241022-060119_RC00_688494744#) in <module>
      1 import datasets
      2 from datasets import load_dataset
----> 3 from datasets.tasks import TextClassification
      4
      5 DESCRIPTION = """

ModuleNotFoundError: No module named 'datasets.tasks'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

### Steps to reproduce the bug

!pip install datasets

from datasets import load_dataset

dataset = load_dataset('knowledgator/events_classification_biotech')

### Expected behavior

no ModuleNotFoundError

### Environment info

google colab


Adding column with dict struction when mapping lead to wrong order

### Describe the bug

in `map()` function, I want to add a new column with a dict structure.
```
def map_fn(example):
  example['text'] = {'user': ..., 'assistant': ...}
  return example
```
However this leads to a wrong order `{'assistant':..., 'user':...}` in the dataset.
Thus I can't concatenate two datasets due to the different feature structures.
[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example
This seems an issue in low level pyarrow library instead of datasets, however, I think datasets should allow concatenate two datasets actually in the same structure.

### Steps to reproduce the bug

[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example

### Expected behavior

two datasets could be concatenated.

### Environment info

N/A


Set dev version


Release: 3.0.2


use huggingface_hub offline mode

and better handling of LocalEntryNotfoundError cc @Wauplin

follow up to #7234


ArrayXD with None as leading dim incompatible with DatasetCardData

### Describe the bug

Creating a dataset with ArrayXD features leads to errors when downloading from hub due to DatasetCardData removing the Nones

@lhoestq

### Steps to reproduce the bug

```python
import numpy as np
from datasets import Array2D, Dataset, Features, load_dataset

def examples_generator():
    for i in range(4):
        yield {
            "array_1d": np.zeros((10,1), dtype="uint16"),
            "array_2d": np.zeros((10, 1), dtype="uint16"),
        }

features = Features(array_1d=Array2D((None,1), "uint16"), array_2d=Array2D((None, 1), "uint16"))
dataset = Dataset.from_generator(examples_generator, features=features)
dataset.push_to_hub("alex-hh/test_array_1d2d")
ds = load_dataset("alex-hh/test_array_1d2d")
```

Source of error appears to be DatasetCardData.to_dict invoking DatasetCardData._remove_none
```python
from huggingface_hub import DatasetCardData
from datasets.info import DatasetInfosDict

dataset_card_data = DatasetCardData()
DatasetInfosDict({"default": dataset.info.copy()}).to_dataset_card_data(dataset_card_data)
print(dataset_card_data.to_dict())  # removes Nones in shape
```

### Expected behavior

Should be possible to load datasets saved with shape None in leading dimension

### Environment info

3.0.2 and latest huggingface_hub


`push_to_hub` overwrite argument

enhancement

New feature or request

### Feature request

Add an `overwrite` argument to the `push_to_hub` method.

### Motivation

I want to overwrite a repo without deleting it on Hugging Face. Is this possible? I couldn't find anything in the documentation or tutorials.

### Your contribution

I can create a PR.


Feature Request: Add functionality to pass split types like train, test in DatasetDict.map

Hello datasets!

We often encounter situations where we need to preprocess data differently depending on split types such as train, valid, and test.

However, while DatasetDict.map has features to pass rank or index, there's no functionality to pass split types.
Therefore, I propose adding a 'with_splits' parameter to DatasetDict, which would allow passing the split type through fn_kwargs.


incompatibily issue when using load_dataset with datasets==3.0.1

### Describe the bug

There is a bug when using load_dataset with dataset version at 3.0.1 .
Please see below in the "steps to reproduce the bug".
To resolve the bug, I had to downgrade to version 2.21.0
OS: Ubuntu 24 (AWS instance)
Python: same bug under 3.12 and 3.10

The error I had was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/load.py", line 2096, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 1647, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 977, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_0/cb17afd34f5799f97e8f48398748f83006335b702bd785f9880797838d541b81/common_voice_6_0.py", line 159, in _split_generators
    archive_path = dl_manager.download(self._get_bundle_url(self.config.name, bundle_url_template))
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_manager.py", line 150, in download
    download_config = self.download_config.copy()
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_config.py", line 73, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
TypeError: DownloadConfig.__init__() got an unexpected keyword argument 'ignore_url_params'

### Steps to reproduce the bug

1. install dataset with ```pip install datasets --upgrade```
2. launch python; from datasets import loaad_dataset
3. run load_dataset("mozilla-foundation/common_voice_6_0")
4. exit python
5. uninstall datasets; then ```pip install datasets==2.21.0```
6. launch python; from datasets import loaad_dataset
7. run load_dataset("mozilla-foundation/common_voice_6_0")
8. Everything runs great now

### Expected behavior

Be able to download a dataset without error

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 3.0.1
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1


[MINOR:TYPO] Update arrow_dataset.py

Fix wrong link.

csv kwargs docstring link was pointing to pandas json docs.


No need for dataset_info

save a useless call to /api/datasets/repo_id


数据集数量问题

### Describe the bug

这里我进行大模型微调，当数据集数量为718时，模型可以正常微调，但是当我添加一个在前718个数据集中的数据或者新增一个数据就会报错

### Steps to reproduce the bug

1.
这里我的数据集可以微调的最后两个数据集是：
{
  "messages": [
    {
      "role": "user",
      "content": "完成校正装置设计后需要进行哪些工作？"
    },
    {
      "role": "assistant",
      "content": "一旦完成校正装置设计后，需要进行系统实际调校工作，或者进行MATLAB仿真以检查系统的时间响应特性。这时，需将系统建模时省略的部分尽可能加入系统，以保证仿真结果的逼真度。如果由于系统各种固有非线性因素影响，或者由于系统噪声和负载效应等因素的影响，使已校正系统不能满足全部性能指标要求，则需要适当调整校正装置的形式或参数，直到已校正系统满足全部性能指标为止。"
    }
  ]
}

{
  "messages": [
    {
      "role": "user",
      "content": "为什么闭环带宽要求会影响串联超前校正的效果？"
    },
    {
      "role": "assistant",
      "content": "如果待校正系统不稳定，为了满足相角裕度的要求，超前网络需要提供很大的相角超前量。这会导致超前网络的a值必须选得很大，从而使已校正系统的带宽过大。带宽过大会使通过系统的高频噪声电平变高，可能导致系统失控。因此，闭环带宽的要求会影响串联超前校正的效果。"
    }
  ]
}
2.
当我把倒数第二个数据集再次添加到数据集中时，此时我的最后三个数据集是
{
  "messages": [
    {
      "role": "user",
      "content": "完成校正装置设计后需要进行哪些工作？"
    },
    {
      "role": "assistant",
      "content": "一旦完成校正装置设计后，需要进行系统实际调校工作，或者进行MATLAB仿真以检查系统的时间响应特性。这时，需将系统建模时省略的部分尽可能加入系统，以保证仿真结果的逼真度。如果由于系统各种固有非线性因素影响，或者由于系统噪声和负载效应等因素的影响，使已校正系统不能满足全部性能指标要求，则需要适当调整校正装置的形式或参数，直到已校正系统满足全部性能指标为止。"
    }
  ]
}

{
  "messages": [
    {
      "role": "user",
      "content": "为什么闭环带宽要求会影响串联超前校正的效果？"
    },
    {
      "role": "assistant",
      "content": "如果待校正系统不稳定，为了满足相角裕度的要求，超前网络需要提供很大的相角超前量。这会导致超前网络的a值必须选得很大，从而使已校正系统的带宽过大。带宽过大会使通过系统的高频噪声电平变高，可能导致系统失控。因此，闭环带宽的要求会影响串联超前校正的效果。"
    }
  ]
}

{
  "messages": [
    {
      "role": "user",
      "content": "完成校正装置设计后需要进行哪些工作？"
    },
    {
      "role": "assistant",
      "content": "一旦完成校正装置设计后，需要进行系统实际调校工作，或者进行MATLAB仿真以检查系统的时间响应特性。这时，需将系统建模时省略的部分尽可能加入系统，以保证仿真结果的逼真度。如果由于系统各种固有非线性因素影响，或者由于系统噪声和负载效应等因素的影响，使已校正系统不能满足全部性能指标要求，则需要适当调整校正装置的形式或参数，直到已校正系统满足全部性能指标为止。"
    }
  ]
}
这时系统会显示bug：
root@autodl-container-027f4cad3d-6baf4e64:~/autodl-tmp# python GLM-4/finetune_demo/finetune.py  datasets/  ZhipuAI/glm-4-9b-chat  GLM-4/finetune_demo/configs/lora.yaml
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  4.04it/s]
The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
trainable params: 2,785,280 || all params: 9,402,736,640 || trainable%: 0.0296
Generating train split: 0 examples [00:00, ? examples/s]Failed to load JSON from file '/root/autodl-tmp/datasets/train.jsonl' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Missing a name for object member. in row 718
Generating train split: 0 examples [00:00, ? examples/s]
╭──────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /root/miniconda3/lib/python3.10/site-packages/datasets/packaged_modules/json/json.py:153 in _generate_tables                                                                                │
│                                                                                │
│   150 │   │   │   │   │   │   │   │   with open(                                                                                │
│   151 │   │   │   │   │   │   │   │   │   file, encoding=self.config.encoding, errors=self.con                                                                                │
│   152 │   │   │   │   │   │   │   │   ) as f:                                                                                │
│ ❱ 153 │   │   │   │   │   │   │   │   │   df = pd.read_json(f, dtype_backend="pyarrow")                                                                                │
│   154 │   │   │   │   │   │   │   except ValueError:                                                                                │
│   155 │   │   │   │   │   │   │   │   logger.error(f"Failed to load JSON from file '{file}' wi                                                                                │
│   156 │   │   │   │   │   │   │   │   raise e                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/pandas/io/json/_json.py:815 in read_json                                                                                │
│                                                                                │
│    812 │   if chunksize:                                                                                │
│    813 │   │   return json_reader                                                                                │
│    814 │   else:                                                                                │
│ ❱  815 │   │   return json_reader.read()                                                                                │
│    816                                                                                │
│    817                                                                                │
│    818 class JsonReader(abc.Iterator, Generic[FrameSeriesStrT]):                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/pandas/io/json/_json.py:1025 in read                                                                                │
│                                                                                │
│   1022 │   │   │   │   │   │   data_lines = data.split("\n")                                                                                │
│   1023 │   │   │   │   │   │   obj = self._get_object_parser(self._combine_lines(data_lines))                                                                                │
│   1024 │   │   │   │   else:                                                                                │
│ ❱ 1025 │   │   │   │   │   obj = self._get_object_parser(self.data)                                                                                │
│   1026 │   │   │   │   if self.dtype_backend is not lib.no_default:                                                                                │
│   1027 │   │   │   │   │   return obj.convert_dtypes(                                                                                │
│   1028 │   │   │   │   │   │   infer_objects=False, dtype_backend=self.dtype_backend                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/pandas/io/json/_json.py:1051 in _get_object_parser                                                                                │
│                                                                                │
│   1048 │   │   }                                                                                │
│   1049 │   │   obj = None                                                                                │
│   1050 │   │   if typ == "frame":                                                                                │
│ ❱ 1051 │   │   │   obj = FrameParser(json, **kwargs).parse()                                                                                │
│   1052 │   │                                                                                │
│   1053 │   │   if typ == "series" or obj is None:                                                                                │
│   1054 │   │   │   if not isinstance(dtype, bool):                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/pandas/io/json/_json.py:1187 in parse                                                                                │
│                                                                                │
│   1184 │                                                                                │
│   1185 │   @final                                                                                │
│   1186 │   def parse(self):                                                                                │
│ ❱ 1187 │   │   self._parse()                                                                                │
│   1188 │   │                                                                                │
│   1189 │   │   if self.obj is None:                                                                                │
│   1190 │   │   │   return None                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/pandas/io/json/_json.py:1403 in _parse                                                                                │
│                                                                                │
│   1400 │   │                                                                                │
│   1401 │   │   if orient == "columns":                                                                                │
│   1402 │   │   │   self.obj = DataFrame(                                                                                │
│ ❱ 1403 │   │   │   │   ujson_loads(json, precise_float=self.precise_float), dtype=None                                                                                │
│   1404 │   │   │   )                                                                                │
│   1405 │   │   elif orient == "split":                                                                                │
│   1406 │   │   │   decoded = {                                                                                │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Trailing data

During handling of the above exception, another exception occurred:

╭──────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /root/miniconda3/lib/python3.10/site-packages/datasets/builder.py:1997 in _prepare_split_single                                                                                │
│                                                                                │
│   1994 │   │   │   )                                                                                │
│   1995 │   │   │   try:                                                                                │
│   1996 │   │   │   │   _time = time.time()                                                                                │
│ ❱ 1997 │   │   │   │   for _, table in generator:                                                                                │
│   1998 │   │   │   │   │   if max_shard_size is not None and writer._num_bytes > max_shard_size                                                                                │
│   1999 │   │   │   │   │   │   num_examples, num_bytes = writer.finalize()                                                                                │
│   2000 │   │   │   │   │   │   writer.close()                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/datasets/packaged_modules/json/json.py:156 in _generate_tables                                                                                │
│                                                                                │
│   153 │   │   │   │   │   │   │   │   │   df = pd.read_json(f, dtype_backend="pyarrow")                                                                                │
│   154 │   │   │   │   │   │   │   except ValueError:                                                                                │
│   155 │   │   │   │   │   │   │   │   logger.error(f"Failed to load JSON from file '{file}' wi                                                                                │
│ ❱ 156 │   │   │   │   │   │   │   │   raise e                                                                                │
│   157 │   │   │   │   │   │   │   if df.columns.tolist() == [0]:                                                                                │
│   158 │   │   │   │   │   │   │   │   df.columns = list(self.config.features) if self.config.f                                                                                │
│   159 │   │   │   │   │   │   │   try:                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/datasets/packaged_modules/json/json.py:130 in _generate_tables                                                                                │
│                                                                                │
│   127 │   │   │   │   │   │   try:                                                                                │
│   128 │   │   │   │   │   │   │   while True:                                                                                │
│   129 │   │   │   │   │   │   │   │   try:                                                                                │
│ ❱ 130 │   │   │   │   │   │   │   │   │   pa_table = paj.read_json(                                                                                │
│   131 │   │   │   │   │   │   │   │   │   │   io.BytesIO(batch), read_options=paj.ReadOptions(                                                                                │
│   132 │   │   │   │   │   │   │   │   │   )                                                                                │
│   133 │   │   │   │   │   │   │   │   │   break                                                                                │
│                                                                                │
│ in pyarrow._json.read_json:308                                                                                │
│                                                                                │
│ in pyarrow.lib.pyarrow_internal_check_status:154                                                                                │
│                                                                                │
│ in pyarrow.lib.check_status:91                                                                                │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ArrowInvalid: JSON parse error: Missing a name for object member. in row 718

The above exception was the direct cause of the following exception:

╭──────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /root/autodl-tmp/GLM-4/finetune_demo/finetune.py:406 in main                                                                                │
│                                                                                │
│   403 ):                                                                                │
│   404 │   ft_config = FinetuningConfig.from_file(config_file)                                                                                │
│   405 │   tokenizer, model = load_tokenizer_and_model(model_dir, peft_config=ft_config.peft_co                                                                                │
│ ❱ 406 │   data_manager = DataManager(data_dir, ft_config.data_config)                                                                                │
│   407 │                                                                                │
│   408 │   train_dataset = data_manager.get_dataset(                                                                                │
│   409 │   │   Split.TRAIN,                                                                                │
│                                                                                │
│ /root/autodl-tmp/GLM-4/finetune_demo/finetune.py:204 in __init__                                                                                │
│                                                                                │
│   201 │   def __init__(self, data_dir: str, data_config: DataConfig):                                                                                │
│   202 │   │   self._num_proc = data_config.num_proc                                                                                │
│   203 │   │                                                                                │
│ ❱ 204 │   │   self._dataset_dct = _load_datasets(                                                                                │
│   205 │   │   │   data_dir,                                                                                │
│   206 │   │   │   data_config.data_format,                                                                                │
│   207 │   │   │   data_config.data_files,                                                                                │
│                                                                                │
│ /root/autodl-tmp/GLM-4/finetune_demo/finetune.py:189 in _load_datasets                                                                                │
│                                                                                │
│   186 │   │   num_proc: Optional[int],                                                                                │
│   187 ) -> DatasetDict:                                                                                │
│   188 │   if data_format == '.jsonl':                                                                                │
│ ❱ 189 │   │   dataset_dct = load_dataset(                                                                                │
│   190 │   │   │   data_dir,                                                                                │
│   191 │   │   │   data_files=data_files,                                                                                │
│   192 │   │   │   split=None,                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/datasets/load.py:2616 in load_dataset                                                                                │
│                                                                                │
│   2613 │   │   return builder_instance.as_streaming_dataset(split=split)                                                                                │
│   2614 │                                                                                │
│   2615 │   # Download and prepare data                                                                                │
│ ❱ 2616 │   builder_instance.download_and_prepare(                                                                                │
│   2617 │   │   download_config=download_config,                                                                                │
│   2618 │   │   download_mode=download_mode,                                                                                │
│   2619 │   │   verification_mode=verification_mode,                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/datasets/builder.py:1029 in download_and_prepare                                                                                │
│                                                                                │
│   1026 │   │   │   │   │   │   │   prepare_split_kwargs["max_shard_size"] = max_shard_size                                                                                │
│   1027 │   │   │   │   │   │   if num_proc is not None:                                                                                │
│   1028 │   │   │   │   │   │   │   prepare_split_kwargs["num_proc"] = num_proc                                                                                │
│ ❱ 1029 │   │   │   │   │   │   self._download_and_prepare(                                                                                │
│   1030 │   │   │   │   │   │   │   dl_manager=dl_manager,                                                                                │
│   1031 │   │   │   │   │   │   │   verification_mode=verification_mode,                                                                                │
│   1032 │   │   │   │   │   │   │   **prepare_split_kwargs,                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/datasets/builder.py:1124 in _download_and_prepare                                                                                │
│                                                                                │
│   1121 │   │   │                                                                                │
│   1122 │   │   │   try:                                                                                │
│   1123 │   │   │   │   # Prepare split will record examples associated to the split                                                                                │
│ ❱ 1124 │   │   │   │   self._prepare_split(split_generator, **prepare_split_kwargs)                                                                                │
│   1125 │   │   │   except OSError as e:                                                                                │
│   1126 │   │   │   │   raise OSError(                                                                                │
│   1127 │   │   │   │   │   "Cannot find data file. "                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/datasets/builder.py:1884 in _prepare_split                                                                                │
│                                                                                │
│   1881 │   │   │   gen_kwargs = split_generator.gen_kwargs                                                                                │
│   1882 │   │   │   job_id = 0                                                                                │
│   1883 │   │   │   with pbar:                                                                                │
│ ❱ 1884 │   │   │   │   for job_id, done, content in self._prepare_split_single(                                                                                │
│   1885 │   │   │   │   │   gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args                                                                                │
│   1886 │   │   │   │   ):                                                                                │
│   1887 │   │   │   │   │   if done:                                                                                │
│                                                                                │
│ /root/miniconda3/lib/python3.10/site-packages/datasets/builder.py:2040 in _prepare_split_single                                                                                │
│                                                                                │
│   2037 │   │   │   │   e = e.__context__                                                                                │
│   2038 │   │   │   if isinstance(e, DatasetGenerationError):                                                                                │
│   2039 │   │   │   │   raise                                                                                │
│ ❱ 2040 │   │   │   raise DatasetGenerationError("An error occurred while generating the dataset                                                                                │
│   2041 │   │                                                                                │
│   2042 │   │   yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_                                                                                │
│   2043                                                                                │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
DatasetGenerationError: An error occurred while generating the dataset
3.请问是否可以帮我解决

### Expected behavior

希望问题可以得到解决

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.20.0
- Platform: Linux-4.19.90-2107.6.0.0192.8.oe1.bclinux.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.8
- `huggingface_hub` version: 0.24.6
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2023.12.2


(Super tiny doc update) Mention to_polars

polars is also quite popular now, thus this tiny update can tell users polars is supported


Fix typo in image dataset docs

Fix typo in image dataset docs.

Typo reported by @datavistics.


Video support

(wip and experimental)

adding the `Video` type based on `VideoReader` from `decord`

```python
>>>from datasets import load_dataset
>>> ds = load_dataset("path/to/videos", split="train").with_format("torch")
>>> print(ds[0]["video"])
<decord.video_reader.VideoReader object at 0x337a47910>
>>> print(ds[0]["video"][0])
tensor([[[73, 73, 73],
         [73, 73, 73],
         [73, 73, 73],
         ...,
         [23, 23, 23],
         [23, 23, 23],
         [23, 23, 23]]], dtype=torch.uint8)
```

the storage is the same as for audio and images: `{"path": pa.string(), "bytes": pa.binary()}` and I did a small to keep the hf:// URL in the "path" field if possible, this way the viewer can link to fiels on the hub if possible


handle config_name=None in push_to_hub

This caught me out - thought it might be better to explicitly handle None?


Composite (multi-column) features

New feature or request

### Feature request

Structured data types (graphs etc.) might often be most efficiently stored as multiple columns, which then need to be combined during feature decoding

Although it is currently possible to nest features as structs, my impression is that in particular when dealing with e.g. a feature composed of multiple numpy array / ArrayXD's, it would be more efficient to store each ArrayXD as a separate column (though I'm not sure by how much)

Perhaps specification / implementation could be supported by something like:

```
features=Features(**{("feature0", "feature1")=Features(feature0=Array2D((None,10), dtype="float32"), feature1=Array2D((None,10), dtype="float32"))
```

### Motivation

Defining efficient composite feature types based on numpy arrays for representing data such as graphs with multiple node and edge attributes is currently challenging.

### Your contribution

Possibly able to contribute


fast array extraction

Implements #7210 using method suggested in https://github.com/huggingface/datasets/pull/7207#issuecomment-2411789307

```python
import numpy as np
from datasets import Dataset, Features, Array3D
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```
~0.02 s vs 0.9s on main

```python
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
< 0.01 s vs 1.3 s on main

@lhoestq I can see this breaks a bunch of array-related tests but can update the test cases if you would support making this change?

I also added an Array1D feature which will always be decoded into a numpy array and likewise improves extraction performance:

```python
from datasets import Dataset, Features, Array1D, Sequence, Value
array_features=Features(**{"array0": Array1D((None,), dtype="float32"), "array1": Array1D((None,), dtype="float32")})
sequence_features=Features(**{"array0": Sequence(feature=Value("float32"), length=-1), "array1": Sequence(feature=Value("float32"), length=-1)})
array_dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [20000,10000]*25] for i in range(2)}, features=array_features)
sequence_dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [20000,10000]*25] for i in range(2)}, features=sequence_features)

```python
t0 = time.time()
for ex in array_dataset.to_iterable_dataset():
    pass
t1 = time.time()
```
< 0.01 s

```python
t0 = time.time()
for ex in sequence_dataset.to_iterable_dataset():
    pass
t1 = time.time()
```
~1.1s

And also added support for extracting structs of arrays as dicts of numpy arrays:

```python
import numpy as np
from datasets import Dataset, Features, Array3D, Sequence
features=Features(struct={"array0": Array3D((None,10,10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")}, _list=Sequence(feature=Array3D((None,10,10), dtype="float32")))
dataset = Dataset.from_dict({"struct": [{f"array{i}": np.zeros((x,10,10), dtype=np.float32) for i in range(2)} for x in [2000,1000]*25], "_list": [[np.zeros((x,10,10), dtype=np.float32) for i in range(2)] for x in [2000,1000]*25]}, features=features)
```

```python
t0 = time.time()
for ex in dataset.to_iterable_dataset():
    pass
t1 = time.time()
assert isinstance(ex["struct"]["array0"], np.ndarray) and ex["struct"]["array0"].ndim == 3
```
~0.02 s and no exception vs ~7s with an exception on main


Add R as a How to use from the Polars (R) Library as an option

New feature or request

### Feature request

The boiler plate code to access a dataset via the hugging face file system is very useful. Please addd

## Add Polars (R) option
The equivailent code works, because the [Polars-R](https://github.com/pola-rs/r-polars) wrapper has hugging faces funcitonaliy as well.

```r
library(polars)

df <- pl$read_parquet("hf://datasets/SALURBAL/core__admin_cube_public/core__admin_cube_public.parquet")
```

## Polars (python) option
![image](https://github.com/user-attachments/assets/8f1bcd19-e578-4b18-b324-7cc00b80ac0a)

## Libraries Currently

![image](https://github.com/user-attachments/assets/0cf50063-f9db-443c-97b4-3ef0664b6e6e)

### Motivation

There are many data/analysis/research/statistics teams (particularly in academia and pharma) that use R as the default language. R has great integration with most of the newer data techs (arrow, parquet, polars) and having this included could really help in bringing this community into the hugging faces ecosystem.

**This is a small/low-hanging-fruit front end change but would make a big impact expanding the community**

### Your contribution

I am not sure which repositroy this should be in, but I have experience in R, Python and JS and happy to submit a PR in the appropriate repository.


Huggingface GIT returns null as Content-Type instead of application/x-git-receive-pack-result

### Describe the bug

We push changes to our datasets programmatically. Our git client jGit reports that the hf git server returns null as Content-Type after a push.

### Steps to reproduce the bug

A basic kotlin application:
```
   val person = PersonIdent(
        "padmalcom",
        "padmalcom@sth.com"
    )

    val cp = UsernamePasswordCredentialsProvider(
        "padmalcom",
        "mysecrettoken"
    )

    val git =
        KGit.cloneRepository {
            setURI("https://huggingface.co/datasets/sth/images")
            setTimeout(60)
            setProgressMonitor(TextProgressMonitor())
            setCredentialsProvider(cp)
        }

    FileOutputStream("./images/images.csv").apply { writeCsv(images) }
    git.add {
        addFilepattern("images.csv")
    }

    for (i in images) {
        FileUtils.copyFile(
            File("./files/${i.id}"),
            File("./images/${i.id + File(i.fileName).extension }")
        )
        git.add {
            addFilepattern("${i.id + File(i.fileName).extension }")
        }
    }

    val revCommit = git.commit {
        author = person
        message = "Uploading images at " + LocalDateTime.now()
            .format(DateTimeFormatter.ISO_DATE_TIME)
        setCredentialsProvider(cp)
    }

    val push = git.push {
        setCredentialsProvider(cp)
    }
```

### Expected behavior

The git server is expected to return the Content-Type _application/x-git-receive-pack-result_.

### Environment info

It is independent from the datasets library.


fallback to default feature casting in case custom features not available during dataset loading

a fix for #7223 in case datasets is happy to support this kind of extensibility! seems cool / powerful for allowing sharing of datasets with potentially different feature types


Fallback to arrow defaults when loading dataset with custom features that aren't registered locally

### Describe the bug

Datasets allows users to create and register custom features.

However if datasets are then pushed to the hub, this means that anyone calling load_dataset without registering the custom Features in the same way as the dataset creator will get an error message.

It would be nice to offer a fallback in this case.

### Steps to reproduce the bug

```python
load_dataset("alex-hh/custom-features-example")
```
(Dataset creation process - must be run in separate session so that NewFeature isn't registered in session in which download is attempted:)

```python
from dataclasses import dataclass, field
import pyarrow as pa
from datasets.features.features import register_feature

from datasets import Dataset, Features, Value, load_dataset
from datasets import Feature

@dataclass
class NewFeature(Feature):
    _type: str = field(default="NewFeature", init=False, repr=False)
    def __call__(self):
        return pa.int32()

def examples_generator():
    for i in range(5):
        yield {"feature": i}

ds = Dataset.from_generator(examples_generator, features=Features(feature=NewFeature()))
ds.push_to_hub("alex-hh/custom-features-example")
register_feature(NewFeature, "NewFeature")
```

### Expected behavior

It would be nice, and offer greater extensibility, if there was some kind of graceful fallback mechanism in place for cases where user-defined features are stored in the dataset but not available locally.

### Environment info

3.0.2


TypeError: Couldn't cast array of type string to null in long json

### Describe the bug

In general, changing the type from string to null is allowed within a dataset — there are even examples of this in the documentation.

However, if the dataset is large and unevenly distributed, this allowance stops working. The schema gets locked in after reading a chunk.

Consequently, if all values in the first chunk of a field are, for example, null, the field will be locked as type null, and if a string appears in that field in the second chunk, it will trigger this error:

<details>
  <summary>Traceback </summary>

```
   TypeError                                 Traceback (most recent call last)

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1868                     try:
-> 1869                         writer.write_table(table)
   1870                     except CastError as cast_error:

14 frames

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in write_table(self, pa_table, writer_batch_size)
    579         pa_table = pa_table.combine_chunks()
--> 580         pa_table = table_cast(pa_table, self._schema)
    581         if self.embed_local_files:

[/usr/local/lib/python3.10/dist-packages/datasets/table.py](https://localhost:8080/#) in table_cast(table, schema)
   2291     if table.schema != schema:
-> 2292         return cast_table_to_schema(table, schema)
   2293     elif table.schema.metadata != schema.metadata:

[/usr/local/lib/python3.10/dist-packages/datasets/table.py](https://localhost:8080/#) in cast_table_to_schema(table, schema)
   2244         )
-> 2245     arrays = [
   2246         cast_array_to_feature(

[/usr/local/lib/python3.10/dist-packages/datasets/table.py](https://localhost:8080/#) in <listcomp>(.0)
   2245     arrays = [
-> 2246         cast_array_to_feature(
   2247             table[name] if name in table_column_names else pa.array([None] * len(table), type=schema.field(name).type),

[/usr/local/lib/python3.10/dist-packages/datasets/table.py](https://localhost:8080/#) in wrapper(array, *args, **kwargs)
   1794         if isinstance(array, pa.ChunkedArray):
-> 1795             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1796         else:

[/usr/local/lib/python3.10/dist-packages/datasets/table.py](https://localhost:8080/#) in <listcomp>(.0)
   1794         if isinstance(array, pa.ChunkedArray):
-> 1795             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1796         else:

[/usr/local/lib/python3.10/dist-packages/datasets/table.py](https://localhost:8080/#) in cast_array_to_feature(array, feature, allow_primitive_to_str, allow_decimal_to_str)
   2101     elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 2102         return array_cast(
   2103             array,

[/usr/local/lib/python3.10/dist-packages/datasets/table.py](https://localhost:8080/#) in wrapper(array, *args, **kwargs)
   1796         else:
-> 1797             return func(array, *args, **kwargs)
   1798

[/usr/local/lib/python3.10/dist-packages/datasets/table.py](https://localhost:8080/#) in array_cast(array, pa_type, allow_primitive_to_str, allow_decimal_to_str)
   1947         if pa.types.is_null(pa_type) and not pa.types.is_null(array.type):
-> 1948             raise TypeError(f"Couldn't cast array of type {_short_str(array.type)} to {_short_str(pa_type)}")
   1949         return array.cast(pa_type)

TypeError: Couldn't cast array of type string to null

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)

[<ipython-input-353-e02f83980611>](https://localhost:8080/#) in <cell line: 1>()
----> 1 dd = load_dataset("json", data_files=["TEST.json"])

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, keep_in_memory, save_infos, revision, token, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   2094
   2095     # Download and prepare data
-> 2096     builder_instance.download_and_prepare(
   2097         download_config=download_config,
   2098         download_mode=download_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, dl_manager, base_path, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    922                     if num_proc is not None:
    923                         prepare_split_kwargs["num_proc"] = num_proc
--> 924                     self._download_and_prepare(
    925                         dl_manager=dl_manager,
    926                         verification_mode=verification_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
    997             try:
    998                 # Prepare split will record examples associated to the split
--> 999                 self._prepare_split(split_generator, **prepare_split_kwargs)
   1000             except OSError as e:
   1001                 raise OSError(

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split(self, split_generator, file_format, num_proc, max_shard_size)
   1738             job_id = 0
   1739             with pbar:
-> 1740                 for job_id, done, content in self._prepare_split_single(
   1741                     gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
   1742                 ):

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1894             if isinstance(e, DatasetGenerationError):
   1895                 raise
-> 1896             raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1897
   1898         yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```

</details>

### Steps to reproduce the bug

```python
import json
from datasets import load_dataset

with open("TEST.json", "w") as f:
    row = {"ballast": "qwerty" * 1000, "b": None}
    row_str = json.dumps(row) + "\n"
    line_size = len(row_str)
    chunk_size = 10 << 20
    lines_in_chunk = chunk_size // line_size + 1
    print(f"Writing {lines_in_chunk} lines")
    for i in range(lines_in_chunk):
        f.write(row_str)
    null_row = {"ballast": "Gotcha", "b": "Not Null"}
    f.write(json.dumps(null_row) + "\n")

load_dataset("json", data_files=["TEST.json"])
```

### Expected behavior

Concatenation of the chunks without errors

### Environment info

- `datasets` version: 3.0.1
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.24.7
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1


add CustomFeature base class to support user-defined features with encoding/decoding logic

intended as fix for #7220 if this kind of extensibility is something that datasets is willing to support!

```python
from datasets.features.features import CustomFeature

class ListOfStrs(CustomFeature):
    requires_encoding = True
    def _encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
feats.encode_example({"strlist": "a"})["strlist"] == feats["strlist"].encode_example("a")
```


Custom features not compatible with special encoding/decoding logic

### Describe the bug

It is possible to register custom features using datasets.features.features.register_feature (https://github.com/huggingface/datasets/pull/6727)

However such features are not compatible with Features.encode_example/decode_example if they require special encoding / decoding logic because encode_nested_example / decode_nested_example checks whether the feature is in a fixed list of encodable types:

https://github.com/huggingface/datasets/blob/16a121d7821a7691815a966270f577e2c503473f/src/datasets/features/features.py#L1349

This prevents the extensibility of features to complex cases

### Steps to reproduce the bug

```python
class ListOfStrs:
    def encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
assert feats.encode_example({"strlist": "a"})["strlist"] = feats["strlist"].encode_example("a")}
```

### Expected behavior

Registered feature types should be encoded based on some property of the feature (e.g. requires_encoding)?

### Environment info

3.0.2


bump fsspec


ds.map(f, num_proc=10) is slower than df.apply

### Describe the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

time cost:
1. df.apply: 100%|██████████| 12500592/12500592 [00:13<00:00, 959825.47it/s]
2. ds.map: Map (num_proc=10):  31%
 3899028/12500592 [00:28<00:38, 222532.89 examples/s]

### Steps to reproduce the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

### Expected behavior

ds.map is ~num_proc faster than df.apply

### Environment info

pandas: 2.2.2
datasets: 2.19.1


Iterable dataset map with explicit features causes slowdown for Sequence features

### Describe the bug

When performing map, it's nice to be able to pass the new feature type, and indeed required by interleave and concatenate datasets.

However, this can cause a major slowdown for certain types of array features due to the features being re-encoded.

This is separate to the slowdown reported in #7206

### Steps to reproduce the bug

```
from datasets import Dataset, Features, Array3D, Sequence, Value
import numpy as np
import time
features=Features(**{"array0": Sequence(feature=Value("float32"), length=-1), "array1": Sequence(feature=Value("float32"), length=-1)})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [5000,10000]*25] for i in range(2)}, features=features)
```

```
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~1.5 s on main

```
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x, features=features)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~ 3 s on main

### Expected behavior

I'm not 100% sure whether passing new feature types to formatted outputs of map should be supported or not, but assuming it should, then there should be a cost-free way to specify the new feature type - knowing feature type is required by interleave_datasets and concatenate_datasets for example

### Environment info

3.0.2


Formatted map + with_format(None) changes array dtype for iterable datasets

### Describe the bug

When applying with_format -> map -> with_format(None), array dtypes seem to change, even if features are passed

### Steps to reproduce the bug

```python
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32")})
dataset = Dataset.from_dict({f"array0": [np.zeros((100,10,10), dtype=np.float32)]*25}, features=features)
ds = dataset.to_iterable_dataset().with_format("numpy").map(lambda x: x, features=features)
ex_0 = next(iter(ds))
ds = dataset.to_iterable_dataset().with_format("numpy").map(lambda x: x, features=features).with_format(None)
ex_1 = next(iter(ds))
assert ex_1["array0"].dtype == ex_0["array0"].dtype, f"{ex_1['array0'].dtype} {ex_0['array0'].dtype}"
```

### Expected behavior

Dtypes should be preserved.

### Environment info

3.0.2


Add with_rank to Dataset.from_generator

New feature or request

### Feature request

Add `with_rank` to `Dataset.from_generator` similar to `Dataset.map` and `Dataset.filter`.

### Motivation

As for `Dataset.map` and `Dataset.filter`, this is useful when creating cache files using multi-GPU, where the rank can be used to select GPU IDs. For now, rank can be added in the `gen_kwars` argument; however, this, in turn, includes the rank when computing the fingerprint.

### Your contribution

Added #7199 which passes rank based on the `job_id` set by `num_proc`.


Windows do not supprot signal.alarm and singal.signal

### Describe the bug

signal.alarm and signal.signal are used in the load.py module, but these are not supported by Windows.

### Steps to reproduce the bug

 lighteval accelerate --model_args "pretrained=gpt2,trust_remote_code=True" --tasks "community|kinit_sts" --custom_tasks "community_tasks/kinit_evals.py" --output_dir "./evals"

### Expected behavior

proceed with input(..) method

### Environment info

Windows 11


Describe only selected fields in README

New feature or request

### Feature request

Hi Datasets team!

Is it possible to add the ability to describe only selected fields of the dataset files in `README.md`? For example, I have this open dataset ([open-llm-leaderboard/results](https://huggingface.co/datasets/open-llm-leaderboard/results?row=0)) and I want to describe only some fields in order not to overcomplicate the Dataset Preview and filter out some fields

### Motivation

The `Results` dataset for the Open LLM Leaderboard contains json files with a complex nested structure. I would like to add `README.md` there to use the SQL console, for example. But if I describe the structure of this dataset completely, it will overcomplicate the use of Dataset Preview and the total number of columns will exceed 50

### Your contribution

I'm afraid I'm not familiar with the project structure, so I won't be able to open a PR, but I'll try to help with something else if possible


Convert Array features to numpy arrays rather than lists by default

New feature or request

### Feature request

It is currently quite easy to cause massive slowdowns when using datasets and not familiar with the underlying data conversions by e.g. making bad choices of formatting.

Would it be more user-friendly to set defaults that avoid this as much as possible? e.g. format Array features as numpy arrays rather than python lists

### Motivation

Default array formatting leads to slow performance: e.g.

```python
import numpy as np
from datasets import Dataset, Features, Array3D
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```

```python
t0 = time.time()
for ex in ds:
   pass
t1 = time.time()
```
~1.4 s

```python
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~10s

```python
ds = dataset.with_format("numpy")
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~0.04s

```python
ds = dataset.to_iterable_dataset().with_format("numpy")
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~0.04s

### Your contribution

May be able to contribute


Preserve features in iterable dataset.filter

Fixes example in #7208 - I'm not sure what other checks I should do? @lhoestq

I also haven't thought hard about the concatenate / interleaving example iterables but think this might work assuming that features are either all identical or None?


Iterable dataset.filter should not override features

### Describe the bug

When calling filter on an iterable dataset, the features get set to None

### Steps to reproduce the bug

import numpy as np
import time
from datasets import Dataset, Features, Array3D

```python
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
ds = dataset.to_iterable_dataset()
orig_column_names = ds.column_names
ds = ds.filter(lambda x: True)
assert ds.column_names == orig_column_names
```

### Expected behavior

Filter should preserve features information

### Environment info

3.0.2


apply formatting after iter_arrow to speed up format -> map, filter for iterable datasets

I got to this by hacking around a bit but it seems to solve #7206

I have no idea if this approach makes sense or would break something else?

Could maybe work on a full pr if this looks reasonable @lhoestq ? I imagine the same issue might affect other iterable dataset methods?


Slow iteration for iterable dataset with numpy formatting for array data

### Describe the bug

When working with large arrays, setting with_format to e.g. numpy then applying map causes a significant slowdown for iterable datasets.

### Steps to reproduce the bug

```python
import numpy as np
import time
from datasets import Dataset, Features, Array3D

features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```

Then
```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
print(t1-t0)
```

takes 27 s, whereas

```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy")
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
print(t1 - t0)
```

takes ~1s

### Expected behavior

Map should not introduce a slowdown when formatting is enabled.

### Environment info

3.0.2


fix ci benchmark

we're not using the benchmarks anymore + they were not working anyway due to token permissions

I keep the code in case we ever want to re-run the benchmark manually


fix unbatched arrow map for iterable datasets

Fixes the bug when applying map to an arrow-formatted iterable dataset described here:

https://github.com/huggingface/datasets/issues/6833#issuecomment-2399903885

```python

from datasets import load_dataset
ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
ds = ds.with_format("arrow").map(lambda x: x)
for ex in ds:
    pass
```

@lhoestq


with_format docstring

reported at https://github.com/huggingface/datasets/issues/3444


`from_parquet` return type annotation

### Describe the bug

As already posted in https://github.com/microsoft/pylance-release/issues/6534, the correct type hinting fails when building a dataset using the `from_parquet` constructor.
Their suggestion is to comprehensively annotate the method's return type to better align with the docstring information.

### Steps to reproduce the bug

```python
from datasets import Dataset

dataset = Dataset.from_parquet(path_or_paths="file")
dataset.map(lambda x: {"new": x["old"]}, batched=True)
```

### Expected behavior

map is a [valid](https://huggingface.co/docs/datasets/v3.0.1/en/package_reference/main_classes#datasets.Dataset.map), no error should be thrown.

### Environment info

- `datasets` version: 3.0.1
- Platform: macOS-15.0.1-arm64-arm-64bit
- Python version: 3.12.6
- `huggingface_hub` version: 0.25.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1


`load_dataset()` of images from a single directory where `train.png` image exists

### Describe the bug

Hey!
Firstly, thanks for maintaining such framework!

I had a small issue, where I wanted to load a custom dataset of image+text captioning. I had all of my images in a single directory, and one of the images had the name `train.png`. Then, the loaded dataset had only this image.

I guess it's related to "train" as a split name, but it's definitely an unexpected behavior :)
Unfortunately I don't have time to submit a proper PR. I'm attaching a toy example to reproduce the issue.

Thanks,
Sagi

### Steps to reproduce the bug

All of the steps I'm attaching are in a fresh env :)

```
(base) sagipolaczek@Sagis-MacBook-Pro ~ % conda activate hf_issue_env
(hf_issue_env) sagipolaczek@Sagis-MacBook-Pro ~ % python --version
Python 3.10.15
(hf_issue_env) sagipolaczek@Sagis-MacBook-Pro ~ % pip list | grep datasets
datasets           3.0.1
(hf_issue_env) sagipolaczek@Sagis-MacBook-Pro ~ % ls -la Documents/hf_datasets_issue
total 352
drwxr-xr-x   6 sagipolaczek  staff     192 Oct  7 11:59 .
drwx------@ 23 sagipolaczek  staff     736 Oct  7 11:46 ..
-rw-r--r--@  1 sagipolaczek  staff      72 Oct  7 11:59 metadata.csv
-rw-r--r--@  1 sagipolaczek  staff  160154 Oct  6 18:00 pika.png
-rw-r--r--@  1 sagipolaczek  staff    5495 Oct  6 12:02 pika_pika.png
-rw-r--r--@  1 sagipolaczek  staff    1753 Oct  6 11:50 train.png
(hf_issue_env) sagipolaczek@Sagis-MacBook-Pro ~ % cat Documents/hf_datasets_issue/metadata.csv
file_name,text
train.png,A train
pika.png,Pika
pika_pika.png,Pika Pika!

(hf_issue_env) sagipolaczek@Sagis-MacBook-Pro ~ % python
Python 3.10.15 (main, Oct  3 2024, 02:33:33) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("imagefolder", data_dir="Documents/hf_datasets_issue/")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['image', 'text'],
        num_rows: 1
    })
})
>>> dataset["train"][0]
{'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=354x84 at 0x10B50FD90>, 'text': 'A train'}

### DELETING `train.png` sample ###
(hf_issue_env) sagipolaczek@Sagis-MacBook-Pro ~ % vim Documents/hf_datasets_issue/metadata.csv
(hf_issue_env) sagipolaczek@Sagis-MacBook-Pro ~ % rm Documents/hf_datasets_issue/train.png
(hf_issue_env) sagipolaczek@Sagis-MacBook-Pro ~ % python
Python 3.10.15 (main, Oct  3 2024, 02:33:33) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("imagefolder", data_dir="Documents/hf_datasets_issue/")
Generating train split: 2 examples [00:00, 65.99 examples/s]
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['image', 'text'],
        num_rows: 2
    })
})
>>> dataset["train"]
Dataset({
    features: ['image', 'text'],
    num_rows: 2
})
>>> dataset["train"][0],dataset["train"][1]
({'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=2356x1054 at 0x10DD11E70>, 'text': 'Pika'}, {'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=343x154 at 0x10E258C70>, 'text': 'Pika Pika!'})
```

### Expected behavior

My expected behavior would be to get a dataset with the sample `train.png` in it (along with the others data points).

### Environment info

I've attached it in the example:

Python 3.10.15
datasets           3.0.1


Fix the environment variable for huggingface cache

Resolve  #6256. As far as I tested, `HF_DATASETS_CACHE` was ignored and I could not specify the cache directory at all except for the default one by this environment variable. `HF_HOME` has worked. Perhaps the recent change on file downloading by `huggingface_hub` could affect this bug.

In my testing, I could not specify the cache directory even by `load_dataset("dataset_name" cache_dir="...")`. It might be another issue. I also welcome any advice to solve this issue.


Add  with_rank to Dataset.from_generator

Adds `with_rank` to `Dataset.from_generator`.  As for `Dataset.map` and `Dataset.filter`, this is useful when creating cache files using multi-GPU.


Add repeat method to datasets

Following up on discussion in #6623 and #7198 I thought this would be pretty useful for my case so had a go at implementing.

My main motivation is to be able to call iterable_dataset.repeat(None).take(samples_per_epoch) to safely avoid timeout issues in a distributed training setting. This would provide a straightforward workaround for several open issues related to this situation:  https://github.com/huggingface/datasets/issues/6437, https://github.com/huggingface/datasets/issues/6594, https://github.com/huggingface/datasets/issues/6623, https://github.com/huggingface/datasets/issues/6719.

@lhoestq let me know if this looks on the right track!


ConnectionError: Couldn't reach 'allenai/c4' on the Hub (ConnectionError)数据集下不下来，怎么回事

### Describe the bug

from datasets import load_dataset
print("11")
traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')
print("22")
valdata = load_dataset('ptb_text_only',
                           'penn_treebank',
                           split='validation')

### Steps to reproduce the bug

1

### Expected behavior

1

### Environment info

1


concatenate_datasets does not preserve shuffling state

### Describe the bug

After concatenate datasets on an iterable dataset, the shuffling state is destroyed, similar to #7156

This means concatenation cant be used for resolving uneven numbers of samples across devices when using iterable datasets in a distributed setting as discussed in #6623

I also noticed that the number of shards is the same after concatenation, which I found surprising, but I don't understand the internals well enough to know whether this is actually surprising or not

### Steps to reproduce the bug

```python
import datasets
import torch.utils.data

def gen(shards):
    yield {"shards": shards}

def main():
    dataset1 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25))}  # TODO: how to understand this?
    )
    dataset2 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25, 50))}  # TODO: how to understand this?
    )
    dataset1 = dataset1.shuffle(buffer_size=1)
    dataset2 = dataset2.shuffle(buffer_size=1)
    print(dataset1.n_shards)
    print(dataset2.n_shards)

    dataset = datasets.concatenate_datasets(
        [dataset1, dataset2]
    )
    print(dataset.n_shards)
    # dataset = dataset1

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=0,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
    print("\nNew epoch")

    dataset = dataset.set_epoch(1)

    for i, batch in enumerate(dataloader):
        print(batch)

if __name__ == "__main__":
    main()
```

### Expected behavior

Shuffling state should be preserved

### Environment info

Latest datasets


Add support for 3D datasets

New feature or request

See https://huggingface.co/datasets/allenai/objaverse for example