`load_dataset` from Hub with `name` to specify `config` using incorrect builder type when multiple data formats are present

Following [documentation](https://huggingface.co/docs/datasets/repository_structure#define-your-splits-and-subsets-in-yaml) I had defined different configs for [`Dataception`](https://huggingface.co/datasets/bigdata-pw/Dataception), a dataset of datasets:

```yaml
configs:
- config_name: dataception
  data_files:
  - path: dataception.parquet
    split: train
  default: true
- config_name: dataset_5423
  data_files:
  - path: datasets/5423.tar
    split: train
...
- config_name: dataset_721736
  data_files:
  - path: datasets/721736.tar
    split: train
```

The intent was for metadata to be browsable via Dataset Viewer, in addition to each individual dataset, and to allow datasets to be loaded by specifying the config/name to `load_dataset`.

While testing `load_dataset` I encountered the following error:

```python
>>> dataset = load_dataset("bigdata-pw/Dataception", "dataset_7691")
Downloading readme: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 467k/467k [00:00<00:00, 1.99MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 71.0M/71.0M [00:02<00:00, 26.8MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "datasets\load.py", line 2145, in load_dataset
    builder_instance.download_and_prepare(
  File "datasets\builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "datasets\builder.py", line 1100, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "datasets\packaged_modules\parquet\parquet.py", line 58, in _split_generators
    self.info.features = datasets.Features.from_arrow_schema(pq.read_schema(f))
                                                             ^^^^^^^^^^^^^^^^^
  File "pyarrow\parquet\core.py", line 2325, in read_schema
    file = ParquetFile(
           ^^^^^^^^^^^^
  File "pyarrow\parquet\core.py", line 318, in __init__
    self.reader.open(
  File "pyarrow\_parquet.pyx", line 1470, in pyarrow._parquet.ParquetReader.open
  File "pyarrow\error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```

The correct file is downloaded, however the incorrect builder type is detected; `parquet` due to other content of the repository. It would appear that the config needs to be taken into account.

Note that I have removed the additional configs from the repository because of this issue and there is a limit of 3000 configs anyway so the Dataset Viewer doesn't work as I intended. I'll add them back in if it assists with testing.


IterableDataset: cannot resolve features from list of numpy arrays

### Describe the bug

when resolve features of `IterableDataset`, got `pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values` error.

```
Traceback (most recent call last):
  File "test.py", line 6
    iter_ds = iter_ds._resolve_features()
  File "lib/python3.10/site-packages/datasets/iterable_dataset.py", line 2876, in _resolve_features
    features = _infer_features_from_batch(self.with_format(None)._head())
  File "lib/python3.10/site-packages/datasets/iterable_dataset.py", line 63, in _infer_features_from_batch
    pa_table = pa.Table.from_pydict(batch)
  File "pyarrow/table.pxi", line 1813, in pyarrow.lib._Tabular.from_pydict
  File "pyarrow/table.pxi", line 5339, in pyarrow.lib._from_pydict
  File "pyarrow/array.pxi", line 374, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 344, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 42, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
```

### Steps to reproduce the bug

```python
from datasets import Dataset
import numpy as np

# create list of numpy
iter_ds = Dataset.from_dict({'a': [[[1, 2, 3], [1, 2, 3]]]}).to_iterable_dataset().map(lambda x: {'a': [np.array(x['a'])]})
iter_ds = iter_ds._resolve_features()  # errors here
```

### Expected behavior

features can be successfully resolved

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-5.15.0-94-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Set dev version


Release: 2.21.0


Some of DownloadConfig's properties are always being overridden in load.py

### Describe the bug

The `extract_compressed_file` and `force_extract` properties of DownloadConfig are always being set to True in the function `dataset_module_factory` in the `load.py` file. This behavior is very annoying because data extracted will just be ignored the next time the dataset is loaded.

See this image below:
![image](https://github.com/user-attachments/assets/9e76ebb7-09b1-4c95-adc8-a959b536f93c)

### Steps to reproduce the bug

1. Have a local dataset that contains archived files (zip, tar.gz, etc)
2. Build a dataset loading script to download and extract these files
3. Run the load_dataset function with a DownloadConfig that specifically set `force_extract` to False
4. The extraction process will start no matter if the archives was extracted previously

### Expected behavior

The extraction process should not run when the archives were previously extracted and `force_extract` is set to False.

### Environment info

datasets==2.20.0
python3.9


Automatically create `cache_dir` from `cache_file_name`

You get a pretty unhelpful error message when specifying a `cache_file_name` in a directory that doesn't exist, e.g. `cache_file_name="./cache/data.map"`

```python
import datasets

cache_file_name="./cache/train.map"
dataset = datasets.load_dataset("ylecun/mnist")
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name)
```

```
FileNotFoundError: [Errno 2] No such file or directory: '/.../cache/tmp48r61siw'
```

It is simple enough to create and I was expecting that this would have been the case.

cc: @albertvillanova @lhoestq


Add Arabic Docs to Datasets

Translate Docs into Arabic  issue-number : #7093
[Arabic Docs](https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx)
[English Docs](https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/en/index.mdx)

@stevhliu


Add Arabic Docs to datasets

enhancement

New feature or request

### Feature request

Add Arabic Docs to datasets
[Datasets Arabic](https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx)

### Motivation

@AhmedAlmaghz
https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx

### Your contribution

@AhmedAlmaghz
https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx


load_dataset with multiple jsonlines files interprets datastructure too early

### Describe the bug

likely related to #6460

using `datasets.load_dataset("json", data_dir= ... )` with multiple `.jsonl` files will error if one of the files (maybe the first file?) contains a full column of empty data.

### Steps to reproduce the bug

real world example:
data is available in this [PR-branch](https://github.com/Vipitis/shadertoys-dataset/pull/3/commits/cb1e7157814f74acb09d5dc2f1be3c0a868a9933). Because my files are chunked by months, some months contain all empty data for some columns, just by chance - these are `[]`. Otherwise it's all the same structure.

```python
from datasets import load_dataset
ds = load_dataset("json", data_dir="./data/annotated/api")
```

you get a long error trace, where in the middle it says something like
```cs
TypeError: Couldn't cast array of type struct<id: int64, src: string, ctype: string, channel: int64, sampler: struct<filter: string, wrap: string, vflip: string, srgb: string, internal: string>, published: int64> to null
```

toy example: (on request)

### Expected behavior

Some suggestions
1. give a better error message to the user
2. consider all files before deciding on a data structure for a given column.
3. if you encounter a new structure, and can't cast that to null, replace the null-hypothesis. (maybe something for pyarrow)

as a workaround I have lazily implemented the following (essentially step 2)
```python
import os
import jsonlines
import datasets

api_files = os.listdir("./data/annotated/api")

api_files = [f"./data/annotated/api/{f}" for f in api_files]

api_file_contents = []
for f in api_files:
    with jsonlines.open(f) as reader:
        for obj in reader:
            api_file_contents.append(obj)

ds = datasets.Dataset.from_list(api_file_contents)
```
this works fine for my usecase, but is potentially slower and less memory efficient for really large datasets (where this is unlikely to happen in the first place).

### Environment info

- `datasets` version: 2.20.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.4
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2023.10.0


The test test_move_script_doesnt_change_hash fails because it runs the 'python' command while the python executable has a different name

### Describe the bug

Tests should use the same pythin path as they are launched with, which in the case of FreeBSD is /usr/local/bin/python3.11

Failure:
```
                if err_filename is not None:
>                   raise child_exception_type(errno_num, err_msg, err_filename)
E                   FileNotFoundError: [Errno 2] No such file or directory: 'python'
```

### Steps to reproduce the bug

regular test run using PyTest

### Expected behavior

n/a

### Environment info

FreeBSD 14.1


Missing pyspark dependency causes the testsuite to error out, instead of a few tests to be skipped

### Describe the bug

see the subject

### Steps to reproduce the bug

regular tests

### Expected behavior

n/a

### Environment info

version 2.20.0


Disable warning when using with_format format on tensors

New feature or request

### Feature request

If we write this code:

```python
"""Get data and define datasets."""

from enum import StrEnum
from datasets import load_dataset
from torch.utils.data import DataLoader
from torchvision import transforms

class Split(StrEnum):
    """Describes what type of split to use in the dataloader"""

    TRAIN = "train"
    TEST = "test"
    VAL = "validation"

class ImageNetDataLoader(DataLoader):
    """Create an ImageNetDataloader"""

    _preprocess_transform = transforms.Compose(
        [
            transforms.Resize(256),
            transforms.CenterCrop(224),
        ]
    )

    def __init__(self, batch_size: int = 4, split: Split = Split.TRAIN):
        dataset = (
            load_dataset(
                "imagenet-1k",
                split=split,
                trust_remote_code=True,
                streaming=True,
            )
            .with_format("torch")
            .map(self._preprocess)
        )

        super().__init__(dataset=dataset, batch_size=batch_size)

    def _preprocess(self, data):
        if data["image"].shape[0] < 3:
            data["image"] = data["image"].repeat(3, 1, 1)
        data["image"] = self._preprocess_transform(data["image"].float())
        return data

if __name__ == "__main__":

    dataloader = ImageNetDataLoader(batch_size=2)
    for batch in dataloader:
        print(batch["image"])
        break
```
This will trigger an user warning :

```bash
datasets\formatting\torch_formatter.py:85: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return torch.tensor(value, **{**default_dtype, **self.torch_tensor_kwargs})
```

### Motivation

This happens because the the way the formatted tensor is returned in `TorchFormatter._tensorize`.
This function handle values of different types, according to some tests it seems that possible value types are `int`, `numpy.ndarray` and `torch.Tensor`.
In particular this warning is triggered when the value type is `torch.Tensor`, because is not the suggested Pytorch way of doing it:

- https://stackoverflow.com/questions/55266154/pytorch-preferred-way-to-copy-a-tensor
- https://discuss.pytorch.org/t/it-is-recommended-to-use-source-tensor-clone-detach-or-sourcetensor-clone-detach-requires-grad-true/101218#:~:text=The%20warning%20points%20to%20wrapping%20a%20tensor%20in%20torch.tensor%2C%20which%20is%20not%20recommended.%0AInstead%20of%20torch.tensor(outputs)%20use%20outputs.clone().detach()%20or%20the%20same%20with%20.requires_grad_(True)%2C%20if%20necessary.

### Your contribution

A solution that I found to be working is to change the current way of doing it:

```python
return torch.tensor(value, **{**default_dtype, **self.torch_tensor_kwargs})
```

To:

```python
if (isinstance(value, torch.Tensor)):
    tensor = value.clone().detach()
    if self.torch_tensor_kwargs.get('requires_grad', False):
        tensor.requires_grad_()
    return tensor
else:
    return torch.tensor(value, **{**default_dtype, **self.torch_tensor_kwargs})
```


Unable to create dataset card for Lushootseed language

New feature or request

### Feature request

While I was creating the dataset which contained all documents from the Lushootseed Wikipedia, the dataset card asked me to enter which language the dataset was in. Since Lushootseed is a critically endangered language, it was not available as one of the options. Is it possible to allow entering languages that aren't available in the options?

### Motivation

I'd like to add more information about my dataset in the dataset card, and the language is one of the most important pieces of information, since the entire dataset is primarily concerned collecting Lushootseed documents.

### Your contribution

I can submit a pull request


load_dataset ignores cached datasets and tries to hit HF Hub, resulting in API rate limit errors

### Describe the bug

I have been running lm-eval-harness a lot which has results in an API rate limit. This seems strange, since all of the data should be cached locally. I have in fact verified this.

### Steps to reproduce the bug

1. Be Me
2. Run `load_dataset("TAUR-Lab/MuSR")`
3. Hit rate limit error
4. Dataset is in .cache/huggingface/datasets
5. ???

### Expected behavior

We should not run into API rate limits if we have cached the dataset

### Environment info

datasets 2.16.0
python 3.10.4


[Regression] IterableDataset is broken on 2.20.0

### Describe the bug

In the latest version of datasets there is a major regression, after creating an `IterableDataset` from a generator and applying a few operations (`map`, `select`), you can no longer iterate through the dataset multiple times.

The issue seems to stem from the recent addition of "resumable IterableDatasets" (#6658) (@lhoestq). It seems like it's keeping state when it shouldn't.

### Steps to reproduce the bug

Minimal Reproducible Example (comparing `datasets==2.17.0` and `datasets==2.20.0`)

```
#!/bin/bash

# List of dataset versions to test
versions=("2.17.0" "2.20.0")

# Loop through each version
for version in "${versions[@]}"; do
    # Install the specific version of the datasets library
    pip3 install -q datasets=="$version" 2>/dev/null

    # Run the Python script
    python3 - <<EOF
from datasets import IterableDataset
from datasets.features.features import Features, Value

def test_gen():
    yield from [{"foo": i} for i in range(10)]

features = Features([("foo", Value("int64"))])
d = IterableDataset.from_generator(test_gen, features=features)
mapped = d.map(lambda row: {"foo": row["foo"] * 2})
column = mapped.select_columns(["foo"])
print("Version $version - Iterate Once:", list(column))
print("Version $version - Iterate Twice:", list(column))
EOF
done
```

The output looks like this:
```
Version 2.17.0 - Iterate Once: [{'foo': 0}, {'foo': 2}, {'foo': 4}, {'foo': 6}, {'foo': 8}, {'foo': 10}, {'foo': 12}, {'foo': 14}, {'foo': 16}, {'foo': 18}]
Version 2.17.0 - Iterate Twice: [{'foo': 0}, {'foo': 2}, {'foo': 4}, {'foo': 6}, {'foo': 8}, {'foo': 10}, {'foo': 12}, {'foo': 14}, {'foo': 16}, {'foo': 18}]
Version 2.20.0 - Iterate Once: [{'foo': 0}, {'foo': 2}, {'foo': 4}, {'foo': 6}, {'foo': 8}, {'foo': 10}, {'foo': 12}, {'foo': 14}, {'foo': 16}, {'foo': 18}]
Version 2.20.0 - Iterate Twice: []
```

### Expected behavior

The expected behavior is it version 2.20.0 should behave the same as 2.17.0.

### Environment info

`datasets==2.20.0` on any platform.


More easily support streaming local files

New feature or request

### Feature request

Simplify downloading and streaming datasets locally. Specifically, perhaps add an option to `load_dataset(..., streaming="download_first")` or add better support for streaming symlinked or arrow files.

### Motivation

I have downloaded FineWeb-edu locally and currently trying to stream the dataset from the local files. I have both the raw parquet files using `hugginface-cli download --repo-type dataset HuggingFaceFW/fineweb-edu` and the processed arrow files using `load_dataset("HuggingFaceFW/fineweb-edu")`.

Streaming the files locally does not work well for both file types for two different reasons.

**Arrow files**

When running `load_dataset("arrow", data_files={"train": "~/.cache/huggingface/datasets/HuggingFaceFW___fineweb-edu/default/0.0.0/5b89d1ea9319fe101b3cbdacd89a903aca1d6052/fineweb-edu-train-*.arrow"})` resolving the data files is fast, but because `arrow` is not included in the known [extensions file list](https://github.com/huggingface/datasets/blob/ce4a0c573920607bc6c814605734091b06b860e7/src/datasets/utils/file_utils.py#L738) , all files are opened and scanned to determine the compression type. Adding `arrow` to the known extension types resolves this issue.

**Parquet files**

When running `load_dataset("arrow", data_files={"train": "~/.cache/huggingface/hub/dataset-HuggingFaceFW___fineweb-edu/snapshots/5b89d1ea9319fe101b3cbdacd89a903aca1d6052/data/CC-MAIN-*/train-*.parquet"})` the paths do not get resolved because the parquet files are symlinked from the blobs (which contain all files in case there are different versions). This occurs because the [pattern matching](https://github.com/huggingface/datasets/blob/ce4a0c573920607bc6c814605734091b06b860e7/src/datasets/data_files.py#L389) checks if the path is a file and does not check for symlinks. Symlinks (at least on my machine) are of type "other".

### Your contribution

I have created a PR for fixing arrow file streaming and symlinks. However, I have not checked locally if the tests work or new tests need to be added.

IMO, the easiest option would be to add a `streaming=download_first` option, but I'm afraid that exceeds my current knowledge of how the datasets library works. https://github.com/huggingface/datasets/pull/7083


fix streaming from arrow files


Support HTTP authentication in non-streaming mode, by support passing HTTP storage_options in non-streaming mode.
- Note that currently, HTTP authentication is supported only in streaming mode.

For example, this is necessary if a remote HTTP host requires authentication to download the data.


Set load_from_disk path type as PathLike

Set `load_from_disk` path type as `PathLike`. This way it is aligned with `save_to_disk`.


Generating train split takes a long time

### Describe the bug

Loading a simple webdataset takes ~45 minutes.

### Steps to reproduce the bug

```
from datasets import load_dataset
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M")
```

### Expected behavior

The dataset should load immediately as it does when loaded through a normal indexed WebDataset loader. Generating splits should be optional and there should be a message showing how to disable it.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


HfHubHTTPError: 500 Server Error: Internal Server Error for url:

### Describe the bug

newly uploaded datasets, since yesterday, yields an error.

old datasets, works fine.

Seems like the datasets api server returns a 500

I'm getting the same error, when I invoke `load_dataset` with my dataset.

Long discussion about it here, but I'm not sure anyone from huggingface have seen it.
https://discuss.huggingface.co/t/hfhubhttperror-500-server-error-internal-server-error-for-url/99580/1

### Steps to reproduce the bug

this api url:

https://huggingface.co/api/datasets/neoneye/simon-arc-shape-v4-rev3

respond with:

```
{"error":"Internal Error - We're working hard to fix this as soon as possible!"}
```

### Expected behavior

return no error with newer datasets.

With older datasets I can load the datasets fine.

### Environment info

# Browser
When I access the api in the browser:
https://huggingface.co/api/datasets/neoneye/simon-arc-shape-v4-rev3

```
{"error":"Internal Error - We're working hard to fix this as soon as possible!"}
```

### Request headers

```
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding	gzip, deflate, br, zstd
Accept-Language	en-US,en;q=0.5
Connection	keep-alive
Host	huggingface.co
Priority	u=1
Sec-Fetch-Dest	document
Sec-Fetch-Mode	navigate
Sec-Fetch-Site	cross-site
Upgrade-Insecure-Requests	1
User-Agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0
```

### Response headers

```
X-Firefox-Spdy	h2
access-control-allow-origin	https://huggingface.co
access-control-expose-headers	X-Repo-Commit,X-Request-Id,X-Error-Code,X-Error-Message,X-Total-Count,ETag,Link,Accept-Ranges,Content-Range
content-length	80
content-type	application/json; charset=utf-8
cross-origin-opener-policy	same-origin
date	Fri, 26 Jul 2024 19:09:45 GMT
etag	W/"50-9qrwU+BNI4SD0Fe32p/nofkmv0c"
referrer-policy	strict-origin-when-cross-origin
vary	Origin
via	1.1 1624c79cd07e6098196697a6a7907e4a.cloudfront.net (CloudFront)
x-amz-cf-id	SP8E7n5qRaP6i9c9G83dNAiOzJBU4GXSrDRAcVNTomY895K35H0nJQ==
x-amz-cf-pop	CPH50-C1
x-cache	Error from cloudfront
x-error-message	Internal Error - We're working hard to fix this as soon as possible!
x-powered-by	huggingface-moon
x-request-id	Root=1-66a3f479-026417465ef42f49349fdca1
```


Fix CI test_convert_to_parquet

Fix `test_convert_to_parquet` by patching `HfApi.preupload_lfs_files` and revert temporary fix:
- #7074


column_names ignored by load_dataset() when loading CSV file

### Describe the bug

load_dataset() ignores the column_names kwarg when loading a CSV file.  Instead, it uses whatever values are on the first line of the file.

### Steps to reproduce the bug

Call `load_dataset` to load data from a CSV file and specify `column_names` kwarg.

### Expected behavior

The resulting dataset should have the specified column names **and** the first line of the file should be considered as data values.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.10.0-30-cloud-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- `huggingface_hub` version: 0.24.2
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


🧪 Do not mock create_commit


Update required soxr version from pre-release to release

Update required `soxr` version from pre-release to release 0.4.0: https://github.com/dofuuz/python-soxr/releases/tag/v0.4.0


Fix CI by temporarily marking test_convert_to_parquet as expected to fail

As a hotfix for CI, temporarily mark test_convert_to_parquet as expected to fail.

Fix #7073.

Revert once root cause is fixed.


CI is broken for convert_to_parquet: Invalid rev id: refs/pr/1 404 error causes RevisionNotFoundError

See: https://github.com/huggingface/datasets/actions/runs/10095313567/job/27915185756
```
FAILED tests/test_hub.py::test_convert_to_parquet - huggingface_hub.utils._errors.RevisionNotFoundError: 404 Client Error. (Request ID: Root=1-66a25839-31ce7b475e70e7db1e4d44c2;b0c8870f-d5ef-4bf2-a6ff-0191f3df0f64)

Revision Not Found for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/test-dataset-5188a8-17219154347516/preupload/refs%2Fpr%2F1.
Invalid rev id: refs/pr/1
```
```
 /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/datasets/hub.py:86: in convert_to_parquet
    dataset.push_to_hub(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/datasets/dataset_dict.py:1722: in push_to_hub
    split_additions, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/datasets/arrow_dataset.py:5511: in _push_parquet_shards_to_hub
    api.preupload_lfs_files(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/huggingface_hub/hf_api.py:4231: in preupload_lfs_files
    _fetch_upload_modes(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py:118: in _inner_fn
    return fn(*args, **kwargs)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/huggingface_hub/_commit_api.py:507: in _fetch_upload_modes
    hf_raise_for_status(resp)
```


nm


Filter hangs

### Describe the bug

When trying to filter my custom dataset, the process hangs, regardless of the lambda function used. It appears to be an issue with the way the Images are being handled. The dataset in question is a preprocessed version of https://huggingface.co/datasets/danaaubakirova/patfig where notably, I have converted the data to the Parquet format.

### Steps to reproduce the bug

```python
from datasets import load_dataset
ds = load_dataset('lcolonn/patfig', split='test')
ds_filtered = ds.filter(lambda row: row['cpc_class'] != 'Y')
```

Eventually I ctrl+C and I obtain this stack trace:

```
>>> ds_filtered = ds.filter(lambda row: row['cpc_class'] != 'Y')
Filter:   0%|                                                                | 0/998 [00:00<?, ? examples/s]Filter:   0%|                                                                | 0/998 [00:35<?, ? examples/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 567, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/fingerprint.py", line 482, in wrapper
    out = func(dataset, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3714, in filter
    indices = self.map(
              ^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 602, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 567, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3161, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3552, in _map_single
    batch = apply_function_on_filtered_inputs(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3421, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 6478, in get_indices_from_mask_function
    num_examples = len(batch[next(iter(batch.keys()))])
                       ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 273, in __getitem__
    value = self.format(key)
            ^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 376, in format
    return self.formatter.format_column(self.pa_table.select([key]))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 443, in format_column
    column = self.python_features_decoder.decode_column(column, pa_table.column_names[0])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 219, in decode_column
    return self.features.decode_column(column, column_name) if self.features else column
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/features/features.py", line 2008, in decode_column
    [decode_nested_example(self[column_name], value) if value is not None else None for value in column]
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/features/features.py", line 2008, in <listcomp>
    [decode_nested_example(self[column_name], value) if value is not None else None for value in column]
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/features/features.py", line 1351, in decode_nested_example
    return schema.decode_example(obj, token_per_repo_id=token_per_repo_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/datasets/features/image.py", line 188, in decode_example
    image.load()  # to avoid "Too many open files" errors
    ^^^^^^^^^^^^
  File "/home/l-walewski/miniconda3/envs/patentqa/lib/python3.11/site-packages/PIL/ImageFile.py", line 293, in load
    n, err_code = decoder.decode(b)
                  ^^^^^^^^^^^^^^^^^
KeyboardInterrupt
```

Warning! This can even seem to cause some computers to crash.

### Expected behavior

Should return the filtered dataset

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- `huggingface_hub` version: 0.24.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


how set_transform affects batch size?

### Describe the bug

I am trying to fine-tune w2v-bert for ASR task. Since my dataset is so big, I preferred to use the on-the-fly method with set_transform. So i change the preprocessing function to this:
```
def prepare_dataset(batch):
    input_features = processor(batch["audio"], sampling_rate=16000).input_features[0]
    input_length = len(input_features)
    labels = processor.tokenizer(batch["text"], padding=False).input_ids

    batch = {
        "input_features": [input_features],
        "input_length": [input_length],
        "labels": [labels]
    }

    return batch

train_ds.set_transform(prepare_dataset)
val_ds.set_transform(prepare_dataset)
```

After this, I also had to change the DataCollatorCTCWithPadding class like this:
```
@dataclass
class DataCollatorCTCWithPadding:
    processor: Wav2Vec2BertProcessor
    padding: Union[bool, str] = True

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        # Separate input_features and labels
        input_features = [{"input_features": feature["input_features"][0]} for feature in features]
        labels = [feature["labels"][0] for feature in features]

        # Pad input features
        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            return_tensors="pt",
        )

        # Pad and process labels
        label_features = self.processor.tokenizer.pad(
            {"input_ids": labels},
            padding=self.padding,
            return_tensors="pt",
        )

        labels = label_features["input_ids"]
        attention_mask = label_features["attention_mask"]

        # Replace padding with -100 to ignore these tokens during loss calculation
        labels = labels.masked_fill(attention_mask.ne(1), -100)

        batch["labels"] = labels

        return batch
```

But now a strange thing is happening, no matter how much I increase the batch size, the amount of V-RAM GPU usage does not change, while the number of total steps in the progress-bar (logging) changes. Is this normal or have I made a mistake?

### Steps to reproduce the bug

i can share my code if needed

### Expected behavior

Equal to the batch size value, the set_transform function is applied to the dataset and given to the model as a batch.

### Environment info

all updated versions


Fix push_to_hub by not calling create_branch if PR branch (e.g. `refs/pr/1`).

Note that currently create_branch raises a 400 Bad Request error if the user passes a PR branch (e.g. `refs/pr/1`).

EDIT:
~~Fix push_to_hub by not calling create_branch if branch exists.~~

Note that currently create_branch raises a 403 Forbidden error even if all these conditions are met:
- exist_ok is passed
- the branch already exists
- the user does not have WRITE permission

Fix #7067.

Related issue:
- https://github.com/huggingface/huggingface_hub/issues/2419


Fix prepare_single_hop_path_and_storage_options

Fix `_prepare_single_hop_path_and_storage_options`:
- Do not pass HF authentication headers and HF user-agent to non-HF HTTP URLs
- Do not overwrite passed `storage_options` nested values:
  - Before, when passed
    ```DownloadConfig(storage_options={"https": {"client_kwargs": {"raise_for_status": True}}})```,
    it was overwritten to
    ```{"https": {"client_kwargs": {"trust_env": True}}}```
  - Now, the result combines both:
    ```{"https": {"client_kwargs": {"trust_env": True, "raise_for_status": True}}}```


Convert_to_parquet fails for datasets with multiple configs

If the dataset has multiple configs, when using the `datasets-cli convert_to_parquet` command to avoid issues with the data viewer caused by loading scripts, the conversion process only successfully converts the data corresponding to the first config. When it starts converting the second config, it throws an error:

```
Traceback (most recent call last):
  File "/opt/anaconda3/envs/dl/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/commands/convert_to_parquet.py", line 83, in run
    dataset.push_to_hub(
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/dataset_dict.py", line 1713, in push_to_hub
    api.create_branch(repo_id, branch=revision, token=token, repo_type="dataset", exist_ok=True)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 358, in hf_raise_for_status
    raise BadRequestError(message, response=response) from e
huggingface_hub.utils._errors.BadRequestError:  (Request ID: Root=1-669fc665-7c2e80d75f4337496ee95402;731fcdc7-0950-4eec-99cf-ce047b8d003f)

Bad request:
Invalid reference for a branch: refs/pr/1
```


One subset per file in repo ?

Right now we consider all the files of a dataset to be the same data, e.g.
```
single_subset_dataset/
├── train0.jsonl
├── train1.jsonl
└── train2.jsonl
```
but in cases like this, each file is actually a different subset of the dataset and should be loaded separately
```
many_subsets_dataset/
├── animals.jsonl
├── trees.jsonl
└── metadata.jsonl
```

It would be nice to detect those subsets automatically using a simple heuristic. For example we can group files together if their paths names are the same except some digits ?


Cannot get item after loading from disk and then converting to iterable.

### Describe the bug

The dataset generated from local file works fine.
```py
root = "/home/data/train"
file_list1 = glob(os.path.join(root, "*part1.flac"))
file_list2 = glob(os.path.join(root, "*part2.flac"))

ds = (
    Dataset.from_dict({"part1": file_list1, "part2": file_list2})
    .cast_column("part1", Audio(sampling_rate=None, mono=False))
    .cast_column("part2", Audio(sampling_rate=None, mono=False))
)

ids = ds.to_iterable_dataset(128)
ids = ids.shuffle(buffer_size=10000, seed=42)
dataloader = DataLoader(ids, num_workers=4, batch_size=8, persistent_workers=True)

for batch in dataloader:
    break
```

But after saving it to disk and then loading it from disk, I cannot get data as expected.
```py
root = "/home/data/train"
file_list1 = glob(os.path.join(root, "*part1.flac"))
file_list2 = glob(os.path.join(root, "*part2.flac"))

ds = (
    Dataset.from_dict({"part1": file_list1, "part2": file_list2})
    .cast_column("part1", Audio(sampling_rate=None, mono=False))
    .cast_column("part2", Audio(sampling_rate=None, mono=False))
)

ds.save_to_disk("./train")

ds = datasets.load_from_disk("./train")

ids = ds.to_iterable_dataset(128)
ids = ids.shuffle(buffer_size=10000, seed=42)
dataloader = DataLoader(ids, num_workers=4, batch_size=8, persistent_workers=True)

for batch in dataloader:
    break
```

After a long time waiting, an error occurs:
```
Loading dataset from disk: 100%|█████████████████████████████████████████████████████████████████████████| 165/165 [00:00<00:00, 6422.18it/s]
Traceback (most recent call last):
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1133, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/multiprocessing/queues.py", line 113, in get
    if not self._poll(timeout):
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/multiprocessing/connection.py", line 424, in _poll
    r = wait([self], timeout)
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 3490529) is killed by signal: Killed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/hanzerui/.vscode-server/extensions/ms-python.debugpy-2024.9.12011011/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/hanzerui/.vscode-server/extensions/ms-python.debugpy-2024.9.12011011/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/hanzerui/.vscode-server/extensions/ms-python.debugpy-2024.9.12011011/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/hanzerui/.vscode-server/extensions/ms-python.debugpy-2024.9.12011011/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/home/hanzerui/.vscode-server/extensions/ms-python.debugpy-2024.9.12011011/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/hanzerui/.vscode-server/extensions/ms-python.debugpy-2024.9.12011011/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/home/hanzerui/workspace/NetEase/test/test_datasets.py", line 60, in <module>
    for batch in dataloader:
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
    data = self._next_data()
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1329, in _next_data
    idx, data = self._get_data()
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1295, in _get_data
    success, data = self._try_get_data()
  File "/home/hanzerui/.conda/envs/mss/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1146, in _try_get_data
    raise RuntimeError(f'DataLoader worker (pid(s) {pids_str}) exited unexpectedly') from e
RuntimeError: DataLoader worker (pid(s) 3490529) exited unexpectedly
```

It seems that streaming is not supported by `laod_from_disk`, so does that mean I cannot convert it to iterable?

### Steps to reproduce the bug

1. Create a `Dataset` from local files with `from_dict`
2. Save it to disk with `save_to_disk`
3. Load it from disk with `load_from_disk`
4. Convert to iterable with `to_iterable_dataset`
5. Loop the dataset

### Expected behavior

Get items faster than the original dataset generated from dict.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- `huggingface_hub` version: 0.23.2
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


Add `batch` method to `Dataset` class

This PR introduces a new `batch` method to the `Dataset` class, aligning its functionality with the `IterableDataset.batch()` method (implemented in #7054). The implementation uses as well the existing `map` method for efficient batching of examples.

Key changes:
- Add `batch` method to `Dataset` class in `arrow_dataset.py`
- Utilize `map` method for batching

Closes #7063

Once the approach is approved, i will create the tests and update the documentation.


Add `batch` method to `Dataset`

New feature or request

### Feature request

Add a `batch` method to the Dataset class, similar to the one recently implemented for `IterableDataset` in PR #7054.

### Motivation

A batched iteration speeds up data loading significantly (see e.g. #6279)

### Your contribution

I plan to open a PR to implement this.


Avoid calling http_head for non-HTTP URLs

Avoid calling `http_head` for non-HTTP URLs, by adding and `else` statement.

Currently, it makes an unnecessary HTTP call (which adds latency) for non-HTTP protocols, like FTP, S3,...

I discovered this while working in an unrelated issue.


Custom Dataset | Still Raise Error while handling errors in _generate_examples

### Describe the bug

I follow this [example](https://discuss.huggingface.co/t/error-handling-in-iterabledataset/72827/3) to handle errors in custom dataset. I am writing a dataset script which read jsonl files and i need to handle errors and continue reading files without raising exception and exit the execution.

```
def _generate_examples(self, filepaths):
        errors=[]

        id_ = 0
        for filepath in filepaths:
            try:
                with open(filepath, 'r') as f:
                    for line in f:
                        json_obj = json.loads(line)

                        yield id_, json_obj
                        id_ += 1
            except Exception as exc:
                logger.error(f"error occur at filepath: {filepath}")
                errors.append(error)
```

seems the logger.error is printed but still exception is raised the the run is exit.

```
Downloading and preparing dataset custom_dataset/default to /home/myuser/.cache/huggingface/datasets/custom_dataset/default-a14cdd566afee0a6/1.0.0/acfcc9fb9c57034b580c4252841
ERROR: datasets_modules.datasets.custom_dataset.acfcc9fb9c57034b580c4252841bb890a5617cbd28678dd4be5e52b81188ad02.custom_dataset: 2024-07-22 10:47:42,167: error occur at filepath: '/home/myuser/ds/corrupted-file.jsonl
Traceback (most recent call last):
  File "/home/myuser/.cache/huggingface/modules/datasets_modules/datasets/custom_dataset/ac..2/custom_dataset.py", line 48, in _generate_examples
    json_obj = json.loads(line)
  File "myenv/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "myenv/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "myenv/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 4 (char 3)
Generating train split: 0 examples [00:06, ? examples/s]>
RemoteTraceback:
"""
Traceback (most recent call last):
  File "myenv/lib/python3.8/site-packages/datasets/builder.py", line 1637, in _prepare_split_single
    num_examples, num_bytes = writer.finalize()
  File "myenv/lib/python3.8/site-packages/datasets/arrow_writer.py", line 594, in finalize
    raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "myenv/lib/python3.8/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "myenv/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 1353, in
_write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "myenv/lib/python3.8/site-packages/datasets/builder.py", line 1646, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
"""

The above exception was the direct cause of the following exception:

│                                                                                │
│ myenv/lib/python3.8/site-packages/datasets/utils/py_utils. │
│ py:1377 in <listcomp>                                                                            │
│                                                                                │
│   1374 │   │   │   │   if all(async_result.ready() for async_result in async_results) and queue  │
│   1375 │   │   │   │   │   break                                                                 │
│   1376 │   │   # we get the result in case there's an error to raise                             │
│ ❱ 1377 │   │   [async_result.get() for async_result in async_results]                            │
│   1378                                                                                │
│                                                                                │
│ ╭──────────────────────────────── locals ─────────────────────────────────╮                      │
│ │           .0 = <list_iterator object at 0x7f2cc1f0ce20>                 │                      │
│ │ async_result = <multiprocess.pool.ApplyResult object at 0x7f2cc1f79c10> │                      │
│ ╰─────────────────────────────────────────────────────────────────────────╯                      │
│                                                                                │
│ myenv/lib/python3.8/site-packages/multiprocess/pool.py:771 │
│ in get                                                                                │
│                                                                                │
│   768 │   │   if self._success:                                                                  │
│   769 │   │   │   return self._value                                                             │
│   770 │   │   else:                                                                              │
│ ❱ 771 │   │   │   raise self._value                                                              │
│   772 │                                                                                │
│   773 │   def _set(self, i, obj):                                                                │
│   774 │   │   self._success, self._value = obj                                                   │
│                                                                                │
│ ╭────────────────────────────── locals ──────────────────────────────╮                           │
│ │    self = <multiprocess.pool.ApplyResult object at 0x7f2cc1f79c10> │                           │
│ │ timeout = None                                                     │                           │
│ ╰────────────────────────────────────────────────────────────────────╯                           │

DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

same as above

### Expected behavior

should handle error and continue reading remaining files

### Environment info

python 3.9


WebDataset BuilderConfig

This PR adds `WebDatasetConfig`.

Closes #7055


None values are skipped when reading jsonl in subobjects

### Describe the bug

I have been fighting against my machine since this morning only to find out this is some kind of a bug.

When loading a dataset composed of `metadata.jsonl`, if you have nullable values (Optional[str]), they can be ignored by the parser, shifting things around.

E.g., let's take this example

Here are two version of a same dataset:

[not-buggy.tar.gz](https://github.com/user-attachments/files/16333532/not-buggy.tar.gz)
[buggy.tar.gz](https://github.com/user-attachments/files/16333553/buggy.tar.gz)

### Steps to reproduce the bug

1. Load the `buggy.tar.gz` dataset
2. Print baseline of `dts = load_dataset("./data")["train"][0]["baselines]`
3. Load the `not-buggy.tar.gz` dataset
4. Print baseline of `dts = load_dataset("./data")["train"][0]["baselines]`

### Expected behavior

Both should have 4 baseline entries:
1. Buggy should have None followed by three lists
2. Non-Buggy should have four lists, and the first one should be an empty list.

One does not work, 2 works. Despite accepting None in another position than the first one.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.5.0-44-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.0
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


New feature type: Document

It would be useful for PDF.

https://github.com/huggingface/dataset-viewer/issues/2991#issuecomment-2242656069


Update load_hub.mdx


Make `BufferShuffledExamplesIterable` resumable

This PR aims to implement a resumable `BufferShuffledExamplesIterable`.
Instead of saving the entire buffer content, which is very memory-intensive, the newly implemented `BufferShuffledExamplesIterable` saves only the minimal state necessary for recovery, e.g., the random generator states and the state of the first example in the buffer dict.

The idea is that since the buffer size is limited, even if the entire buffer is discarded, we can rebuild it as long as the state of the oldest example is recorded. For buffer size $B$, the expected distance between when an example is pushed and when it is yielded is
$d = \sum_{k=1}^{\infty} k\frac{1}{B} (1 - \frac{1}{B} )^{k-1} =B$.
Simulation experiments support these claims:
```py
from random import randint

BUFFER_SIZE = 1024

dists = []
buffer = []
for i in range(10000000):
    if i < BUFFER_SIZE:
        buffer.append(i)
    else:
        index = randint(0, BUFFER_SIZE - 1)
        dists.append(i - buffer[index])
        buffer[index] = i

print(f"MIN DIST: {min(dists)}\nMAX DIST: {max(dists)}\nAVG DIST: {sum(dists) / len(dists):.2f}\n")
```
which produces the following output:
```py
MIN DIST: 1
MAX DIST: 15136
AVG DIST: 1023.95

```
The overall time for reconstructing the buffer and recovery should not be too long.
The following code mimics the cases of resuming online tokenization by `datasets` and `StatefulDataLoader` under distributed scenarios,

```py
import pickle
import time
from itertools import chain
from typing import Any, Dict, List

import torch
from datasets import load_dataset
from torchdata.stateful_dataloader import StatefulDataLoader
from tqdm import tqdm
from transformers import AutoTokenizer, DataCollatorForLanguageModeling

tokenizer = AutoTokenizer.from_pretrained('fla-hub/gla-1.3B-100B')
tokenizer.pad_token = tokenizer.eos_token
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

torch.manual_seed(42)

def tokenize(examples: Dict[str, List[Any]]) -> Dict[str, List[List[int]]]:
    input_ids = tokenizer(examples['text'])['input_ids']
    input_ids = list(chain(*input_ids))
    total_length = len(input_ids)
    chunk_size = 2048
    total_length = (total_length // chunk_size) * chunk_size
    # the last chunk smaller than chunk_size will be discarded
    return {'input_ids': [input_ids[i: i+chunk_size] for i in range(0, total_length, chunk_size)]}

batch_size = 16
num_workers = 5
context_length = 2048
rank = 1
world_size = 32
prefetch_factor = 2
steps = 2048
path = 'fla-hub/slimpajama-test'
dataset = load_dataset(
    path=path,
    split='train',
    streaming=True,
    trust_remote_code=True
)
dataset = dataset.map(tokenize, batched=True, remove_columns=next(iter(dataset)).keys())
dataset = dataset.shuffle(seed=42)
loader = StatefulDataLoader(dataset=dataset,
                            batch_size=batch_size,
                            collate_fn=data_collator,
                            num_workers=num_workers,
                            persistent_workers=False,
                            prefetch_factor=prefetch_factor)
start = time.time()
for i, batch in tqdm(enumerate(loader)):
    if i == 0:
        print(f'{i}\n{batch["input_ids"]}')
    if i == steps - 1:
        print(f'{i}\n{batch["input_ids"]}')
        state_dict = loader.state_dict()
    if i == steps:
        print(f'{i}\n{batch["input_ids"]}')
        break
print(f"{time.time() - start:.2f}s elapsed")
print(f"{len(pickle.dumps(state_dict)) / 1024**2:.2f}MB states in total")
for worker in state_dict['_snapshot']['_worker_snapshots'].keys():
    print(f"{worker} {len(pickle.dumps(state_dict['_snapshot']['_worker_snapshots'][worker])) / 1024**2:.2f}MB")
print(state_dict['_snapshot']['_worker_snapshots']['worker_0']['dataset_state'])

loader = StatefulDataLoader(dataset=dataset,
                            batch_size=batch_size,
                            collate_fn=data_collator,
                            num_workers=num_workers,
                            persistent_workers=False,
                            prefetch_factor=prefetch_factor)
print("Loading state dict")
loader.load_state_dict(state_dict)
start = time.time()
for batch in loader:
    print(batch['input_ids'])
    break

print(f"{time.time() - start:.2f}s elapsed")

```
and the outputs are
```py
0
tensor([[  909,   395, 19082,  ..., 13088, 16232,   395],
        [  601, 28705, 28770,  ..., 28733,   923,   288],
        [21753, 15071, 13977,  ...,  9369, 28723,   415],
        ...,
        [21763, 28751, 20300,  ..., 28781, 28734,  4775],
        [  354,   396, 10214,  ...,   298,   429, 28770],
        [  333,  6149, 28768,  ...,  2773,   340,   351]])
2047
tensor([[28723,   415,  3889,  ...,   272,  3065,  2609],
        [  403,  3214,  3629,  ...,   403, 21163, 16434],
        [28723,    13, 28749,  ..., 28705, 28750, 28734],
        ...,
        [ 2778,  2251, 28723,  ...,   354,   684,   429],
        [ 5659,   298,  1038,  ...,  5290,   297, 22153],
        [  938, 28723,  1537,  ...,  9123, 28733, 12154]])
2048
tensor([[  769,   278, 12531,  ..., 28721, 19309, 28739],
        [  415, 23347,   622,  ...,  3937,  2426, 28725],
        [28745,  4345, 28723,  ...,   338, 28725,   583],
        ...,
        [ 1670, 28709,  5809,  ..., 28734, 28760,   393],
        [  340,  1277,   624,  ...,   325, 28790,  1329],
        [  523,  1144,  3409,  ...,   359,   359, 17422]])
65.97s elapsed
0.00MB states in total
worker_0 0.00MB
worker_1 0.00MB
worker_2 0.00MB
worker_3 0.00MB
worker_4 0.00MB
{'ex_iterable': {'ex_iterable': {'shard_idx': 0, 'shard_example_idx': 14000}, 'num_examples_since_previous_state': 166, 'previous_state_example_idx': 7394, 'previous_state': {'shard_idx': 0, 'shard_example_idx': 13000}}, 'num_taken': 6560, 'global_example_idx': 7560, 'buffer_state_dict': {'num_taken': 6560, 'global_example_idx': 356, 'index_offset': 0, 'first_state': {'ex_iterable': {'shard_idx': 0, 'shard_example_idx': 1000}, 'num_examples_since_previous_state': 356, 'previous_state_example_idx': 0, 'previous_state': {'shard_idx': 0, 'shard_example_idx': 0}}, 'bit_generator_state': {'state': {'state': 274674114334540486603088602300644985544, 'inc': 332724090758049132448979897138935081983}, 'bit_generator': 'PCG64', 'has_uint32': 0, 'uinteger': 0}}}
Loading state dict
tensor([[  769,   278, 12531,  ..., 28721, 19309, 28739],
        [  415, 23347,   622,  ...,  3937,  2426, 28725],
        [28745,  4345, 28723,  ...,   338, 28725,   583],
        ...,
        [ 1670, 28709,  5809,  ..., 28734, 28760,   393],
        [  340,  1277,   624,  ...,   325, 28790,  1329],
        [  523,  1144,  3409,  ...,   359,   359, 17422]])
24.60s elapsed
```

Not sure if this PR complies with the `datasets` code style. Looking for your help @lhoestq, also very willing to further improve the code if any suggestions are given.


WebDataset with different prefixes are unsupported

### Describe the bug

Consider a WebDataset with multiple images for each item where the number of images may vary: [example](https://huggingface.co/datasets/bigdata-pw/fashion-150k)

Due to this [code](https://github.com/huggingface/datasets/blob/87f4c2088854ff33e817e724e75179e9975c1b02/src/datasets/packaged_modules/webdataset/webdataset.py#L76-L80) an error is given.
```
The TAR archives of the dataset should be in WebDataset format, but the files in the archive don't share the same prefix or the same types.
```

The purpose of this check is unclear because PyArrow supports different keys.

Removing the check allows the dataset to be loaded and there's no issue when iterating through the dataset.

```
>>> from datasets import load_dataset
>>> path = "shards/*.tar"
>>> dataset = load_dataset("webdataset", data_files={"train": path}, split="train", streaming=True)
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 152/152 [00:00<00:00, 56458.93it/s]
>>> dataset
IterableDataset({
    features: ['__key__', '__url__', '1.jpg', '2.jpg', '3.jpg', '4.jpg', 'json'],
    n_shards: 152
})
```

### Steps to reproduce the bug

```python
from datasets import load_dataset
load_dataset("bigdata-pw/fashion-150k")
```

### Expected behavior

Dataset loads without error

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.14.0-467.el9.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.19
- `huggingface_hub` version: 0.23.4
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


Add batching to `IterableDataset`

I've taken a try at implementing a batched `IterableDataset` as requested in issue #6279. This PR adds a new `BatchedExamplesIterable` class and a `.batch()` method to the `IterableDataset` class.

The main changes are:
1. A new `BatchedExamplesIterable` that groups examples into batches.
2. A `.batch()` method for `IterableDataset` to easily create batched versions.
3. Support for shuffling and sharding to work with PyTorch DataLoader and multiple workers.

I'm not sure if this is exactly what you had in mind and also have not fully tested it atm, so I'd really appreciate your feedback. Does this seem like it's heading in the right direction? I'm happy to make any changes or explore different approaches if needed.

Pinging @lhoestq


Datasets.datafiles resolve_pattern `TypeError: can only concatenate tuple (not "str") to tuple`

### Describe the bug

in data_files.py, line 332,
`fs,  _, _ = get_fs_token_paths(pattern, storage_options=storage_options)`
If we run the code on AWS, as fs.protocol will be a tuple like: `('file', 'local')`
So,  `isinstance(fs.protocol, str) == False` and
`protocol_prefix = fs.protocol + "://" if fs.protocol != "file" else ""` will raise
`TypeError: can only concatenate tuple (not "str") to tuple`.

### Steps to reproduce the bug

Steps to reproduce:
1. Run on a cloud server like AWS,
2. `import datasets.data_files as datafile`
3. datafile.resolve_pattern('path/to/dataset', '.')
4. `TypeError: can only concatenate tuple (not "str") to tuple`

### Expected behavior

Should return path of the dataset, with fs.protocol at the beginning

### Environment info

- `datasets` version: 2.14.0
- Platform: Linux-3.10.0-1160.119.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.19
- Huggingface_hub version: 0.23.5
- PyArrow version: 16.1.0
- Pandas version: 1.1.5


Adding `Music` feature for symbolic music modality (MIDI, abc)

⚠ (WIP) ⚠

### What this PR does

This PR adds a `Music` feature for the symbolic music modality, in particular [MIDI](https://en.wikipedia.org/wiki/Musical_Instrument_Digital_Interface) and [abc](https://en.wikipedia.org/wiki/ABC_notation) files.

### Motivations

These two file formats are widely used in the [Music Information Retrieval (MIR)](https://en.wikipedia.org/wiki/Music_information_retrieval) for tasks such as music generation, music transcription, music synthesis or music transcription. Having a dedicated feature in the datasets library would allow to both encourage researchers to share datasets of this modality as well as making them more easily usable for end users, benefitting from the perks of the library.

These file formats are supported by [symusic](https://github.com/Yikai-Liao/symusic), a lightweight Python library with C bindings (using nanobind) allowing to efficiently read, write and manipulate them. The library is actively developed, and can in the future also implement other file formats such as [musicXML](https://en.wikipedia.org/wiki/MusicXML). As such, this PR relies on it.

The music data can then easily be tokenized with appropriate tokenizers such as [MidiTok](https://github.com/Natooz/MidiTok) or converted to pianorolls matrices by symusic.

**Jul 16th 2024:**

* the tests for the `Music` feature are currently failing due to non-supported access to the LazyBatch in `test_dataset_with_music_feature_map` and `test_dataset_with_music_feature_map_resample_music` (see TODOs). I am a beginner with pyArrow, I'll take any advice to make this work;
* additional tests including the `Music` feature with parquet and WebDataset should be implemented. As of right now, I am waiting for your feedback before taking further steps;
* a `MusicFolder` should also be implemented to comply with the usages of the `Image` and `Audio` features, waiting for your feedback too.

CCing @lhoestq and @albertvillanova


How to set_epoch with interleave_datasets?

Let's say I have dataset A which has 100k examples, and dataset B which has 100m examples.

I want to train on an interleaved dataset of A+B, with stopping_strategy='all_exhausted' so dataset B doesn't repeat any examples. But every time A is exhausted I want it to be reshuffled (eg. calling set_epoch)

Of course I want to interleave as IterableDatasets / streaming mode so B doesn't have to get tokenized completely at the start.

How could I achieve this? I was thinking something like, if I wrap dataset A in some new IterableDataset with from_generator() and manually call set_epoch before interleaving it? But I'm not sure how to keep the number of shards in that dataset...

Something like

```
dataset_a = load_dataset(...)
dataset_b = load_dataset(...)

def epoch_shuffled_dataset(ds):
  # How to make this maintain the number of shards in ds??
  for epoch in itertools.count():
    ds.set_epoch(epoch)
    yield from iter(ds)

shuffled_dataset_a = IterableDataset.from_generator(epoch_shuffled_dataset, gen_kwargs={'ds': dataset_a})
interleaved = interleave_datasets([shuffled_dataset_a, dataset_b], probs, stopping_strategy='all_exhausted')
```


add checkpoint and resume title in docs

(minor) just to make it more prominent in the docs page for the soon-to-be-released new torchdata


Save nparray as list

### Describe the bug

When I use the `map` function to convert images into features, datasets saves nparray as a list. Some people use the `set_format` function to convert the column back, but doesn't this lose precision?

### Steps to reproduce the bug

the map function
```python
def convert_image_to_features(inst, processor, image_dir):
    image_file = inst["image_url"]
    file = image_file.split("/")[-1]
    image_path = os.path.join(image_dir, file)
    image = Image.open(image_path)
    image = image.convert("RGBA")

    inst["pixel_values"] = processor(images=image, return_tensors="np")["pixel_values"]
    return inst
```

main function
```python
map_fun = partial(
      convert_image_to_features, processor=processor, image_dir=image_dir
  )
ds = ds.map(map_fun, batched=False, num_proc=20)
print(type(ds[0]["pixel_values"])
```

### Expected behavior

(type < list>)

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-4.19.91-009.ali4000.alios7.x86_64-x86_64-with-glibc2.35
- Python version: 3.11.5
- `huggingface_hub` version: 0.23.4
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0


ImportError: numpy.core.multiarray when using `filter`

### Describe the bug

I can't apply the filter method on my dataset.

### Steps to reproduce the bug

The following snippet generates a bug:

```python
from datasets import load_dataset

ami = load_dataset('kamilakesbi/ami', 'ihm')

ami['train'].filter(
    lambda example: example["file_name"] == 'EN2001a'
)
```

I get the following error:

`ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).`

### Expected behavior

It should work properly!

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


Save Dataset as Sharded Parquet

New feature or request

### Feature request

`to_parquet` currently saves the dataset as one massive, monolithic parquet file, rather than as several small parquet files. It should shard large datasets automatically.

### Motivation

This default behavior makes me very sad because a program I ran for 6 hours saved its results using `to_parquet`, putting the entire billion+ row dataset into a 171 GB *single shard parquet file* which pyarrow, apache spark, etc. all cannot work with without completely exhausting the memory of my system. I was previously able to work with larger-than-memory parquet files, but not this one. I *assume* the reason why this is happening is because it is a single shard. Making sharding the default behavior puts datasets in parity with other frameworks, such as spark, which automatically shard when a large dataset is saved as parquet.

### Your contribution

I could change the logic here https://github.com/huggingface/datasets/blob/bf6f41e94d9b2f1c620cf937a2e85e5754a8b960/src/datasets/io/parquet.py#L109-L158
to use `pyarrow.dataset.write_dataset`, which seems to support sharding, or periodically open new files. We would only shard if the user passed in a path rather than file handle.


Support librosa and numpy 2.0 for Python 3.10 by installing soxr 0.4.0b1 pre-release:
- https://github.com/dofuuz/python-soxr/releases/tag/v0.4.0b1
- https://github.com/dofuuz/python-soxr/issues/28


Fix tensorflow min version depending on Python version.

Related to:
- #6991


Mark tests that require librosa

Mark tests that require `librosa`.

Note that `librosa` is an optional dependency (installed with `audio` option) and we should be able to test environments without that library installed. This is the case if we want to test Numpy 2.0, which is currently incompatible with `librosa` due to its dependency on `soxr`:
- https://github.com/dofuuz/python-soxr/issues/28


Add decorator as explicit test dependency.

We use `decorator` library in our CI test since PR:
- #4845

However we did not add it as an explicit test requirement, and we depended on it indirectly through other libraries' dependencies.

I discovered this while testing Numpy 2.0 and removing incompatible libraries.


Improved the tutorial by adding a link for loading datasets

Improved the tutorial by letting readers know about loading datasets with common files and including a link. I left the local files section alone because the methods were already listed with code snippets.


`sort` after `filter` unreasonably  slow

### Describe the bug

as the tittle says ...

### Steps to reproduce the bug

`sort` seems to be normal.

```python
from datasets import Dataset
import random

nums = [{"k":random.choice(range(0,1000))} for _ in range(100000)]

ds = Dataset.from_list(nums)

print("start sort")

ds = ds.sort("k")

print("finish sort")
```

but `sort` after `filter` is extremely slow.

```python
from datasets import Dataset
import random

nums = [{"k":random.choice(range(0,1000))} for _ in range(100000)]

ds = Dataset.from_list(nums)

ds = ds.filter(lambda x:x > 100, input_columns="k")

print("start sort")

ds = ds.sort("k")

print("finish sort")
```

### Expected behavior

Is this a bug, or is it a misuse of the `sort` function?

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2023.10.0


load `streaming=True` dataset with downloaded cache

### Describe the bug

We build a dataset which contains several hdf5 files and write a script using `h5py` to generate the dataset. The hdf5 files are large and the processed dataset cache takes more disk space. So we hope to try streaming iterable dataset. Unfortunately, `h5py` can't convert a remote URL into a hdf5 file descriptor. So we use `fsspec` as an interface like below:

```python
def _generate_examples(self, filepath, split):
        for file in filepath:
            with fsspec.open(file, "rb") as fs:
                with h5py.File(fs, "r") as fp:
                    # for event_id in sorted(list(fp.keys())):
                    event_ids = list(fp.keys())
                    ......
```

### Steps to reproduce the bug

The `fsspec` works, but it takes 10+ min to print the first 10 examples, which is even longer than the downloading time. I'm not sure if it just caches the whole hdf5 file and generates the examples.

### Expected behavior

So does the following make sense so far?

1. download the files
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True)
```

2. load the iterable dataset faster (using the raw file cache at path `.cache/huggingface/datasets/downloads`)
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True, streaming=true)
```

I made some tests, but the code above can't get the expected result. I'm not sure if this is supported. I also find the issue #6327 . It seemed similar to mine, but I couldn't find a solution.

### Environment info

- `datasets` = 2.18.0
- `h5py` = 3.10.0
- `fsspec` = 2023.10.0


Fix export to JSON when dataset larger than batch size

Fix export to JSON (`lines=False`) when dataset larger than batch size.

Fix #7037.


A bug of Dataset.to_json() function

bug

Something isn't working

### Describe the bug

When using the Dataset.to_json() function, an unexpected error occurs if the parameter is set to lines=False. The stored data should be in the form of a list, but it actually turns into multiple lists, which causes an error when reading the data again.
The reason is that to_json() writes to the file in several segments based on the batch size. This is not a problem when lines=True, but it is incorrect when lines=False, because writing in several times will produce multiple lists(when len(dataset) > batch_size).

### Steps to reproduce the bug

try this code:
```python
from datasets import load_dataset
import json

train_dataset = load_dataset("Anthropic/hh-rlhf", data_dir="harmless-base")["train"]
output_path = "./harmless-base_hftojs.json"
print(len(train_dataset))
train_dataset.to_json(output_path, lines=False, force_ascii=False, indent=2)

with open(output_path, encoding="utf-8") as f:
    data = json.loads(f.read())

```
it raise error: json.decoder.JSONDecodeError: Extra data: line 4003 column 1 (char 1373709)

Extra square brackets have appeared here:
<img width="265" alt="image" src="https://github.com/huggingface/datasets/assets/26499566/81492332-386d-42e8-88d1-b6d4ae3682cc">

### Expected behavior

The code runs normally.

### Environment info

datasets=2.20.0


Fix doc generation when NamedSplit is used as parameter default value

Fix doc generation when `NamedSplit` is used as parameter default value.

Fix #7035.


Docs are not generated when a parameter defaults to a NamedSplit value

maintenance

Maintenance tasks

While generating the docs, we get an error when some parameter defaults to a `NamedSplit` value, like:
```python
def call_function(split=Split.TRAIN):
    ...
```
The error is: ValueError: Equality not supported between split train and <class 'inspect._empty'>

See: https://github.com/huggingface/datasets/actions/runs/9869660902/job/27254359863?pr=7015
```
Building the MDX files:  97%|█████████▋| 58/60 [00:00<00:00, 91.94it/s]
Traceback (most recent call last):
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 197, in build_mdx_files
    content, new_anchors, source_files, errors = resolve_autodoc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 123, in resolve_autodoc
    doc = autodoc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 499, in autodoc
    method_doc, check = document_object(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 395, in document_object
    signature = format_signature(obj)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 126, in format_signature
    if param.default != inspect._empty:
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/datasets/splits.py", line 136, in __ne__
    return not self.__eq__(other)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/datasets/splits.py", line 379, in __eq__
    raise ValueError(f"Equality not supported between split {self} and {other}")
ValueError: Equality not supported between split train and <class 'inspect._empty'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/datasets/datasets/.venv/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/commands/build.py", line 102, in build_command
    build_doc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 367, in build_doc
    anchors_mapping, source_files_mapping = build_mdx_files(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 230, in build_mdx_files
    raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Equality not supported between split train and <class 'inspect._empty'>
```


chore: fix typos in docs


`from_generator` does not allow to specify the split name

### Describe the bug

I'm building train, dev, and test using `from_generator`; however, in all three cases, the logger prints `Generating train split:`
It's not possible to change the split name since it seems to be hardcoded: https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/generator/generator.py

### Steps to reproduce the bug

```
In [1]: from datasets import Dataset

In [2]: def gen():
   ...:     yield {"pokemon": "bulbasaur", "type": "grass"}
   ...:

In [3]: ds = Dataset.from_generator(gen)
Generating train split: 1 examples [00:00, 133.89 examples/s]
```

### Expected behavior

It should be possible to specify any split name

### Environment info

- `datasets` version: 2.19.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- `huggingface_hub` version: 0.23.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0


Register `.zstd` extension for zstd-compressed files

For example, https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0 dataset files have `.zstd` extension which is currently ignored (only `.zst` is registered).


CI quality is broken: use ruff check instead

CI quality is broken: https://github.com/huggingface/datasets/actions/runs/9838873879/job/27159697027
```
error: `ruff <path>` has been removed. Use `ruff check <path>` instead.
```


Add option to disable progress bar when reading a dataset ("Loading dataset from disk")

enhancement

New feature or request

### Feature request

Add an option in load_from_disk to disable the progress bar even if the number of files is larger than 16.

### Motivation

I am reading a lot of datasets that it creates lots of logs.

<img width="1432" alt="image" src="https://github.com/huggingface/datasets/assets/57996478/8d4bbf03-6b89-44b6-937c-932f01b4eb2a">

### Your contribution

Seems like an easy fix to make.  I can create a PR if necessary.


load_dataset on AWS lambda throws OSError(30, 'Read-only file system') error

### Describe the bug

I'm using AWS lambda to run a python application. I run the `load_dataset` function with cache_dir="/tmp" and is still throws the OSError(30, 'Read-only file system') error. Is even updated all the HF envs to point to /tmp dir but the issue still persists. I can confirm that the I can write to /tmp directory.

### Steps to reproduce the bug

```python
d = load_dataset(
      path=hugging_face_link,
      split=split,
      token=token,
      cache_dir="/tmp/hugging_face_cache",
 )
```

### Expected behavior

Everything written to the file system as part of the load_datasets function should be in the /tmp directory.

### Environment info

datasets version: 2.16.1
Platform: Linux-5.10.216-225.855.amzn2.x86_64-x86_64-with-glibc2.26
Python version: 3.11.9
huggingface_hub version: 0.19.4
PyArrow version: 16.1.0
Pandas version: 2.2.2
fsspec version: 2023.10.0


Fix ci

...after last pr errors


Missing line from previous pr


Fix check_library_imports

move it to after the `trust_remote_code` check

Note that it only affects local datasets that already exist on disk, not datasets loaded from HF directly


feat: support non streamable arrow file binary format

Support Arrow files (`.arrow`) that are in non streamable binary file formats.


Streaming dataset not returning data

### Describe the bug

I'm deciding to post here because I'm still not sure what the issue is, or if I am using IterableDatasets wrongly.
I'm following the guide on here https://huggingface.co/learn/cookbook/en/fine_tuning_code_llm_on_single_gpu pretty much to a tee and have verified that it works when I'm fine-tuning on the provided dataset.

However, I'm doing some data preprocessing steps (filtering out entries), when I try to swap out the dataset for mine, it fails to train. However, I eventually fixed this by simply setting `stream=False` in `load_dataset`.

Coud this be some sort of network / firewall issue I'm facing?

### Steps to reproduce the bug

I made a post with greater description about how I reproduced this problem before I found my workaround: https://discuss.huggingface.co/t/problem-with-custom-iterator-of-streaming-dataset-not-returning-anything/94551

Here is the problematic dataset snippet, which works when streaming=False (and with buffer keyword removed from shuffle)

```
commitpackft = load_dataset(
    "chargoddard/commitpack-ft-instruct", split="train", streaming=True
).filter(lambda example: example["language"] == "Python")

def form_template(example):
    """Forms a template for each example following the alpaca format for CommitPack"""
    example["content"] = (
        "### Human: " + example["instruction"] + " " + example["input"] + " ### Assistant: " + example["output"]
    )
    return example

dataset = commitpackft.map(
    form_template,
    remove_columns=["id", "language", "license", "instruction", "input", "output"],
).shuffle(
    seed=42, buffer_size=10000
)  # remove everything since its all inside "content" now
validation_data = dataset.take(4000)
train_data = dataset.skip(4000)
```

The annoying part about this is that it only fails during training and I don't know when it will fail, except that it always fails during evaluation.

### Expected behavior

The expected behavior is that I should be able to get something from the iterator when called instead of getting nothing / stuck in a loop somewhere.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.4.0-121-generic-x86_64-with-glibc2.31
- Python version: 3.11.7
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


Remove dead code for pyarrow < 15.0.0

Remove dead code for pyarrow < 15.0.0.

Code is dead since the merge of:
- #6892

Fix #7022.


There is dead code after we require pyarrow >= 15.0.0

maintenance

Maintenance tasks

There are code lines specific for pyarrow versions < 15.0.0.

However, we require pyarrow >= 15.0.0 since the merge of PR:
- #6892

Those code lines are now dead code and should be removed.


Fix casting list array to fixed size list.

This bug was introduced in [datasets-2.17.0](https://github.com/huggingface/datasets/releases/tag/2.17.0) by PR: https://github.com/huggingface/datasets/pull/6283/files#diff-1cb2b66aa9311d729cfd83013dad56cf5afcda35b39dfd0bfe9c3813a049eab0R1899
- #6283

Fix #7020.


Casting list array to fixed size list raises error

bug

Something isn't working

When trying to cast a list array to fixed size list, an AttributeError is raised:
> AttributeError: 'pyarrow.lib.FixedSizeListType' object has no attribute 'length'

Steps to reproduce the bug:
```python
import pyarrow as pa
from datasets.table import array_cast

arr = pa.array([[0, 1]])
array_cast(arr, pa.list_(pa.int64(), 2))
```

Stack trace:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-6cb90a1d8216> in <module>
      3
      4 arr = pa.array([[0, 1]])
----> 5 array_cast(arr, pa.list_(pa.int64(), 2))

~/huggingface/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1802             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1803         else:
-> 1804             return func(array, *args, **kwargs)
   1805
   1806     return wrapper

~/huggingface/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_primitive_to_str, allow_decimal_to_str)
   1920                 else:
   1921                     array_values = array.values[
-> 1922                         array.offset * pa_type.length : (array.offset + len(array)) * pa_type.length
   1923                     ]
   1924                     return pa.FixedSizeListArray.from_arrays(_c(array_values, pa_type.value_type), pa_type.list_size)

AttributeError: 'pyarrow.lib.FixedSizeListType' object has no attribute 'length'
```


Support pyarrow large_list

Allow Polars round trip by supporting pyarrow large list.

Fix #6834, fix #6984.

Supersede and close #4800, close #6835, close #6986.


`load_dataset` fails to load dataset saved by `save_to_disk`

### Describe the bug

This code fails to load the dataset it just saved:

```python
from datasets import load_dataset
from transformers import AutoTokenizer

MODEL = "google-bert/bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

dataset = load_dataset("yelp_review_full")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets.save_to_disk("dataset")

tokenized_datasets = load_dataset("dataset/")  # raises
```

It raises `ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}`.

I believe this bug is caused by the [logic that tries to infer dataset format](https://github.com/huggingface/datasets/blob/9af8dd3de7626183a9a9ec8973cebc672d690400/src/datasets/load.py#L556). It counts the most common file extension. However, a small dataset can fit in a single `.arrow` file and have two JSON metadata files, causing the format to be inferred as JSON:

```shell
$ ls -l dataset/test
-rw-r--r-- 1 sliedes sliedes 191498784 Jul  1 13:55 data-00000-of-00001.arrow
-rw-r--r-- 1 sliedes sliedes      1730 Jul  1 13:55 dataset_info.json
-rw-r--r-- 1 sliedes sliedes       249 Jul  1 13:55 state.json
```

### Steps to reproduce the bug

Execute the code above.

### Expected behavior

The dataset is loaded successfully.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-6.9.3-arch1-1-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


Support fsspec 2024.6.1

Support fsspec 2024.6.1.


`drop_duplicates` method

duplicate

This issue or pull request already exists

enhancement

New feature or request

### Feature request

`drop_duplicates` method for huggingface datasets (similiar in simplicity to the `pandas` one)

### Motivation

Ease of use

### Your contribution

I don't think i am good enough to help


add split argument to Generator

## Actual

When creating a multi-split dataset using generators like

```python
datasets.DatasetDict({
  "val": datasets.Dataset.from_generator(
      generator=generator_val,
      features=features
  ),
  "test": datasets.Dataset.from_generator(
      generator=generator_test,
      features=features,
  )
})
```

It displays (for both test and val)

```
Generating train split
```

## Expected

I would like to be able to improve this behavior by doing

```python
datasets.DatasetDict({
  "val": datasets.Dataset.from_generator(
      generator=generator_val,
      features=features,
      split="val"
  ),
  "test": datasets.Dataset.from_generator(
      generator=generator_test,
      features=features,
      split="test"
  )
})
```

It would display

```
Generating val split
```

and

```
Generating test split
```

## Proposal

Current PR is adding an explicit `split` argument and replace the implicit "train" split in the following classes/function :
* Generator
* from_generator
* AbstractDatasetInputStream
* GeneratorDatasetInputStream

Please share your feedbacks


Skip faiss tests on Windows to avoid running CI for 360 minutes.

Fix #7013.

Revert once the underlying issue is fixed.


CI is broken for faiss tests on Windows: node down: Not properly terminated

maintenance

Maintenance tasks

Faiss tests on Windows make the CI run indefinitely until maximum execution time (360 minutes) is reached.

See: https://github.com/huggingface/datasets/actions/runs/9712659783
```
test (integration, windows-latest, deps-minimum)
The job running on runner GitHub Actions 60 has exceeded the maximum execution time of 360 minutes.
test (integration, windows-latest, deps-latest)
The job running on runner GitHub Actions 238 has exceeded the maximum execution time of 360 minutes.
```
```
____________________________ tests/test_search.py _____________________________
[gw1] win32 -- Python 3.8.10 C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe
worker 'gw1' crashed while running 'tests/test_search.py::IndexableDatasetTest::test_add_faiss_index'
____________________________ tests/test_search.py _____________________________
[gw2] win32 -- Python 3.8.10 C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe
worker 'gw2' crashed while running 'tests/test_search.py::IndexableDatasetTest::test_add_faiss_index'
```
```
tests/test_search.py::IndexableDatasetTest::test_add_faiss_index
[gw0] node down: Not properly terminated
[gw0] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index

replacing crashed worker gw0

tests/test_search.py::IndexableDatasetTest::test_add_faiss_index
[gw1] node down: Not properly terminated
[gw1] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index

replacing crashed worker gw1

tests/test_search.py::IndexableDatasetTest::test_add_faiss_index
[gw2] node down: Not properly terminated
[gw2] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index

replacing crashed worker gw2
```


Raise an error when a nested object is expected to be a mapping that displays the object


Re-enable raising error from huggingface-hub FutureWarning in CI

Re-enable raising error from huggingface-hub FutureWarning in tests, once that the fix in transformers
- https://github.com/huggingface/transformers/pull/31007

was just released yesterday in transformers-4.42.0: https://github.com/huggingface/transformers/releases/tag/v4.42.0

Fix #7010.


Maintenance tasks

Re-enable raising error from huggingface-hub FutureWarning in CI, which was disabled by PR:
- #6876

Note that this can only be done once transformers releases the fix:
- https://github.com/huggingface/transformers/pull/31007


Support ruff 0.5.0 in CI

Support ruff 0.5.0 in CI and revert:
- #7007

Fix #7008.


Support ruff 0.5.0 in CI

Maintenance tasks

Support ruff 0.5.0 in CI.

Also revert:
- #7007


Fix CI by temporarily pinning ruff < 0.5.0

As a hotfix for CI, temporarily pin ruff upper version < 0.5.0.

Fix #7006.

Revert once root cause is fixed.


CI is broken after ruff-0.5.0: E721

Maintenance tasks

After ruff-0.5.0 release (https://github.com/astral-sh/ruff/releases/tag/0.5.0), our CI is broken due to E721 rule.

See: https://github.com/huggingface/datasets/actions/runs/9707641618/job/26793170961?pr=6983
> src/datasets/features/features.py:844:12: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks


EmptyDatasetError: The directory at /metadata.jsonl doesn't contain any data files

### Describe the bug

while trying to load custom dataset from jsonl file, I get the error: "metadata.jsonl doesn't contain any data files"

### Steps to reproduce the bug

This is my [metadata_v2.jsonl](https://github.com/user-attachments/files/16016011/metadata_v2.json) file.  I have this file in the folder with all images mentioned in that json(l) file.

Through below mentioned command I am trying to load_dataset so that I can upload it as mentioned here on the [official website](https://huggingface.co/docs/datasets/en/image_dataset#upload-dataset-to-the-hub).

````
from datasets import load_dataset

dataset = load_dataset("imagefolder", data_dir="path/to/jsonl/metadata.jsonl")
````

error:

````
EmptyDatasetError                         Traceback (most recent call last)
Cell In[18], line 3
      1 from datasets import load_dataset
----> 3 dataset = load_dataset("imagefolder",
      4                        data_dir="path/to/jsonl/file/metadata.jsonl")
      5 dataset[0]["objects"]

File ~/anaconda3/envs/lvis/lib/python3.11/site-packages/datasets/load.py:2594, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   2589 verification_mode = VerificationMode(
   2590     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   2591 )
   2593 # Create a dataset builder
-> 2594 builder_instance = load_dataset_builder(
   2595     path=path,
   2596     name=name,
   2597     data_dir=data_dir,
   2598     data_files=data_files,
   2599     cache_dir=cache_dir,
   2600     features=features,
   2601     download_config=download_config,
   2602     download_mode=download_mode,
   2603     revision=revision,
   2604     token=token,
   2605     storage_options=storage_options,
   2606     trust_remote_code=trust_remote_code,
   2607     _require_default_config_name=name is None,
   2608     **config_kwargs,
   2609 )
   2611 # Return iterable dataset in case of streaming
   2612 if streaming:

File ~/anaconda3/envs/lvis/lib/python3.11/site-packages/datasets/load.py:2266, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, use_auth_token, storage_options, trust_remote_code, _require_default_config_name, **config_kwargs)
   2264     download_config = download_config.copy() if download_config else DownloadConfig()
   2265     download_config.storage_options.update(storage_options)
-> 2266 dataset_module = dataset_module_factory(
   2267     path,
   2268     revision=revision,
   2269     download_config=download_config,
   2270     download_mode=download_mode,
   2271     data_dir=data_dir,
   2272     data_files=data_files,
   2273     cache_dir=cache_dir,
   2274     trust_remote_code=trust_remote_code,
   2275     _require_default_config_name=_require_default_config_name,
   2276     _require_custom_configs=bool(config_kwargs),
   2277 )
   2278 # Get dataset builder class from the processing script
   2279 builder_kwargs = dataset_module.builder_kwargs

File ~/anaconda3/envs/lvis/lib/python3.11/site-packages/datasets/load.py:1805, in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, cache_dir, trust_remote_code, _require_default_config_name, _require_custom_configs, **download_kwargs)
   1782 # We have several ways to get a dataset builder:
   1783 #
   1784 # - if path is the name of a packaged dataset module
   (...)
   1796
   1797 # Try packaged
   1798 if path in _PACKAGED_DATASETS_MODULES:
   1799     return PackagedDatasetModuleFactory(
   1800         path,
   1801         data_dir=data_dir,
   1802         data_files=data_files,
   1803         download_config=download_config,
   1804         download_mode=download_mode,
-> 1805     ).get_module()
   1806 # Try locally
   1807 elif path.endswith(filename):

File ~/anaconda3/envs/lvis/lib/python3.11/site-packages/datasets/load.py:1140, in PackagedDatasetModuleFactory.get_module(self)
   1135 def get_module(self) -> DatasetModule:
   1136     base_path = Path(self.data_dir or "").expanduser().resolve().as_posix()
   1137     patterns = (
   1138         sanitize_patterns(self.data_files)
   1139         if self.data_files is not None
-> 1140         else get_data_patterns(base_path, download_config=self.download_config)
   1141     )
   1142     data_files = DataFilesDict.from_patterns(
   1143         patterns,
   1144         download_config=self.download_config,
   1145         base_path=base_path,
   1146     )
   1147     supports_metadata = self.name in _MODULE_SUPPORTS_METADATA

File ~/anaconda3/envs/lvis/lib/python3.11/site-packages/datasets/data_files.py:503, in get_data_patterns(base_path, download_config)
    501     return _get_data_files_patterns(resolver)
    502 except FileNotFoundError:
--> 503     raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None

EmptyDatasetError: The directory at path/to/jsonl/file/metadata.jsonl doesn't contain any data files`
```

### Expected behavior

It should be able load the whole file in a format of "dataset" inside the dataset variable. But it gives error "The directory at "path/to/jsonl/metadata.jsonl" doesn't contain any data files."

### Environment info

I am using conda environment.


Fix WebDatasets KeyError for user-defined Features when a field is missing in an example

Fixes: https://github.com/huggingface/datasets/issues/6900

Not sure if this needs any addition stuff before merging


minor fix for bfloat16


Fix dump of bfloat16 torch tensor

close https://github.com/huggingface/datasets/issues/7000