WinError 32 The process cannot access the file during load_dataset

### Describe the bug

When I try to load the opus_book from hugging face (following the [guide on the website](https://huggingface.co/docs/transformers/main/en/tasks/translation))

```python
from datasets import load_dataset, Dataset

dataset = load_dataset("Helsinki-NLP/opus_books", "en-fr", features=["id", "translation"])
```

I get an error:
`PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:/Users/Me/.cache/huggingface/datasets/Helsinki-NLP___parquet/ca-de-a39f1ef185b9b73b/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec.incomplete\\parquet-train-00000-00000-of-NNNNN.arrow'
`

Full stacktrace

```python

AttributeError                            Traceback (most recent call last)
File c:\Users\Me\.conda\envs\ia\lib\site-packages\datasets\builder.py:1858, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   [1857](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1857) _time = time.time()
-> [1858](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1858) for _, table in generator:
   [1859](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1859)     if max_shard_size is not None and writer._num_bytes > max_shard_size:

File c:\Users\Me\.conda\envs\ia\lib\site-packages\datasets\packaged_modules\parquet\parquet.py:59, in Parquet._generate_tables(self, files)
     [58](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/packaged_modules/parquet/parquet.py:58) def _generate_tables(self, files):
---> [59](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/packaged_modules/parquet/parquet.py:59)     schema = self.config.features.arrow_schema if self.config.features is not None else None
     [60](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/packaged_modules/parquet/parquet.py:60)     if self.config.features is not None and self.config.columns is not None:

AttributeError: 'list' object has no attribute 'arrow_schema'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
File c:\Users\Me\.conda\envs\ia\lib\site-packages\datasets\builder.py:1882, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   [1881](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1881) num_shards = shard_id + 1
-> [1882](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1882) num_examples, num_bytes = writer.finalize()
   [1883](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1883) writer.close()

File c:\Users\Me\.conda\envs\ia\lib\site-packages\datasets\arrow_writer.py:584, in ArrowWriter.finalize(self, close_stream)
    [583](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/arrow_writer.py:583) # If schema is known, infer features even if no examples were written
--> [584](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/arrow_writer.py:584) if self.pa_writer is None and self.schema:
...
--> [627](file:///C:/Users/Me/.conda/envs/ia/lib/shutil.py:627)         os.unlink(fullname)
    [628](file:///C:/Users/Me/.conda/envs/ia/lib/shutil.py:628)     except OSError:
    [629](file:///C:/Users/Me/.conda/envs/ia/lib/shutil.py:629)         onerror(os.unlink, fullname, sys.exc_info())

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:/Users/Me/.cache/huggingface/datasets/Helsinki-NLP___parquet/ca-de-a39f1ef185b9b73b/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec.incomplete\\parquet-train-00000-00000-of-NNNNN.arrow'
```

### Steps to reproduce the bug

Steps to reproduce:

Just execute these lines

```python
from datasets import load_dataset, Dataset

dataset = load_dataset("Helsinki-NLP/opus_books", "en-fr", features=["id", "translation"])
```

### Expected behavior

I expect the dataset to be loaded without any errors.

### Environment info

| Package| Version|
|--------|--------|
| transformers| 4.37.2|
| python| 3.9.19|
| pytorch| 2.3.0|
| datasets|2.12.0 |
| arrow  |  1.2.3|

I am using Conda on Windows 11.


```push_to_hub()``` - Prevent Automatic Generation of Splits

### Describe the bug

I currently have a dataset which has not been splited. When pushing the dataset to my hugging face dataset repository, it is split into a testing and training set. How can I prevent the split from happening?

### Steps to reproduce the bug

1. Have a unsplit dataset

```python
Dataset({ features: ['input', 'output', 'Attack', '__index_level_0__'], num_rows: 944685 })
```

2. Push it to huggingface

```python
dataset.push_to_hub(dataset_name)
```

3. On the hugging face dataset repo, the dataset then appears to be splited:

![image](https://github.com/huggingface/datasets/assets/29337128/b4fbc141-42b0-4f49-98df-dd479648fe09)

4. Indeed, when loading the dataset from this repo, the dataset is split in two testing and training set.

```python
from datasets import load_dataset, Dataset

dataset = load_dataset("Jetlime/NF-CSE-CIC-IDS2018-v2", streaming=True)
dataset
```
output:

```
IterableDatasetDict({
    train: IterableDataset({
        features: ['input', 'output', 'Attack', '__index_level_0__'],
        n_shards: 2
    })
    test: IterableDataset({
        features: ['input', 'output', 'Attack', '__index_level_0__'],
        n_shards: 1
    })
```

### Expected behavior

The dataset shall not be splited, as not requested.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.2.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


Validate config name and data_files in packaged modules

Validate the config attributes `name` and `data_files` in packaged modules by making the derived classes call their parent `__post_init__` method.

Note that their parent `BuilderConfig` validates its attributes `name` and `data_files` in its `__post_init__` method: https://github.com/huggingface/datasets/blob/60d21efbc01e15d0b596ac1072750cbecd91548a/src/datasets/builder.py#L128-L137

This PR makes the derived config classes call their parent `__post_init__` method to validate their `name` and `data_files` attributes.


Preserve JSON column order and support list of strings field

Preserve column order when loading from a JSON file with a list of dict (or with a field containing a list of dicts).

Additionally, support JSON file with a list of strings field.

Fix #6913.


Column order is nondeterministic when loading from JSON

bug

Something isn't working

As reported by @meg-huggingface, the order of the JSON object keys is not preserved while loading a dataset from a JSON file with a list of objects.

For example, when loading a JSON files with a list of objects, each with the following ordered keys:
- [ID, Language, Topic],

the resulting dataset may have columns:
- [ID, Topic, Language], or
- [Topic, Language, ID], or
- [Topic, ID, Language],...

This issue is caused by the use of a Python set (which does not preserve the order):
https://github.com/huggingface/datasets/blob/60d21efbc01e15d0b596ac1072750cbecd91548a/src/datasets/packaged_modules/json/json.py#L168
introduced in
- #5772


Add MedImg for streaming

dataset request

Requesting to add a new dataset

### Feature request

Host the MedImg dataset (similar to Imagenet but for biomedical images).

### Motivation

There is a clear need for biomedical image foundation models and large scale biomedical datasets that are easily streamable. This would be an excellent tool for the biomedical community.

### Your contribution

MedImg can be found [here](https://www.cuilab.cn/medimg/#).


Remove dead code for non-dict data_files from packaged modules.

Since the merge of this PR:
- #2986

the builders' variable self.config.data_files is always a dict, which makes the condition on (str, list, tuple) dead code.


Fix wrong type hints in data_files

Fix wrong type hints in data_files introduced in:
- #6493


Update requests >=2.32.1 to fix vulnerability.


Fail to load "stas/c4-en-10k" dataset since 2.16 version

### Describe the bug

When update datasets library to version 2.16+ ( I test it on 2.16, 2.19.0 and 2.19.1), using the following code to load stas/c4-en-10k dataset

```python
from datasets import load_dataset, Dataset
dataset = load_dataset('stas/c4-en-10k')
```

and then it raise UnicodeDecodeError like

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 2523, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 2195, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 1846, in dataset_module_factory
    raise e1 from None
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 1798, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
  File "/home/*/conda3/envs/watermark/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```

I found that fs.open loads a gzip file and parses it like plain text using utf-8 encoder.

```python
fs = HfFileSystem('https://huggingface.co')
fs.open("datasets/stas/c4-en-10k/c4-en-10k.py", "rb")
data = fs.read()    # data is gzip bytes begin with b'\x1f\x8b\x08\x00\x00\tn\x88\x00...'
data2 = unzip_gzip_bytes(data)    #  data2 is what we want: '# coding=utf-8\n# Copyright 2020 The HuggingFace Datasets...'
```

### Steps to reproduce the bug

1. Install datasets between version 2.16 and 2.19
2. Use `datasets.load_dataset` method to load `stas/c4-en-10k` dataset.

### Expected behavior

Load dataset normally.

### Environment info

Platform = Linux-5.4.0-159-generic-x86_64-with-glibc2.35
Python = 3.10.14
Datasets = 2.19


Support the deserialization of json lines files comprised of lists

enhancement

New feature or request

### Feature request

I manage a somewhat large and popular Hugging Face dataset known as the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus). I recently updated my corpus to be stored in a json lines file where each line is an array and each element represents a value at a particular column. Previously, my corpus was stored as a json lines file where each line was a dictionary and the keys were the fields.

Essentially, a line in my json lines file used to look like this:
```json
{"version_id":"","type":"","jurisdiction":"","source":"","citation":"","url":"","when_scraped":"","text":""}
```

And now it looks like this:
```json
["","","","","","","",""]
```

This saves 65 bytes per document and allows me very quickly serialise and deserialise documents via `msgspec`.

After making this change, I found that `datasets` was incapable of deserialising my Corpus without a custom loading script, even if I ensured that the `dataset_info` field in my dataset card contained the desired names of my features.

I would like to request that functionality be added to support this format which is more memory-efficent and faster than using dictionaries.

### Motivation

The [documentation](https://huggingface.co/docs/datasets/en/dataset_script) for creating dataset loading scripts asserts that:
> In the next major release, the new safety features of 🤗 Datasets will disable running dataset loading scripts by default, and you will have to pass trust_remote_code=True to load datasets that require running a dataset script.

I would rather not require my users to pass `trust_remote_code=True` which means that I will need built-in support for this format.

### Your contribution

I would be happy to submit a PR for this if this is something you would incorporate into `datasets` and if I can be pointed to where the code would need to go.


irc_disentangle - Issue with splitting data

### Describe the bug

I am trying to access your database through python using "datasets.load_dataset("irc_disentangle")" and I am getting this error message:

ValueError: Instruction "train" corresponds to no data!

### Steps to reproduce the bug

import datasets
ds = datasets.load_dataset('irc_disentangle')
ds

### Expected behavior

The data is supposed to load into ds and be accessable as such:
ds['train'][1050], ds['train'][1055]

### Environment info

I tired Python 3.12 and 3.10


Extraction protocol for arrow files is not defined

### Describe the bug

Passing files with `.arrow` extension into data_files argument, at least when `streaming=True` is very slow.

### Steps to reproduce the bug

 Basically it goes through the `_get_extraction_protocol` method located [here](https://github.com/huggingface/datasets/blob/main/src/datasets/utils/file_utils.py#L820)
The method then looks at some base known extensions where `arrow` is not defined so it proceeds to determine the compression with the magic number method which is slow when dealing with a lot of files which are stored in s3 and by looking at this predefined list, I don't see `arrow` in there either so in the end it return None:

```
MAGIC_NUMBER_TO_COMPRESSION_PROTOCOL = {
    bytes.fromhex("504B0304"): "zip",
    bytes.fromhex("504B0506"): "zip",  # empty archive
    bytes.fromhex("504B0708"): "zip",  # spanned archive
    bytes.fromhex("425A68"): "bz2",
    bytes.fromhex("1F8B"): "gzip",
    bytes.fromhex("FD377A585A00"): "xz",
    bytes.fromhex("04224D18"): "lz4",
    bytes.fromhex("28B52FFD"): "zstd",
}
```

### Expected behavior

My expectation is that `arrow` would be in the known lists so it would return None without going through the magic number method.

### Environment info

datasets 2.19.0


Fix decoding multi part extension

e.g. a field named `url.txt` should be a treated as text

I also included a small fix to support .npz correctly


Add the option of saving in parquet instead of arrow

New feature or request

### Feature request

In dataset.save_to_disk('/path/to/save/dataset'),

add the option to save in parquet format

dataset.save_to_disk('/path/to/save/dataset', format="parquet"),

because arrow is not used for Production Big data.... (only parquet)

### Motivation

because arrow is not used for Production Big data.... (only parquet)

### Your contribution

I can do the testing !


Make CLI convert_to_parquet not raise error if no rights to create script branch

Make CLI convert_to_parquet not raise error if no rights to create "script" branch.

Not that before this PR, the error was not critical because it was raised at the end of the script, once all the rest of the steps were already performed.

Fix #6901.

Bug introduced in datasets-2.19.0 by:
- #6809


HTTPError 403 raised by CLI convert_to_parquet when creating script branch on 3rd party repos

bug

Something isn't working

CLI convert_to_parquet cannot create "script" branch on 3rd party repos.

It can only create it on repos where the user executing the script has write access.

Otherwise, a 403 Forbidden HTTPError is raised:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/ORG/DATASET/branch/script

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/convert_to_parquet.py", line 92, in run
    create_branch(dataset_id, branch="script", repo_type="dataset", token=token, exist_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 367, in hf_raise_for_status
    raise HfHubHTTPError(message, response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError:  (Request ID: Root=1-6645ee0d-4db1ed8a1fbe04956be15897;139a6e23-df7d-4f62-b5ba-adb6d8e6e696)

403 Forbidden: Forbidden: cannot write to script.
Cannot access content at: https://huggingface.co/api/datasets/ORG/DATASET/branch/script.
If you are trying to create or update content,make sure you have a token with the `write` role.
```


[WebDataset] KeyError with user-defined `Features` when a field is missing in an example

reported at https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-dalle3-high-quality-captions/discussions/1

```
File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 109, in _generate_examples
    example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
```


List of dictionary features get standardized

### Describe the bug

Hi, i’m trying to create a HF dataset from a list using Dataset.from_list.

Each sample in the list is a dict with the same keys (which will be my features). The values for each feature are a list of dictionaries, and each such dictionary has a different set of keys. However, the datasets library standardizes all dictionaries under a feature and adds all possible keys (with None value) from all the dictionaries under that feature.

How can I keep the same set of keys as in the original list for each dictionary under a feature?

### Steps to reproduce the bug

```
from datasets import Dataset

# Define a function to generate a sample with "tools" feature
def generate_sample():
    # Generate random sample data
    sample_data = {
        "text": "Sample text",
        "feature_1": []
    }

    # Add feature_1 with random keys for this sample
    feature_1 = [{"key1": "value1"}, {"key2": "value2"}]  # Example feature_1 with random keys
    sample_data["feature_1"].extend(feature_1)

    return sample_data

# Generate multiple samples
num_samples = 10
samples = [generate_sample() for _ in range(num_samples)]

# Create a Hugging Face Dataset
dataset = Dataset.from_list(samples)
dataset[0]
```

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1', 'key2': None}, {'key1': None, 'key2': 'value2'}]}```

### Expected behavior

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1'}, {'key2': 'value2'}]}```

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.15.0-1040-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Fix YAML error in README files appearing on GitHub.

See error message:
![Screenshot from 2024-05-14 06-58-02](https://github.com/huggingface/datasets/assets/8515462/7984cc4e-96ee-4e83-99a4-4c0c5791fa05)

Fix #6897.


datasets template guide :: issue in documentation YAML

### Describe the bug

There is a YAML error at the top of the page, and I don't think it's supposed to be there

### Steps to reproduce the bug

1. Browse to [this tutorial document](https://github.com/huggingface/datasets/blob/main/templates/README_guide.md)
2. Observe a big red error at the top
3. The rest of the document remains functional

### Expected behavior

I think the YAML block should be displayed or ignored.

### Environment info

N/A


Regression bug: `NonMatchingSplitsSizesError` for (possibly) overwritten dataset

### Describe the bug

While trying to load the dataset `https://huggingface.co/datasets/pysentimiento/spanish-tweets-small`, I get this error:

```python
---------------------------------------------------------------------------
NonMatchingSplitsSizesError               Traceback (most recent call last)
[<ipython-input-1-d6a3c721d3b8>](https://localhost:8080/#) in <cell line: 3>()
      1 from datasets import load_dataset
      2
----> 3 ds = load_dataset("pysentimiento/spanish-tweets-small")

3 frames
[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   2150
   2151     # Download and prepare data
-> 2152     builder_instance.download_and_prepare(
   2153         download_config=download_config,
   2154         download_mode=download_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    946                         if num_proc is not None:
    947                             prepare_split_kwargs["num_proc"] = num_proc
--> 948                         self._download_and_prepare(
    949                             dl_manager=dl_manager,
    950                             verification_mode=verification_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
   1059
   1060         if verification_mode == VerificationMode.BASIC_CHECKS or verification_mode == VerificationMode.ALL_CHECKS:
-> 1061             verify_splits(self.info.splits, split_dict)
   1062
   1063         # Update the info object with the splits.

[/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py](https://localhost:8080/#) in verify_splits(expected_splits, recorded_splits)
     98     ]
     99     if len(bad_splits) > 0:
--> 100         raise NonMatchingSplitsSizesError(str(bad_splits))
    101     logger.info("All the splits matched successfully.")
    102

NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=82649695458, num_examples=597433111, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=3358310095, num_examples=24898932, shard_lengths=[3626991, 3716991, 4036990, 3506990, 3676990, 3716990, 2616990], dataset_name='spanish-tweets-small')}]
```

I think I had this dataset updated, might be related to #6271

It is working fine as late in `2.10.0` , but not in `2.13.0` onwards.

### Steps to reproduce the bug

```python

from datasets import load_dataset

ds = load_dataset("pysentimiento/spanish-tweets-small")
```

You can run it in [this notebook](https://colab.research.google.com/drive/1FdhqLiVimHIlkn7B54DbhizeQ4U3vGVl#scrollTo=YgA50cBSibUg)

### Expected behavior

Load the dataset without any error

### Environment info

- `datasets` version: 2.13.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- PyArrow version: 14.0.2
- Pandas version: 2.0.3


Document that to_json defaults to JSON Lines

Document that `Dataset.to_json` defaults to JSON Lines, by adding explanation in the corresponding docstring.

Fix #6894.


Better document defaults of to_json

documentation

Improvements or additions to documentation

Better document defaults of `to_json`: the default format is [JSON-Lines](https://jsonlines.org/).

Related to:
- #6891


Close gzipped files properly

close https://github.com/huggingface/datasets/issues/6877


Add support for categorical/dictionary types

Arrow has a very useful dictionary/categorical type (https://arrow.apache.org/docs/python/generated/pyarrow.dictionary.html). This data type has significant speed, memory and disk benefits over pa.string() when there are only a few unique text strings in a column.

Unfortunately, huggingface datasets currently does not support this type. So huggingface datasets cannot natively read many parquet files that use this datatype .This PR adds support for Huggingface Datasets to read categorical/dictionary data.

Note: This PR functions by simply converting those dictionary/categorical types to strings. This means that huggingface datasets cannot take advantage of the compute benefits of categoricals, but it significantly simplifies logic. At this time, I do not think it makes sense to optimize categorical support within huggingface datasets and that we should only try to optimize later, if necessary.

Closes #5706


Unable to load JSON saved using `to_json`

### Describe the bug

Datasets stored in the JSON format cannot be loaded using `json.load()`

### Steps to reproduce the bug

```
import json
from datasets import load_dataset

dataset = load_dataset("squad")
train_dataset, test_dataset = dataset["train"], dataset["validation"]
test_dataset.to_json("full_dataset.json")

# This works
loaded_test = load_dataset("json", data_files="full_dataset.json")

# This fails
loaded_test = json.load(open("full_dataset.json", "r"))
```

### Expected behavior

The JSON should be correctly formatted when writing so that it can be loaded using `json.load()`.

### Environment info

Colab: https://colab.research.google.com/drive/1st1iStFUVgu9ZPvnzSzL4vDeYWDwYpUm?usp=sharing


add `with_transform` and/or `set_transform` to IterableDataset

enhancement

New feature or request

### Feature request

when working with a really large dataset it would save us a lot of time (and compute resources) to use either with_transform or the set_transform from the Dataset class instead of waiting for the entire dataset to map

### Motivation

don't want to wait for a really long dataset to map, this would give IterableDataset an extra advantage over the Dataset class.
reducing time and resources

### Your contribution

I am a little busy with my job search lately, but would post about this feature in my social media.
Apologies again (dad going to kick me out soon), if I ever have some free time I will contribute to making this a reality, but that's going to be hard
      / (┬┬﹏┬┬)\


fix bug #6877

fix bug #6877 due to maybe f becomes invaild after yield process
the results are below:

Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:01<00:00, 420.41it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26148.48it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 409731.44it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 289720.84it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26663.42it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 434056.21it/s]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 13222.33files/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:04<00:00, 180.67files/s]
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [01:35<00:00,  8.70files/s]
Generating train split: 1571592 examples [00:08, 176736.09 examples/s]
Generating test split: 85533 examples [00:01, 48224.56 examples/s]
Generating validation split: 86246 examples [00:01, 50164.16 examples/s]

Fix https://github.com/huggingface/datasets/issues/6877.

CC: @natolambert


Support WebDataset containing file basenames with dots.

Fix #6880.


FAISS load to None

### Describe the bug

I've use FAISS with Datasets and save to FAISS.

Then load to save FAISS then no error, then ds to None

```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Steps to reproduce the bug

# 1.
```python
ds_with_embeddings = ds.map(lambda example: {'embeddings': model(transforms(example['image']).unsqueeze(0)).squeeze()}, batch_size=64)

ds_with_embeddings.add_faiss_index(column='embeddings')

ds_with_embeddings.save_faiss_index('embeddings', 'index.faiss')
```

# 2.
```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Expected behavior

Add column in Datasets.

### Environment info

Google Colab, SageMaker Notebook


load_dataset with data_dir and cache_dir set fail with not supported

### Describe the bug

with python 3.11 I execute:
```py
from transformers import Wav2Vec2Processor, Data2VecAudioModel
import torch
from torch import nn
from datasets import load_dataset, concatenate_datasets

# load demo audio and set processor
dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
```
This fails in the last line with
```log
Found cached dataset librispeech_asr (file:///Users/as/Documents/Project/git/audio2vec/cache/librispeech_asr/clean-data_dir=data/2.1.0/cff5df6e7955c80a67f80e27e7e655de71c689e2d2364bece785b972acb37fe7)
Traceback (most recent call last):
  File "/Users/as/Documents/Project/git/audio2vec/src/music2vec-v1.py", line 7, in <module>
    dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/load.py", line 1810, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/builder.py", line 1113, in as_dataset
    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
```

### Steps to reproduce the bug

I setup an venv with requirements.txt
```txt
transformers==4.40.2
torch==2.2.2
datasets==2.16.0
fsspec==2023.9.2
```

pip freeze is:
```
aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.16.0
dill==0.3.7
filelock==3.14.0
frozenlist==1.4.1
fsspec==2023.9.2
huggingface-hub==0.23.0
idna==3.7
Jinja2==3.1.4
MarkupSafe==2.1.5
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.15
networkx==3.3
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.0.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
regex==2024.4.28
requests==2.31.0
safetensors==0.4.3
six==1.16.0
sympy==1.12
tokenizers==0.19.1
torch==2.2.2
tqdm==4.66.4
transformers==4.40.2
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4

```

I execute this on a M1 Mac.

### Expected behavior

I don't understand the error message. Why is "local" caching not supported. Would it possible to give some additional hint with the error message how to solve this issue?

### Environment info

source ....
python -u example.py


Support jax 0.4.27 in CI tests

Support jax 0.4.27 in CI tests by using jax Array `devices` method instead of `device` (which no longer exists).

Fix #6884.


CI is broken after jax-0.4.27 release: AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'

bug

Something isn't working

After jax-0.4.27 release (https://github.com/google/jax/releases/tag/jax-v0.4.27), our CI is broken with the error:
```Python traceback
AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'. Did you mean: 'devices'?
```

See: https://github.com/huggingface/datasets/actions/runs/8997488610/job/24715736153
```Python traceback
___________________ FormatterTest.test_jax_formatter_device ____________________
[gw1] linux -- Python 3.10.14 /opt/hostedtoolcache/Python/3.10.14/x64/bin/python

self = <tests.test_formatting.FormatterTest testMethod=test_jax_formatter_device>

    @require_jax
    def test_jax_formatter_device(self):
        import jax

        from datasets.formatting import JaxFormatter

        pa_table = self._create_dummy_table()
        device = jax.devices()[0]
        formatter = JaxFormatter(device=str(device))
        row = formatter.format_row(pa_table)
>       assert row["a"].device() == device
E       AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'. Did you mean: 'devices'?

tests/test_formatting.py:630: AttributeError
```


Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset.

The `PIL.Image.ExifTags` that we use in our code was implemented in Pillow-9.4.0: https://github.com/python-pillow/Pillow/commit/24a5405a9f7ea22f28f9c98b3e407292ea5ee1d3

The bug #6881 was introduced in datasets-2.19.0 by this PR:
- #6739

Fix #6881.


Connection Error When Using By-pass Proxies

### Describe the bug

I'm currently using Clash for Windows as my proxy tunnel, after exporting HTTP_PROXY and HTTPS_PROXY to the port that clash provides🤔, it runs into a connection error saying "Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.19.1/metrics/seqeval/seqeval.py (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.19.1/metrics/seqeval/seqeval.py (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f969d391870>: Failed to establish a new connection: [Errno 111] Connection refused'))")))"
I have already read the documentation provided on the hugginface, but I think I didn't see the detailed instruction on how to set up proxies for this library.

### Steps to reproduce the bug

1. Turn on any proxy software like Clash / ShadosocksR etc.
2. export system varibles to the port provided by your proxy software in wsl (It's ok for other applications to use proxy expect dataset-library)
3. load any dataset from hugginface online

### Expected behavior

---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
Cell In[33], [line 3](vscode-notebook-cell:?execution_count=33&line=3)
      [1](vscode-notebook-cell:?execution_count=33&line=1) from datasets import load_metric
----> [3](vscode-notebook-cell:?execution_count=33&line=3) metric = load_metric("seqeval")

File ~/.local/lib/python3.10/site-packages/datasets/utils/deprecation_utils.py:46, in deprecated.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
     [44](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/utils/deprecation_utils.py:44)     warnings.warn(warning_msg, category=FutureWarning, stacklevel=2)
     [45](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/utils/deprecation_utils.py:45)     _emitted_deprecation_warnings.add(func_hash)
---> [46](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/utils/deprecation_utils.py:46) return deprecated_function(*args, **kwargs)

File ~/.local/lib/python3.10/site-packages/datasets/load.py:2104, in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, trust_remote_code, **metric_init_kwargs)
   [2101](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2101) warnings.filterwarnings("ignore", message=".*https://huggingface.co/docs/evaluate$", category=FutureWarning)
   [2103](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2103) download_mode = DownloadMode(download_mode or DownloadMode.REUSE_DATASET_IF_EXISTS)
-> [2104](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2104) metric_module = metric_module_factory(
   [2105](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2105)     path,
   [2106](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2106)     revision=revision,
   [2107](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2107)     download_config=download_config,
   [2108](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2108)     download_mode=download_mode,
   [2109](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2109)     trust_remote_code=trust_remote_code,
   [2110](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2110) ).module_path
   [2111](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2111) metric_cls = import_main_class(metric_module, dataset=False)
   [2112](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2112) metric = metric_cls(
   [2113](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2113)     config_name=config_name,
   [2114](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/load.py:2114)     process_id=process_id,
...
--> [633](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py:633)     raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
    [634](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py:634) elif response is not None:
    [635](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/home/noodle/Transformers-Tutorials/LayoutLMv3/~/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py:635)     raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})")

ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.19.1/metrics/seqeval/seqeval.py (SSLError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.19.1/metrics/seqeval/seqeval.py (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")))

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.0
- PyArrow version: 16.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.2.0


Something isn't working

When trying to load an image dataset in an old Python environment (with Pillow-8.4.0), an error is raised:
```Python traceback
AttributeError: module 'PIL.Image' has no attribute 'ExifTags'
```
The error traceback:
```Python traceback
~/huggingface/datasets/src/datasets/iterable_dataset.py in __iter__(self)
   1391                 # `IterableDataset` automatically fills missing columns with None.
   1392                 # This is done with `_apply_feature_types_on_example`.
-> 1393                 example = _apply_feature_types_on_example(
   1394                     example, self.features, token_per_repo_id=self._token_per_repo_id
   1395                 )

~/huggingface/datasets/src/datasets/iterable_dataset.py in _apply_feature_types_on_example(example, features, token_per_repo_id)
   1080     encoded_example = features.encode_example(example)
   1081     # Decode example for Audio feature, e.g.
-> 1082     decoded_example = features.decode_example(encoded_example, token_per_repo_id=token_per_repo_id)
   1083     return decoded_example
   1084

~/huggingface/datasets/src/datasets/features/features.py in decode_example(self, example, token_per_repo_id)
   1974
-> 1975         return {
   1976             column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1977             if self._column_requires_decoding[column_name]

~/huggingface/datasets/src/datasets/features/features.py in <dictcomp>(.0)
   1974
   1975         return {
-> 1976             column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1977             if self._column_requires_decoding[column_name]
   1978             else value

~/huggingface/datasets/src/datasets/features/features.py in decode_nested_example(schema, obj, token_per_repo_id)
   1339         # we pass the token to read and decode files from private repositories in streaming mode
   1340         if obj is not None and schema.decode:
-> 1341             return schema.decode_example(obj, token_per_repo_id=token_per_repo_id)
   1342     return obj
   1343

~/huggingface/datasets/src/datasets/features/image.py in decode_example(self, value, token_per_repo_id)
    187             image = PIL.Image.open(BytesIO(bytes_))
    188         image.load()  # to avoid "Too many open files" errors
--> 189         if image.getexif().get(PIL.Image.ExifTags.Base.Orientation) is not None:
    190             image = PIL.ImageOps.exif_transpose(image)
    191         if self.mode and self.mode != image.mode:

~/huggingface/datasets/venv/lib/python3.9/site-packages/PIL/Image.py in __getattr__(name)
     75                 )
     76                 return categories[name]
---> 77         raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
     78
     79

AttributeError: module 'PIL.Image' has no attribute 'ExifTags'
```

### Environment info

Since datasets 2.19.0


Webdataset: KeyError: 'png' on some datasets when streaming

reported at https://huggingface.co/datasets/tbone5563/tar_images/discussions/1

```python
>>> from datasets import load_dataset
>>> ds = load_dataset("tbone5563/tar_images")
Downloading data: 100%
 1.41G/1.41G [00:48<00:00, 17.2MB/s]
Downloading data: 100%
 619M/619M [00:11<00:00, 57.4MB/s]
Generating train split:
 970/0 [00:02<00:00, 534.94 examples/s]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id)
   1747                 _time = time.time()
-> 1748                 for key, record in generator:
   1749                     if max_shard_size is not None and writer._num_bytes > max_shard_size:

7 frames
[/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/webdataset/webdataset.py](https://localhost:8080/#) in _generate_examples(self, tar_paths, tar_iterators)
    108                 for field_name in image_field_names + audio_field_names:
--> 109                     example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
    110                 yield f"{tar_idx}_{example_idx}", example

KeyError: 'png'

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
[<ipython-input-2-8e0fbb7badc9>](https://localhost:8080/#) in <cell line: 3>()
      1 from datasets import load_dataset
      2
----> 3 ds = load_dataset("tbone5563/tar_images")

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   2607
   2608     # Download and prepare data
-> 2609     builder_instance.download_and_prepare(
   2610         download_config=download_config,
   2611         download_mode=download_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
   1025                         if num_proc is not None:
   1026                             prepare_split_kwargs["num_proc"] = num_proc
-> 1027                         self._download_and_prepare(
   1028                             dl_manager=dl_manager,
   1029                             verification_mode=verification_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verification_mode, **prepare_splits_kwargs)
   1787
   1788     def _download_and_prepare(self, dl_manager, verification_mode, **prepare_splits_kwargs):
-> 1789         super()._download_and_prepare(
   1790             dl_manager,
   1791             verification_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
   1120             try:
   1121                 # Prepare split will record examples associated to the split
-> 1122                 self._prepare_split(split_generator, **prepare_split_kwargs)
   1123             except OSError as e:
   1124                 raise OSError(

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split(self, split_generator, check_duplicate_keys, file_format, num_proc, max_shard_size)
   1625             job_id = 0
   1626             with pbar:
-> 1627                 for job_id, done, content in self._prepare_split_single(
   1628                     gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
   1629                 ):

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id)
   1782             if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1783                 e = e.__context__
-> 1784             raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1785
   1786         yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```


Batched mapping does not raise an error if values for an existing column are empty

### Describe the bug

Using `Dataset.map(fn, batched=True)` allows resizing the dataset by returning a dict of lists, all of which must be the same size. If they are not the same size, an error like `pyarrow.lib.ArrowInvalid: Column 1 named x expected length 1 but got length 0` is raised.

This is not the case if the function returns an empty list for an existing column in the dataset. In that case, the dataset is silently resized to 0 rows.

### Steps to reproduce the bug

MWE:

```
import datasets
data = datasets.Dataset.from_dict({"test": [1]})

def mapping_fn(examples):
    return {"test": [], "y": [1]}

data = data.map(mapping_fn, batched=True)
print(len(data))
```

Note  that when returning `"x": []`, the error is raised correctly, also when returning `"test": [1,2]`.

### Expected behavior

Expected an exception: `pyarrow.lib.ArrowInvalid: Column 1 named test expected length 1 but got length 0` or `pyarrow.lib.ArrowInvalid: Column 2 named y expected length 0 but got length 1`.

Any exception would be acceptable.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31
- Python version: 3.11.8
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0


Create function to convert to parquet

Analogously with `delete_from_hub`, this PR:
- creates the Python function `convert_to_parquet`
- makes the corresponding CLI command use that function.

This way, the functionality can be used both from a terminal and from a Python console.

This PR also implements a test for convert_to_parquet function.


OSError: [Errno 24] Too many open files

Something isn't working

### Describe the bug

I am trying to load the 'default' subset of the following dataset which contains lots of files (828 per split): [https://huggingface.co/datasets/mteb/biblenlp-corpus-mmteb](https://huggingface.co/datasets/mteb/biblenlp-corpus-mmteb)

When trying to load it using the `load_dataset` function I get the following error

```python
>>> from datasets import load_dataset
>>> d = load_dataset('mteb/biblenlp-corpus-mmteb')
Downloading readme: 100%|████████████████████████████████████████████████████████████████████████| 201k/201k [00:00<00:00, 1.07MB/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 1069.15it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 436182.33it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 2228.75it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 646478.73it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 831032.24it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 517645.51it/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████| 828/828 [00:33<00:00, 24.87files/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████| 828/828 [00:30<00:00, 27.48files/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████| 828/828 [00:30<00:00, 26.94files/s]
Generating train split: 1571592 examples [00:03, 461438.97 examples/s]
Generating test split: 11163 examples [00:00, 118190.72 examples/s]
Traceback (most recent call last):
  File ".env/lib/python3.12/site-packages/datasets/builder.py", line 1995, in _prepare_split_single
    for _, table in generator:
  File ".env/lib/python3.12/site-packages/datasets/packaged_modules/json/json.py", line 99, in _generate_tables
    with open(file, "rb") as f:
         ^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/datasets/streaming.py", line 75, in wrapper
    return function(*args, download_config=download_config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/datasets/utils/file_utils.py", line 1224, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/fsspec/core.py", line 135, in open
    return self.__enter__()
           ^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/fsspec/core.py", line 103, in __enter__
    f = self.fs.open(self.path, mode=mode)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/fsspec/spec.py", line 1293, in open
    f = self._open(
        ^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/datasets/filesystems/compression.py", line 81, in _open
    return self.file.open()
           ^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/fsspec/core.py", line 135, in open
    return self.__enter__()
           ^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/fsspec/core.py", line 103, in __enter__
    f = self.fs.open(self.path, mode=mode)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/fsspec/spec.py", line 1293, in open
    f = self._open(
        ^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/fsspec/implementations/local.py", line 197, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".env/lib/python3.12/site-packages/fsspec/implementations/local.py", line 322, in __init__
    self._open()
  File ".env/lib/python3.12/site-packages/fsspec/implementations/local.py", line 327, in _open
    self.f = open(self.path, mode=self.mode)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 24] Too many open files: '.cache/huggingface/datasets/downloads/3a347186abfc0f9c924dde0221d246db758c7232c0101523f04a87c17d696618'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".env/lib/python3.12/site-packages/datasets/builder.py", line 981, in incomplete_dir
    yield tmp_dir
  File ".env/lib/python3.12/site-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File ".env/lib/python3.12/site-packages/datasets/builder.py", line 1122, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File ".env/lib/python3.12/site-packages/datasets/builder.py", line 1882, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File ".env/lib/python3.12/site-packages/datasets/builder.py", line 2038, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".env/lib/python3.12/site-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File ".env/lib/python3.12/site-packages/datasets/builder.py", line 1007, in download_and_prepare
    with incomplete_dir(self._output_dir) as tmp_output_dir:
  File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File ".env/lib/python3.12/site-packages/datasets/builder.py", line 988, in incomplete_dir
    shutil.rmtree(tmp_dir)
  File "/usr/lib/python3.12/shutil.py", line 785, in rmtree
    _rmtree_safe_fd(fd, path, onexc)
  File "/usr/lib/python3.12/shutil.py", line 661, in _rmtree_safe_fd
    onexc(os.scandir, path, err)
  File "/usr/lib/python3.12/shutil.py", line 657, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
         ^^^^^^^^^^^^^^^^^
OSError: [Errno 24] Too many open files: '.cache/huggingface/datasets/mteb___biblenlp-corpus-mmteb/default/0.0.0/3912ed967b0834547f35b2da9470c4976b357c9a.incomplete'

```

I looked for the maximum number of open files on my machine (Ubuntu 24.04) and it seems to be 1024, but even when I try to load a single split (`load_dataset('mteb/biblenlp-corpus-mmteb', split='train')`) I get the same error

### Steps to reproduce the bug

```python
from datasets import load_dataset
d = load_dataset('mteb/biblenlp-corpus-mmteb')
```

### Expected behavior

Load the dataset without error

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-6.8.0-31-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.23.0
- PyArrow version: 16.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


Unpin hfh

Needed to use those in dataset-viewer:

- dev version of hfh https://github.com/huggingface/dataset-viewer/pull/2781: don't span the hub with /paths-info requests
- dev version of datasets at https://github.com/huggingface/datasets/pull/6875: don't write too big logs in the viewer

close https://github.com/huggingface/datasets/issues/6863


Shorten long logs

Some datasets may have unexpectedly long features/types (e.g. if the files are not formatted correctly).

In that case we should still be able to log something readable


Use pandas ujson in JSON loader to improve performance.

Note that `datasets` has `pandas` as required dependency. And `pandas` includes `ujson` in `pd.io.json.ujson_loads`.

Fix #6867.

CC: @natolambert


Set dev version


Release 2.19.1


Fix download for dict of dicts of URLs

Fix download for a dict of dicts of URLs when batched (default), introduced by:
- #6794

This PR also implements regression tests.

Fix #6869, fix #6850.


Update tqdm >= 4.66.3 to fix vulnerability,


Download is broken for dict of dicts: FileNotFoundError

Something isn't working

It seems there is a bug when downloading a dict of dicts of URLs introduced by:
- #6794

## Steps to reproduce the bug:
```python
from datasets import DownloadManager

dl_manager = DownloadManager()
paths = dl_manager.download({"train": {"frr": "hf://datasets/wikimedia/wikipedia/20231101.frr/train-00000-of-00001.parquet"}})
```

Stack trace:
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-7-0e0d76d25b09> in <module>
----> 1 paths = dl_manager.download({"train": {"frr": "hf://datasets/wikimedia/wikipedia/20231101.frr/train-00000-of-00001.parquet"}})

.../huggingface/datasets/src/datasets/download/download_manager.py in download(self, url_or_urls)
    255         start_time = datetime.now()
    256         with stack_multiprocessing_download_progress_bars():
--> 257             downloaded_path_or_paths = map_nested(
    258                 download_func,
    259                 url_or_urls,

.../huggingface/datasets/src/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, parallel_min_length, batched, batch_size, types, disable_tqdm, desc)
    506                 batch_size = max(len(iterable) // num_proc + int(len(iterable) % num_proc > 0), 1)
    507             iterable = list(iter_batched(iterable, batch_size))
--> 508         mapped = [
    509             _single_map_nested((function, obj, batched, batch_size, types, None, True, None))
    510             for obj in hf_tqdm(iterable, disable=disable_tqdm, desc=desc)

.../huggingface/datasets/src/datasets/utils/py_utils.py in <listcomp>(.0)
    507             iterable = list(iter_batched(iterable, batch_size))
    508         mapped = [
--> 509             _single_map_nested((function, obj, batched, batch_size, types, None, True, None))
    510             for obj in hf_tqdm(iterable, disable=disable_tqdm, desc=desc)
    511         ]

.../huggingface/datasets/src/datasets/utils/py_utils.py in _single_map_nested(args)
    375         and all(not isinstance(v, types) for v in data_struct)
    376     ):
--> 377         return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
    378
    379     # Reduce logging to keep things readable in multiprocessing with tqdm

.../huggingface/datasets/src/datasets/utils/py_utils.py in <listcomp>(.0)
    375         and all(not isinstance(v, types) for v in data_struct)
    376     ):
--> 377         return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
    378
    379     # Reduce logging to keep things readable in multiprocessing with tqdm

.../huggingface/datasets/src/datasets/download/download_manager.py in _download_batched(self, url_or_filenames, download_config)
    311             )
    312         else:
--> 313             return [
    314                 self._download_single(url_or_filename, download_config=download_config)
    315                 for url_or_filename in url_or_filenames

.../huggingface/datasets/src/datasets/download/download_manager.py in <listcomp>(.0)
    312         else:
    313             return [
--> 314                 self._download_single(url_or_filename, download_config=download_config)
    315                 for url_or_filename in url_or_filenames
    316             ]

.../huggingface/datasets/src/datasets/download/download_manager.py in _download_single(self, url_or_filename, download_config)
    321             # append the relative path to the base_path
    322             url_or_filename = url_or_path_join(self._base_path, url_or_filename)
--> 323         out = cached_path(url_or_filename, download_config=download_config)
    324         out = tracked_str(out)
    325         out.set_origin(url_or_filename)

.../huggingface/datasets/src/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    220     elif is_local_path(url_or_filename):
    221         # File, but it doesn't exist.
--> 222         raise FileNotFoundError(f"Local file {url_or_filename} doesn't exist")
    223     else:
    224         # Something unknown

FileNotFoundError: Local file .../huggingface/datasets/{'frr': 'hf:/datasets/wikimedia/wikipedia/20231101.frr/train-00000-of-00001.parquet'} doesn't exist
```

Related to:
- #6850


datasets.BuilderConfig does not work.

### Describe the bug

I custom a BuilderConfig and GeneratorBasedBuilder.
Here is the code for BuilderConfig
```
class UIEConfig(datasets.BuilderConfig):

    def __init__(
            self,
            *args,
            data_dir=None,
            instruction_file=None,
            instruction_strategy=None,
            task_config_dir=None,
            num_examples=None,
            max_num_instances_per_task=None,
            max_num_instances_per_eval_task=None,
            over_sampling=None,
            **kwargs
    ):
        super().__init__(*args, **kwargs)
        self.data_dir = data_dir
        self.num_examples = num_examples
        self.over_sampling = over_sampling
        self.instructions = self._parse_instruction(instruction_file)
        self.task_configs = self._parse_task_config(task_config_dir)
        self.instruction_strategy = instruction_strategy
        self.max_num_instances_per_task = max_num_instances_per_task
        self.max_num_instances_per_eval_task = max_num_instances_per_eval_task
```
Besides, here is the code for GeneratorBasedBuilder.
```
class UIEInstructions(datasets.GeneratorBasedBuilder):
    VERSION = datasets.Version("2.0.0")
    BUILDER_CONFIG_CLASS = UIEConfig
    BUILDER_CONFIGS = [
        UIEConfig(name="default", description="Default config for NaturalInstructions")
    ]
    DEFAULT_CONFIG_NAME = "default"
```

Here is the load_dataset
```
raw_datasets = load_dataset(
        os.path.join(CURRENT_DIR, "uie_dataset.py"),
        data_dir=data_args.data_dir,
        task_config_dir=data_args.task_config_dir,
        instruction_file=data_args.instruction_file,
        instruction_strategy=data_args.instruction_strategy,
        cache_dir=data_cache_dir,  # for debug, change dataset size, otherwise open it
        max_num_instances_per_task=data_args.max_num_instances_per_task,
        max_num_instances_per_eval_task=data_args.max_num_instances_per_eval_task,
        num_examples=data_args.num_examples,
        over_sampling=data_args.over_sampling
    )
```
Finally, I met the error.
```
BuilderConfig UIEConfig(name='default', version=0.0.0, data_dir=None, data_files=None, description='Default config for NaturalInstructions') doesn't have a 'task_config_dir' key.
```

I debugged the code, but I find the parameters added by me may not work.

### Steps to reproduce the bug

https://github.com/BeyonderXX/InstructUIE/blob/master/src/uie_dataset.py

### Expected behavior

```
BuilderConfig UIEConfig(name='default', version=0.0.0, data_dir=None, data_files=None, description='Default config for NaturalInstructions') doesn't have a 'task_config_dir' key.
```

### Environment info

torch                     2.3.0+cu118
transformers              4.40.1
python 3.8


Improve performance of JSON loader

enhancement

New feature or request

As reported by @natolambert, loading regular JSON files with `datasets` shows poor performance.

The cause is that we use the `json` Python standard library instead of other faster libraries. See my old comment: https://github.com/huggingface/datasets/pull/2638#pullrequestreview-706983714
> There are benchmarks that compare different JSON packages, with the Standard Library one among the worst performant:
> - https://github.com/ultrajson/ultrajson#benchmarks
> - https://github.com/ijl/orjson#performance

I remember having a discussion about this and it was decided that it was better not to include an additional dependency on a 3rd-party library.

However:
- We already depend on `pandas` and `pandas` depends on `ujson`: so we have an indirect dependency on `ujson`
- Even if the above were not the case, we always could include `ujson` as an optional extra dependency, and check at runtime if it is installed to decide which library to use, either json or ujson


DataFilesNotFoundError for datasets in the open-llm-leaderboard

### Describe the bug

When trying to get config names or load any dataset within the open-llm-leaderboard ecosystem (`open-llm-leaderboard/details_`) I receive the DataFilesNotFoundError. For the last month or so I've been loading datasets from the leaderboard almost everyday; yesterday was the first time I started seeing this.

### Steps to reproduce the bug

This snippet has three cells:
1. Loads the modules
2. Tries to get config names
3. Tries to load the dataset

I've chosen "davidkim205"'s Rhea-72b-v0.5 model because it is one of the best performers on the leaderboard should likely have no dataset issues:

```python
In [1]: from datasets import load_dataset, get_dataset_config_names

In [2]: get_dataset_config_names("open-llm-leaderboard/details_davidkim205__Rhea
   ...: -72b-v0.5")
---------------------------------------------------------------------------
DataFilesNotFoundError                    Traceback (most recent call last)
Cell In[2], line 1
----> 1 get_dataset_config_names("open-llm-leaderboard/details_davidkim205__Rhea-72b-v0.5")

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/inspect.py:347, in get_dataset_config_names(path, revision, download_config, download_mode, dynamic_modules_path, data_files, **download_kwargs)
    291 def get_dataset_config_names(
    292     path: str,
    293     revision: Optional[Union[str, Version]] = None,
   (...)
    298     **download_kwargs,
    299 ):
    300     """Get the list of available config names for a particular dataset.
    301
    302     Args:
   (...)
    345     ```
    346     """
--> 347     dataset_module = dataset_module_factory(
    348         path,
    349         revision=revision,
    350         download_config=download_config,
    351         download_mode=download_mode,
    352         dynamic_modules_path=dynamic_modules_path,
    353         data_files=data_files,
    354         **download_kwargs,
    355     )
    356     builder_cls = get_dataset_builder_class(dataset_module, dataset_name=os.path.basename(path))
    357     return list(builder_cls.builder_configs.keys()) or [
    358         dataset_module.builder_kwargs.get("config_name", builder_cls.DEFAULT_CONFIG_NAME or "default")
    359     ]

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/load.py:1821, in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, cache_dir, trust_remote_code, _require_default_config_name, _require_custom_configs, **download_kwargs)
   1812     return LocalDatasetModuleFactoryWithScript(
   1813         combined_path,
   1814         download_mode=download_mode,
   1815         dynamic_modules_path=dynamic_modules_path,
   1816         trust_remote_code=trust_remote_code,
   1817     ).get_module()
   1818 elif os.path.isdir(path):
   1819     return LocalDatasetModuleFactoryWithoutScript(
   1820         path, data_dir=data_dir, data_files=data_files, download_mode=download_mode
-> 1821     ).get_module()
   1822 # Try remotely
   1823 elif is_relative_path(path) and path.count("/") <= 1:

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/load.py:1039, in LocalDatasetModuleFactoryWithoutScript.get_module(self)
   1033     patterns = get_data_patterns(base_path)
   1034 data_files = DataFilesDict.from_patterns(
   1035     patterns,
   1036     base_path=base_path,
   1037     allowed_extensions=ALL_ALLOWED_EXTENSIONS,
   1038 )
-> 1039 module_name, default_builder_kwargs = infer_module_for_data_files(
   1040     data_files=data_files,
   1041     path=self.path,
   1042 )
   1043 data_files = data_files.filter_extensions(_MODULE_TO_EXTENSIONS[module_name])
   1044 # Collect metadata files if the module supports them

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/load.py:597, in infer_module_for_data_files(data_files, path, download_config)
    595     raise ValueError(f"Couldn't infer the same data file format for all splits. Got {split_modules}")
    596 if not module_name:
--> 597     raise DataFilesNotFoundError("No (supported) data files found" + (f" in {path}" if path else ""))
    598 return module_name, default_builder_kwargs

DataFilesNotFoundError: No (supported) data files found in open-llm-leaderboard/details_davidkim205__Rhea-72b-v0.5

In [3]: data = load_dataset("open-llm-leaderboard/details_davidkim205__Rhea-72b-
   ...: v0.5", "harness_winogrande_5")
---------------------------------------------------------------------------
DataFilesNotFoundError                    Traceback (most recent call last)
Cell In[3], line 1
----> 1 data = load_dataset("open-llm-leaderboard/details_davidkim205__Rhea-72b-v0.5", "harness_winogrande_5")

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/load.py:2587, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   2582 verification_mode = VerificationMode(
   2583     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   2584 )
   2586 # Create a dataset builder
-> 2587 builder_instance = load_dataset_builder(
   2588     path=path,
   2589     name=name,
   2590     data_dir=data_dir,
   2591     data_files=data_files,
   2592     cache_dir=cache_dir,
   2593     features=features,
   2594     download_config=download_config,
   2595     download_mode=download_mode,
   2596     revision=revision,
   2597     token=token,
   2598     storage_options=storage_options,
   2599     trust_remote_code=trust_remote_code,
   2600     _require_default_config_name=name is None,
   2601     **config_kwargs,
   2602 )
   2604 # Return iterable dataset in case of streaming
   2605 if streaming:

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/load.py:2259, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, use_auth_token, storage_options, trust_remote_code, _require_default_config_name, **config_kwargs)
   2257     download_config = download_config.copy() if download_config else DownloadConfig()
   2258     download_config.storage_options.update(storage_options)
-> 2259 dataset_module = dataset_module_factory(
   2260     path,
   2261     revision=revision,
   2262     download_config=download_config,
   2263     download_mode=download_mode,
   2264     data_dir=data_dir,
   2265     data_files=data_files,
   2266     cache_dir=cache_dir,
   2267     trust_remote_code=trust_remote_code,
   2268     _require_default_config_name=_require_default_config_name,
   2269     _require_custom_configs=bool(config_kwargs),
   2270 )
   2271 # Get dataset builder class from the processing script
   2272 builder_kwargs = dataset_module.builder_kwargs

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/load.py:1821, in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, cache_dir, trust_remote_code, _require_default_config_name, _require_custom_configs, **download_kwargs)
   1812     return LocalDatasetModuleFactoryWithScript(
   1813         combined_path,
   1814         download_mode=download_mode,
   1815         dynamic_modules_path=dynamic_modules_path,
   1816         trust_remote_code=trust_remote_code,
   1817     ).get_module()
   1818 elif os.path.isdir(path):
   1819     return LocalDatasetModuleFactoryWithoutScript(
   1820         path, data_dir=data_dir, data_files=data_files, download_mode=download_mode
-> 1821     ).get_module()
   1822 # Try remotely
   1823 elif is_relative_path(path) and path.count("/") <= 1:

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/load.py:1039, in LocalDatasetModuleFactoryWithoutScript.get_module(self)
   1033     patterns = get_data_patterns(base_path)
   1034 data_files = DataFilesDict.from_patterns(
   1035     patterns,
   1036     base_path=base_path,
   1037     allowed_extensions=ALL_ALLOWED_EXTENSIONS,
   1038 )
-> 1039 module_name, default_builder_kwargs = infer_module_for_data_files(
   1040     data_files=data_files,
   1041     path=self.path,
   1042 )
   1043 data_files = data_files.filter_extensions(_MODULE_TO_EXTENSIONS[module_name])
   1044 # Collect metadata files if the module supports them

File ~/open-llm-bda/venv/lib/python3.11/site-packages/datasets/load.py:597, in infer_module_for_data_files(data_files, path, download_config)
    595     raise ValueError(f"Couldn't infer the same data file format for all splits. Got {split_modules}")
    596 if not module_name:
--> 597     raise DataFilesNotFoundError("No (supported) data files found" + (f" in {path}" if path else ""))
    598 return module_name, default_builder_kwargs

DataFilesNotFoundError: No (supported) data files found in open-llm-leaderboard/details_davidkim205__Rhea-72b-v0.5
```

### Expected behavior

No exceptions from `get_dataset_config_names` or `load_dataset`

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-6.5.0-1018-aws-aarch64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.23.0
- PyArrow version: 16.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


Example on Semantic segmentation contains bug

### Describe the bug

https://huggingface.co/docs/datasets/en/semantic_segmentation shows wrong example with torchvision transforms.
Specifically, as one can see in screenshot below, the object boundaries have weird colors.

Original example with `albumentations` is correct

That is because `torch vision.transforms.Resize` interpolates with bilinear everything which is wrong when used for segmentation labels - you just cannot mix them. Overall, `torchvision.transforms` is designed for classification only and cannot be used to images and masks together, unless you write two separate branches of augmentations.

The correct way would be to use `v2` version of transforms and convert the segmentation labels to https://pytorch.org/vision/main/generated/torchvision.tv_tensors.Mask.html#torchvision.tv_tensors.Mask object

### Steps to reproduce the bug

Go to the website.

https://huggingface.co/docs/datasets/en/semantic_segmentation

### Expected behavior

Results, similar to `albumentation`. Or remove the torch vision part altogether. Or use `kornia` instead.

### Environment info

Irrelevant


### Describe the bug

The dataset `rewardsignal/reddit_writing_prompts` is missing in Huggingface Hub.

### Steps to reproduce the bug

```
from datasets import load_dataset

prompt_response_dataset = load_dataset("rewardsignal/reddit_writing_prompts", data_files="prompt_responses_full.csv", split='train[:80%]')
```

### Expected behavior

DatasetNotFoundError: Dataset 'rewardsignal/reddit_writing_prompts' doesn't exist on the Hub or cannot be accessed

### Environment info

Nothing to do with versions


Revert temporary pin huggingface-hub < 0.23.0 introduced by
- #6861

once the following issue is fixed and released:
- huggingface/transformers#30618


Fix load_dataset for data_files with protocols other than HF

Fixes huggingface/datasets/issues/6598

I've added a new test case and a solution. Before applying the solution the test case was failing with the same error described in the linked issue.

MRE:
```
pip install "datasets[s3]"
python -c "from datasets import load_dataset; load_dataset('csv', data_files={'train': 's3://noaa-gsod-pds/2024/A5125600451.csv'})"
```


Fix CI by temporarily pinning huggingface-hub < 0.23.0

As a hotfix for CI, temporarily pin `huggingface-hub` upper version

Fix #6860.

Revert once root cause is fixed, see:
- https://github.com/huggingface/transformers/issues/30618


CI fails after huggingface_hub-0.23.0 release: FutureWarning: "resume_download"

bug

Something isn't working

CI fails after latest huggingface_hub-0.23.0 release: https://github.com/huggingface/huggingface_hub/releases/tag/v0.23.0

```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_bertscore - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_frugalscore - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_perplexity - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer_with_cache - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_arrow_dataset.py::MiscellaneousDatasetTest::test_set_format_encode - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
```


Support folder-based datasets with large metadata.jsonl

I tried creating an `imagefolder` dataset with a 714MB `metadata.jsonl` but got the error below.  This pull request fixes the problem by increasing the block size like the message suggests.
```
>>> from datasets import load_dataset
>>> dataset = load_dataset("imagefolder", data_dir="data-for-upload")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
      ...
  File "/path/to/datasets/packaged_modules/folder_based_builder/folder_based_builder.py", line 245, in _read_metadata
    return paj.read_json(f)
  File "pyarrow/_json.pyx", line 308, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```


Segmentation fault

### Describe the bug

Using various version for datasets, I'm no more longer able to load that dataset without a segmentation fault.
Several others files are also concerned.

### Steps to reproduce the bug

# Create a new venv
python3 -m venv venv_test
source venv_test/bin/activate

# Install the latest version
pip install datasets

# Load that dataset
python3 -q -X faulthandler -c "from datasets import load_dataset;  load_dataset('EuropeanParliament/Eurovoc', '1998-09')"

### Expected behavior

Data must be loaded

### Environment info

datasets==2.19.0
Python 3.11.7
Darwin  22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64 x86_64


Fix line-endings in tests on Windows

EDIT:
~~Fix test_delete_from_hub on Windows by passing explicit encoding.~~
Fix test_delete_from_hub and test_xgetsize_private by uploading the README file content directly (encoding the string), instead of writing a local file and uploading it.

Note that local files created on Windows will have "\r\n" line endings, instead of "\n".
These are no longer transformed to "\n" by the Hub.

Fix #6856.


CI fails on Windows for test_delete_from_hub and test_xgetsize_private due to new-line character

Something isn't working

CI fails on Windows for test_delete_from_hub after the merge of:
- #6820

This is weird because the CI was green in the PR branch before merging to main.

```
FAILED tests/test_hub.py::test_delete_from_hub - AssertionError: assert [CommitOperat...\r\n---\r\n')] == [CommitOperat...in/*\n---\n')]

  At index 1 diff: CommitOperationAdd(path_in_repo='README.md', path_or_fileobj=b'---\r\nconfigs:\r\n- config_name: cats\r\n  data_files:\r\n  - split: train\r\n    path: cats/train/*\r\n---\r\n') != CommitOperationAdd(path_in_repo='README.md', path_or_fileobj=b'---\nconfigs:\n- config_name: cats\n  data_files:\n  - split: train\n    path: cats/train/*\n---\n')

  Full diff:
    [
        CommitOperationDelete(
            path_in_repo='dogs/train/0000.csv',
            is_folder=False,
        ),
        CommitOperationAdd(
            path_in_repo='README.md',
  -         path_or_fileobj=b'---\nconfigs:\n- config_name: cats\n  data_files:\n '
  ?                                                                       --------
  +         path_or_fileobj=b'---\r\nconfigs:\r\n- config_name: cats\r\n  data_f'
  ?                               ++          ++                     ++
  -                         b' - split: train\n    path: cats/train/*\n---\n',
  ?                                                                   ^^^^^^ -
  +                         b'iles:\r\n  - split: train\r\n    path: cats/train/*\r'
  ?                           ++++++++++                ++                        ^
  +                         b'\n---\r\n',
        ),
    ]
```


Fix dataset name for community Hub script-datasets by passing explicit dataset_name to HubDatasetModuleFactoryWithScript.

Fix #6854.

CC: @Wauplin


Wrong example of usage when config name is missing for community script-datasets

Something isn't working

As reported by @Wauplin, when loading a community dataset with script, there is a bug in the example of usage of the error message if the dataset has multiple configs (and no default config) and the user does not pass any config. For example:
```python
>>> ds = load_dataset("google/fleurs")
ValueError: Config name is missing.
Please pick one among the available configs: ['af_za', 'am_et', 'ar_eg', 'as_in', 'ast_es', 'az_az', 'be_by', 'bg_bg', 'bn_in', 'bs_ba', 'ca_es', 'ceb_ph', 'ckb_iq', 'cmn_hans_cn', 'cs_cz', 'cy_gb', 'da_dk', 'de_de', 'el_gr', 'en_us', 'es_419', 'et_ee', 'fa_ir', 'ff_sn', 'fi_fi', 'fil_ph', 'fr_fr', 'ga_ie', 'gl_es', 'gu_in', 'ha_ng', 'he_il', 'hi_in', 'hr_hr', 'hu_hu', 'hy_am', 'id_id', 'ig_ng', 'is_is', 'it_it', 'ja_jp', 'jv_id', 'ka_ge', 'kam_ke', 'kea_cv', 'kk_kz', 'km_kh', 'kn_in', 'ko_kr', 'ky_kg', 'lb_lu', 'lg_ug', 'ln_cd', 'lo_la', 'lt_lt', 'luo_ke', 'lv_lv', 'mi_nz', 'mk_mk', 'ml_in', 'mn_mn', 'mr_in', 'ms_my', 'mt_mt', 'my_mm', 'nb_no', 'ne_np', 'nl_nl', 'nso_za', 'ny_mw', 'oc_fr', 'om_et', 'or_in', 'pa_in', 'pl_pl', 'ps_af', 'pt_br', 'ro_ro', 'ru_ru', 'sd_in', 'sk_sk', 'sl_si', 'sn_zw', 'so_so', 'sr_rs', 'sv_se', 'sw_ke', 'ta_in', 'te_in', 'tg_tj', 'th_th', 'tr_tr', 'uk_ua', 'umb_ao', 'ur_pk', 'uz_uz', 'vi_vn', 'wo_sn', 'xh_za', 'yo_ng', 'yue_hant_hk', 'zu_za', 'all']
Example of usage:
	`load_dataset('fleurs', 'af_za')`
```

Note the example of usage in the error message suggests loading "fleurs" instead of "google/fleurs".


Support soft links for load_datasets imagefolder

enhancement

New feature or request

### Feature request

Load_dataset from a folder of images doesn't seem to support soft links. It would be nice if it did, especially during methods development where image folders are being curated.

### Motivation

Images are coming from a complex variety of sources and we'd like to be able to soft link directly from the originating folders as opposed to copying. Having a copy of the file ensures that there may be issues with image versioning as well as having double the amount of required disk space.

### Your contribution

N/A


Write token isn't working while pushing to datasets

### Describe the bug

<img width="1001" alt="Screenshot 2024-05-01 at 3 37 06 AM" src="https://github.com/huggingface/datasets/assets/130903099/00fcf12c-fcc1-4749-8592-d263d4efcbcc">

As you can see I logged in to my account and the write token is valid.

But I can't upload on my main account and I am getting that error. It was okay on my test account at first try.

(I refreshed the token, tried a new token but still doesn't work)

### Steps to reproduce the bug

1. I loaded a dataset.

2. I logged in using both cli and huggingface_hub

3. I pushed to my down dataset
(It went well without any issues on my test account)

### Expected behavior

It should have gone smoothly and this is not even my first time uploading to huggingface datasets

### Environment info

colab, dataset (tried multiple versions)


load_dataset('emotion')    UnicodeDecodeError

### Describe the bug

**emotions = load_dataset('emotion')**

_UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte_

### Steps to reproduce the bug

load_dataset('emotion')

### Expected behavior

succese

### Environment info

py3.10
transformers  4.41.0.dev0
datasets     2.19.0


Problem loading voxpopuli dataset

### Describe the bug

```
Exception has occurred: FileNotFoundError
Couldn't find file at https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/{'en': 'data/en/asr_train.tsv'}
```

Error in logic for link url creation. The link should be  https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/en/asr_train.tsv

Basically there should be links directly under ```metadata["train"]```, not under ```metadata["train"][self.config.languages[0]]```

same for audio urls

### Steps to reproduce the bug

```
from datasets import load_dataset

dataset = load_dataset("facebook/voxpopuli","en")
```

### Expected behavior

Dataset should be loaded successfully.

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.15.0-1041-aws-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.12.2


fix webdataset filename split

use `os.path.splitext` to parse field_name.

fix filename which has dot. like:

```
a.b.jpeg
a.b.txt
```


Cant Downlaod Common Voice 17.0 hy-AM

### Describe the bug

I want to download Common Voice 17.0 hy-AM but it returns an error.
```

The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_name='hfds_config', config_path=None)
/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
/usr/local/lib/python3.10/dist-packages/datasets/load.py:1429: FutureWarning: The repository for mozilla-foundation/common_voice_17_0 contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/mozilla-foundation/common_voice_17_0
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
Reading metadata...: 6180it [00:00, 133224.37it/s]les/s]
Generating train split: 0 examples [00:00, ? examples/s]
HuggingFace datasets failed due to some reason (stack trace below).
For certain datasets (eg: MCV), it may be necessary to login to the huggingface-cli (via `huggingface-cli login`).
Once logged in, you need to set `use_auth_token=True` when calling this script.

Traceback error for reference :

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1743, in _prepare_split_single
    example = self.info.features.encode_example(record) if self.info.features is not None else record
  File "/usr/local/lib/python3.10/dist-packages/datasets/features/features.py", line 1878, in encode_example
    return encode_nested_example(self, example)
  File "/usr/local/lib/python3.10/dist-packages/datasets/features/features.py", line 1243, in encode_nested_example
    {
  File "/usr/local/lib/python3.10/dist-packages/datasets/features/features.py", line 1243, in <dictcomp>
    {
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 326, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 326, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
KeyError: 'sentence_id'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/nemo/scripts/speech_recognition/convert_hf_dataset_to_nemo.py", line 358, in main
    dataset = load_dataset(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2549, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1605, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1762, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

```
from datasets import load_dataset

cv_17 = load_dataset("mozilla-foundation/common_voice_17_0", "hy-AM")
```

### Expected behavior

It works fine with common_voice_16_1

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-1042-nvidia-x86_64-with-glibc2.35
- Python version: 3.11.6
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.2.0


[Streaming] Only load requested splits without resolving files for the other splits

e.g. [thangvip](https://huggingface.co/thangvip)/[cosmopedia_vi_math](https://huggingface.co/datasets/thangvip/cosmopedia_vi_math) has 300 splits and it takes a very long time to load only one split.

This is due to `load_dataset()` resolving the files of all the splits even if only one is needed.

In `dataset-viewer` the splits are loaded in different jobs so it results in 300 jobs that resolve 300 splits -> 90k calls to `/paths-info`


Unimaginable super slow iteration

### Describe the bug

Assuming there is a dataset with 52000 sentences, each with a length of 500, it takes 20 seconds to extract a sentence from the dataset……？Is there something wrong with my iteration?

### Steps to reproduce the bug

```python
import datasets
import time
import random

num_rows = 52000
num_cols = 500

random_input = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]
random_output = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]

s=time.time()
d={'random_input':random_input,'random_output':random_output}
dataset=datasets.Dataset.from_dict(d)
print('from dict',time.time()-s)
print(dataset)

for i in range(len(dataset)):
    aa=time.time()
    a,b=dataset['random_input'][i],dataset['random_output'][i]
    print(time.time()-aa)

```

corresponding output
```bash
from dict 9.215498685836792
Dataset({
    features: ['random_input', 'random_output'],
    num_rows: 52000
})
19.129778146743774
19.329464197158813
19.27668261528015
19.28557538986206
19.247620582580566
19.624247074127197
19.28673791885376
19.301053047180176
19.290496110916138
19.291821718215942
19.357765197753906

```

### Expected behavior

Under normal circumstances, iteration should be very rapid as it does not involve the main tasks other than getting items

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0