Probabilistic High Memory Usage and Freeze on Python 3.10

### Describe the bug

A probabilistic issue encountered when processing datasets containing PIL.Image columns using the huggingface/datasets library on Python 3.10. The process occasionally experiences a sudden and significant memory spike, reaching 100% utilization, leading to a complete freeze. During this freeze, the process becomes unresponsive, cannot be forcefully terminated, and does not throw any exceptions.

I have attempted to mitigate this issue by setting `datasets.config.IN_MEMORY_MAX_SIZE`, but it had no effect. In fact, based on the document of `load_dataset`, I suspect that setting `IN_MEMORY_MAX_SIZE` might even have a counterproductive effect.

This bug is not consistently reproducible, but its occurrence rate significantly decreases or disappears entirely when upgrading Python to version 3.11 or higher. Therefore, this issue also serves to share a potential solution for others who might encounter similar problems.

### Steps to reproduce the bug

Due to the probabilistic nature of this bug, consistent reproduction cannot be guaranteed for every run. However, in my environment, processing large datasets like timm/imagenet-1k-wds(whether reading, casting, or mapping operations) almost certainly triggers the issue at some point.
The probability of the issue occurring drastically increases when num_proc is set to a value greater than 1 during operations.

When the issue occurs, my system logs repeatedly show the following warnings:
```
WARN: very high memory utilization: 57.74GiB / 57.74GiB (100 %)
WARN: container is unhealthy: triggered memory limits (OOM)
WARN: container is unhealthy: triggered memory limits (OOM)
WARN: container is unhealthy: triggered memory limits (OOM)
```

### Expected behavior

The dataset should be read and processed normally without memory exhaustion or freezing. If an unrecoverable error occurs, an appropriate exception should be raised.

I have found that upgrading Python to version 3.11 or above completely resolves this issue. On Python 3.11, when memory usage approaches 100%, it suddenly drops before slowly increasing again. I suspect this behavior is due to an expected memory management action, possibly involving writing to disk cache, which prevents the complete freeze observed in Python 3.10.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.15.0-71-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.33.4
- PyArrow version: 20.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0


Question about iterable dataset and streaming

In the doc, I found the following example: https://github.com/huggingface/datasets/blob/611f5a592359ebac6f858f515c776aa7d99838b2/docs/source/stream.mdx?plain=1#L65-L78

I am confused,
1. If we have already loaded the dataset, why doing `to_iterable_dataset`?  Does it go through the dataset faster than map-style dataset?
2. `load_dataset(streaming=True)` is useful for huge dataset, but the speed is slow. How to make it comparable to `to_iterable_dataset` without loading the whole dataset into RAM?


metric glue breaks with 4.0.0

### Describe the bug

worked fine with 3.6.0, and with 4.0.0 `eval_metric = metric.compute()` in HF Accelerate breaks.

The code that fails is:
https://huggingface.co/spaces/evaluate-metric/glue/blob/v0.4.0/glue.py#L84

```
def simple_accuracy(preds, labels):
    print(preds, labels)
    print(f"{preds==labels}")
    return float((preds == labels).mean())
```
data:
```
Column([1, 0, 0, 1, 1]) Column([1, 0, 0, 1, 0])
False
```

```
[rank0]:     return float((preds == labels).mean())
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'bool' object has no attribute 'mean'
```

Some behavior has changed in this new major release of `datasets` and requires updating HF accelerate and perhaps the glue metric code, all belong to HF.

### Environment info

datasets=4.0.0


In the latest version of datasets==4.0.0, i cannot print the audio data on the Colab notebook. But it works on the 3.6.0 version.

!pip install -q -U datasets huggingface_hub fsspec

from datasets import load_dataset

downloaded_dataset = load_dataset("ymoslem/MediaSpeech", "tr", split="train")
print(downloaded_dataset["audio"][0])

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[/tmp/ipython-input-4-90623240.py](https://localhost:8080/#) in <cell line: 0>()
----> 1 downloaded_dataset["audio"][0]

10 frames
[/usr/local/lib/python3.11/dist-packages/datasets/features/audio.py](https://localhost:8080/#) in decode_example(self, value, token_per_repo_id)
    170             from ._torchcodec import AudioDecoder
    171         else:
--> 172             raise ImportError("To support decoding audio data, please install 'torchcodec'.")
    173
    174         if not self.decode:

ImportError: To support decoding audio data, please install 'torchcodec'.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python version: 3.11.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0


Toxicity fails with datasets 4.0.0

### Describe the bug

With the latest 4.0.0 release, huggingface toxicity evaluation module fails with error: `ValueError: text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).`

### Steps to reproduce the bug

Repro:
```
>>> toxicity.compute(predictions=["This is a response"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/evaluate/module.py", line 467, in compute
    output = self._compute(**inputs, **compute_kwargs)
  File "/Users/serena.ruan/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-measurement--toxicity/2390290fa0bf6d78480143547c6b08f3d4f8805b249df8c7a8e80d0ce8e3778b/toxicity.py", line 135, in _compute
    scores = toxicity(predictions, self.toxic_classifier, toxic_label)
  File "/Users/serena.ruan/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-measurement--toxicity/2390290fa0bf6d78480143547c6b08f3d4f8805b249df8c7a8e80d0ce8e3778b/toxicity.py", line 103, in toxicity
    for pred_toxic in toxic_classifier(preds):
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/text_classification.py", line 159, in __call__
    result = super().__call__(*inputs, **kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1431, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1437, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/text_classification.py", line 183, in preprocess
    return self.tokenizer(inputs, return_tensors=return_tensors, **tokenizer_kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2867, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2927, in _call_one
    raise ValueError(
ValueError: text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
```

### Expected behavior

This works before 4.0.0 release

### Environment info

- `datasets` version: 4.0.0
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.10.16
- `huggingface_hub` version: 0.33.0
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Many things broken since the new 4.0.0 release

### Describe the bug

The new changes in 4.0.0 are breaking many datasets, including those from lm-evaluation-harness.
I am trying to revert back to older versions, like 3.6.0 to make the eval work but I keep getting:
``` Python
File /venv/main/lib/python3.12/site-packages/datasets/features/features.py:1474, in generate_from_dict(obj)
   1471 class_type = _FEATURE_TYPES.get(_type, None) or globals().get(_type, None)
   1473 if class_type is None:
-> 1474     raise ValueError(f"Feature type '{_type}' not found. Available feature types: {list(_FEATURE_TYPES.keys())}")
   1476 if class_type == LargeList:
   1477     feature = obj.pop("feature")

ValueError: Feature type 'List' not found. Available feature types: ['Value', 'ClassLabel', 'Translation', 'TranslationVariableLanguages', 'LargeList', 'Sequence', 'Array2D', 'Array3D', 'Array4D', 'Array5D', 'Audio', 'Image', 'Video', 'Pdf']

```

### Steps to reproduce the bug

``` Python
import lm_eval
model_eval = lm_eval.models.huggingface.HFLM(pretrained=model, tokenizer=tokenizer)
lm_eval.evaluator.simple_evaluate(model_eval, tasks=["winogrande"], num_fewshot=5, batch_size=1)
```

### Expected behavior

Older `datasets` versions should work just fine as before

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-6.8.0-60-generic-x86_64-with-glibc2.39
- Python version: 3.12.11
- `huggingface_hub` version: 0.33.1
- PyArrow version: 20.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0


common_voice_11_0.py failure in dataset library

### Describe the bug

I tried to download dataset but have got this error:

from datasets import load_dataset
load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[8], line 4
      1 from datasets import load_dataset
----> 4 load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)

File c:\Users\ege_g\AppData\Local\Programs\Python\Python312\Lib\site-packages\datasets\load.py:1392, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, keep_in_memory, save_infos, revision, token, streaming, num_proc, storage_options, **config_kwargs)
   1387 verification_mode = VerificationMode(
   1388     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   1389 )
   1391 # Create a dataset builder
-> 1392 builder_instance = load_dataset_builder(
   1393     path=path,
   1394     name=name,
   1395     data_dir=data_dir,
   1396     data_files=data_files,
   1397     cache_dir=cache_dir,
   1398     features=features,
   1399     download_config=download_config,
   1400     download_mode=download_mode,
   1401     revision=revision,
   1402     token=token,
   1403     storage_options=storage_options,
   1404     **config_kwargs,
   1405 )
   1407 # Return iterable dataset in case of streaming
   1408 if streaming:

File c:\Users\ege_g\AppData\Local\Programs\Python\Python312\Lib\site-packages\datasets\load.py:1132, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, storage_options, **config_kwargs)
   1130 if features is not None:
   1131     features = _fix_for_backward_compatible_features(features)
-> 1132 dataset_module = dataset_module_factory(
   1133     path,
   1134     revision=revision,
   1135     download_config=download_config,
   1136     download_mode=download_mode,
   1137     data_dir=data_dir,
   1138     data_files=data_files,
   1139     cache_dir=cache_dir,
   1140 )
   1141 # Get dataset builder class
   1142 builder_kwargs = dataset_module.builder_kwargs

File c:\Users\ege_g\AppData\Local\Programs\Python\Python312\Lib\site-packages\datasets\load.py:1031, in dataset_module_factory(path, revision, download_config, download_mode, data_dir, data_files, cache_dir, **download_kwargs)
   1026             if isinstance(e1, FileNotFoundError):
   1027                 raise FileNotFoundError(
   1028                     f"Couldn't find any data file at {relative_to_absolute_path(path)}. "
   1029                     f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1030                 ) from None
-> 1031             raise e1 from None
   1032 else:
   1033     raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.")

File c:\Users\ege_g\AppData\Local\Programs\Python\Python312\Lib\site-packages\datasets\load.py:989, in dataset_module_factory(path, revision, download_config, download_mode, data_dir, data_files, cache_dir, **download_kwargs)
    981 try:
    982     api.hf_hub_download(
    983         repo_id=path,
    984         filename=filename,
   (...)
    987         proxies=download_config.proxies,
    988     )
--> 989     raise RuntimeError(f"Dataset scripts are no longer supported, but found {filename}")
    990 except EntryNotFoundError:
    991     # Use the infos from the parquet export except in some cases:
    992     if data_dir or data_files or (revision and revision != "main"):

RuntimeError: Dataset scripts are no longer supported, but found common_voice_11_0.py

### Steps to reproduce the bug

from datasets import load_dataset
load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)

### Expected behavior

its supposed to download this dataset.

### Environment info

Python 3.12 , Windows 11


set dev version


Release: 4.0.0


Fix double sequence

```python
>>> Features({"a": Sequence(Sequence({"c": Value("int64")}))})
{'a': List({'c': List(Value('int64'))})}
```

instead of `{'a': {'c': List(List(Value('int64')))}}`


Mapping function not working if the first example is returned as None

### Describe the bug

https://github.com/huggingface/datasets/blob/8a19de052e3d79f79cea26821454bbcf0e9dcd68/src/datasets/arrow_dataset.py#L3652C29-L3652C37

Here we can see the writer is initialized on `i==0`. However, there can be cases where in the user mapping function, the first example is filtered out (length constraints, etc).

In this case, the writer would be a `None` type and the code will report `NoneType has no write function`.

A simple fix is available, simply change line 3652 from `if i == 0:` to `if writer is None:`

### Steps to reproduce the bug

Prepare a dataset

have this function

```
import datasets

def make_map_fn(split, max_prompt_tokens=3):
    def process_fn(example, idx):

        question = example['question']
        reasoning_steps = example['reasoning_steps']
        label = example['label']

        answer_format = ""
        for i in range(len(reasoning_steps)):
            system_message = "Dummy"

        all_steps_formatted = []

        content = f"""Dummy"""

        prompt = [
            {"role": "system", "content": system_message},
            {"role": "user", "content": content},
        ]
        tokenized = tokenizer.apply_chat_template(prompt, return_tensors="pt", truncation=False)
        if tokenized.shape[1] > max_prompt_tokens:
            return None  # skip overly long examples

        data = {
            "dummy": "dummy"
        }

        return data

    return process_fn

...
# load your dataset
...
train = train.map(function=make_map_fn('train'), with_indices=True)
```

### Expected behavior

The dataset mapping shall behave even when the first example is filtered out.

### Environment info

I am using `datasets==3.6.0` but I have observed this issue in the github repo too: https://github.com/huggingface/datasets/blob/8a19de052e3d79f79cea26821454bbcf0e9dcd68/src/datasets/arrow_dataset.py#L3652C29-L3652C37


Fix audio bytes


How can I add my custom data to huggingface datasets

I want to add my custom dataset in huggingface dataset. Please guide me how to achieve that.


Broken EXIF crash the whole program

### Describe the bug

When parsing this image in the ImageNet1K dataset, the `datasets` crashs whole training process just because unable to parse an invalid EXIF tag.
![Image](https://github.com/user-attachments/assets/3c840203-ac8c-41a0-9cf7-45f64488037d)

### Steps to reproduce the bug

Use the `datasets.Image.decode_example` method to decode the aforementioned image could reproduce the bug.
The decoding function will throw an unhandled exception at the `image.getexif()` method call due to invalid utf-8 stream in EXIF tags.
```
File lib/python3.12/site-packages/datasets/features/image.py:188, in Image.decode_example(self, value, token_per_repo_id)
    186     image = PIL.Image.open(BytesIO(bytes_))
    187 image.load()  # to avoid "Too many open files" errors
--> 188 if image.getexif().get(PIL.Image.ExifTags.Base.Orientation) is not None:
    189     image = PIL.ImageOps.exif_transpose(image)
    190 if self.mode and self.mode != image.mode:

File lib/python3.12/site-packages/PIL/Image.py:1542, in Image.getexif(self)
   1540 xmp_tags = self.info.get("XML:com.adobe.xmp")
   1541 if not xmp_tags and (xmp_tags := self.info.get("xmp")):
-> 1542     xmp_tags = xmp_tags.decode("utf-8")
   1543 if xmp_tags:
   1544     match = re.search(r'tiff:Orientation(="|>)([0-9])', xmp_tags)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 4312: invalid start byte
```

### Expected behavior

The invalid EXIF tag should simply be ignored or issue a warning message, instead of crash the whole program at once.

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-6.5.0-18-generic-x86_64-with-glibc2.35
- Python version: 3.12.11
- `huggingface_hub` version: 0.33.0
- PyArrow version: 20.0.0
- Pandas version: 2.3.0
- `fsspec` version: 2025.3.0


Fix infer list of images

cc @kashif


Backward compat list feature

cc @kashif


Function load_dataset() misinterprets string field content as part of dataset schema when dealing with `.jsonl` files

### Describe the bug

When loading a `.jsonl` file using `load_dataset("json", data_files="data.jsonl", split="train")`, the function misinterprets the content of a string field  as if it were part of the dataset schema.

In my case there is a field `body:` with a string value
```
"### Describe the bug (...) ,action: string, datetime: timestamp[s], author: string, (...) Pandas version: 1.3.4"
```
As a result, I got an exception
```
"TypeError: Couldn't cast array of type timestamp[s] to null".
```
Full stack-trace in the attached file below.

I also attach a minimized dataset (data.json, a single entry) that reproduces the error.

**Observations**(on the minimal example):
- if I remove _all fields before_ `body`, a different error appears,
- if I remove _all fields after_ `body`, yet another error appears,
- if `body` is _the only field_, the error disappears.

So this might be one complex bug or several edge cases interacting. I haven’t dug deeper.

Also changing the file extension to `.json` or `.txt` avoids the problem. This suggests **a possible workaround** for the general case: convert `.jsonl` to `.json`. Though I haven’t verified correctness of that workaround yet.

Anyway my understanding is that `load_dataset` with first argument set to "json" should properly handle `.jsonl` files. Correct me if I'm wrong.

[stack_trace.txt](https://github.com/user-attachments/files/21004153/stack_trace.txt)
[data.json](https://github.com/user-attachments/files/21004164/data.json)

P.S.
I discovered this while going through the HuggingFace tutorial. Specifically [this part](https://huggingface.co/learn/llm-course/chapter5/5?fw=pt).I will try to inform the tutorial team about this issue, as it can be a showstopper for young 🤗 adepts.

### Steps to reproduce the bug

1. Download attached [data.json](https://github.com/user-attachments/files/21004164/data.json) file.
2. Run the following code which should work correctly:
```
from datasets import load_dataset
load_dataset("json", data_files="data.json", split="train")
```
3. Change extension of the `data` file to `.jsonl` and run:
```
from datasets import load_dataset
load_dataset("json", data_files="data.jsonl", split="train")
```

This will trigger an error like the one in the attached [stack_trace.txt](https://github.com/user-attachments/files/21004153/stack_trace.txt).

One can also try removing fields before the `body` field and after it. These actions give different errors.

### Expected behavior

Parsing data in `.jsonl` format should yield the same result as parsing the same data in `.json` format. In any case, the content of a string field should never be interpreted as part of the dataset schema.

### Environment info

datasets version: _3.6.0_
pyarrow version: _20.0.0_
Python version: _3.11.9_
platform version: _macOS-15.5-arm64-arm-64bit_


### Describe the bug

When loading a `.jsonl` file using `load_dataset("json", data_files="data.jsonl", split="train")`, the function misinterprets the content of a string field  as if it were part of the dataset schema.

In my case there is a field `body:` with a string value
```
"### Describe the bug (...) ,action: string, datetime: timestamp[s], author: string, (...) Pandas version: 1.3.4"
```
As a result, I got an exception
```
"TypeError: Couldn't cast array of type timestamp[s] to null".
```
Full stack-trace in the attached file below.

I also attach a minimized dataset (data.json, a single entry) that reproduces the error.

**Observations**(on the minimal example):
- if I remove _all fields before_ `body`, a different error appears,
- if I remove _all fields after_ `body`, yet another error appears,
- if `body` is _the only field_, the error disappears.

So this might be one complex bug or several edge cases interacting. I haven’t dug deeper.

Also changing the file extension to `.json` or `.txt` avoids the problem. This suggests **a possible workaround** for the general case: convert `.jsonl` to `.json`. Though I haven’t verified correctness of that workaround yet.

Anyway my understanding is that `load_dataset` with first argument set to "json" should properly handle `.jsonl` files. Correct me if I'm wrong.

[stack_trace.txt](https://github.com/user-attachments/files/21004153/stack_trace.txt)
[data.json](https://github.com/user-attachments/files/21004164/data.json)

P.S.
I discovered this while going through the HuggingFace tutorial. Specifically [this part](https://huggingface.co/learn/llm-course/chapter5/5?fw=pt). I will try to inform the tutorial team about this issue, as it can be a showstopper for young 🤗 adepts.

### Steps to reproduce the bug

1. Download attached [data.json](https://github.com/user-attachments/files/21004164/data.json) file.
2. Run the following code which should work correctly:
```
from datasets import load_dataset
load_dataset("json", data_files="data.json", split="train")
```
3. Change extension of the `data` file to `.jsonl` and run:
```
from datasets import load_dataset
load_dataset("json", data_files="data.jsonl", split="train")
```

This will trigger an error like the one in the attached [stack_trace.txt](https://github.com/user-attachments/files/21004153/stack_trace.txt).

One can also try removing fields before the `body` field and after it. These actions give different errors.

### Expected behavior

Parsing data in `.jsonl` format should yield the same result as parsing the same data in `.json` format. In any case, the content of a string field should never be interpreted as part of the dataset schema.

### Environment info

datasets version: _3.6.0_
pyarrow version: _20.0.0_
Python version: _3.11.9_
platform version: _macOS-15.5-arm64-arm-64bit_


Custom metadata filenames

example: https://huggingface.co/datasets/lhoestq/overlapping-subsets-imagefolder/tree/main

To make multiple subsets for an imagefolder (one metadata file per subset), e.g.

```yaml
configs:
  - config_name: default
    metadata_filenames:
      - metadata.csv
  - config_name: other
    metadata_filenames:
      - metadata2.csv
```


Applying map after transform with multiprocessing will cause OOM

### Describe the bug

I have a 30TB dataset. When I perform add_column and cast_column operations on it and then execute a multiprocessing map, it results in an OOM (Out of Memory) error. However, if I skip the add_column and cast_column steps and directly run the map, there is no OOM. After debugging step by step, I found that the OOM is caused at this point, and I suspect it’s because the add_column and cast_column operations are not cached, which causes the entire dataset to be loaded in each subprocess, leading to the OOM. The critical line of code is: https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/utils/py_utils.py#L607

Note num_process=1 would not cause OOM. I'm confused.

### Steps to reproduce the bug

For reproduce, you can load  dataset and set cache_dir (for caching): amphion/Emilia-Dataset which is a veru large datasets that RAM can not fits.
And apply the map with multiprocessing after a transform operation  (e.g. add_column, cast_column).
As long as num_process>1, it must cause OOM.

### Expected behavior

It should not cause OOM.

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-5.10.134-16.101.al8.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.33.1
- PyArrow version: 20.0.0
- Pandas version: 2.3.0
- `fsspec` version: 2024.6.1


fix del tqdm lock error

fixes https://github.com/huggingface/datasets/issues/7660


### Describe the bug

`AttributeError: type object 'tqdm' has no attribute '_lock'`

It occurs when I'm trying to load datasets in thread pool.

Issue https://github.com/huggingface/datasets/issues/6066 and PR https://github.com/huggingface/datasets/pull/6067 https://github.com/huggingface/datasets/pull/6068 tried to fix this.

### Steps to reproduce the bug

Will have to try several times to reproduce the error because it is concerned with threads.

1. save some datasets for test
```pythonfrom datasets import Dataset, DatasetDict
import os

os.makedirs("test_dataset_shards", exist_ok=True)

for i in range(10):
    data = Dataset.from_dict({"text": [f"example {j}" for j in range(1000000)]})
    data = DatasetDict({'train': data})
    data.save_to_disk(f"test_dataset_shards/shard_{i}")
```

2. load them in a thread pool
```python
from datasets import load_from_disk
from tqdm import tqdm
from concurrent.futures import ThreadPoolExecutor, as_completed
import glob

datas = glob.glob('test_dataset_shards/shard_*')
with ThreadPoolExecutor(max_workers=10) as pool:
    futures = [pool.submit(load_from_disk, it) for it in datas]
datas = []
for future in tqdm(as_completed(futures), total=len(futures)):
    datas.append(future.result())
```

### Expected behavior

no exception raised

### Environment info

datasets==2.19.0
python==3.10


Update the beans dataset link in Preprocess

In the Preprocess tutorial, the to "the beans dataset" is incorrect. Fixed.


Fix: Prevent loss of info.features and column_names in IterableDatasetDict.map when features is None

This PR fixes a bug where calling `IterableDatasetDict.map()` or `IterableDataset.map()` with the default `features=None` argument would overwrite the existing `info.features` attribute with `None`. This, in turn, caused the resulting dataset to lose its schema, breaking downstream usage of attributes like `column_names`.

Why
Previously, the code would always set `info.features = features`, even if `features` was `None`. When mapping with removal of columns or other transformations, this led to the destruction of the schema and caused failures in code that relied on the dataset schema being present.

How
We now update `info.features` only if `features` is not `None`. This preserves the original schema unless the user explicitly provides a new one.

Reference
Fixes #7568


feat: add subset_name as alias for name in load_dataset

fixes #7637

This PR introduces subset_name as a user-facing alias for the name (previously `config_name`) argument in load_dataset. It aligns terminology with the Hugging Face Hub UI (which shows “Subset”), reducing confusion for new users.

Supports `subset_name` in `load_dataset()`

Adds `.subset_name` property to DatasetBuilder

Maintains full backward compatibility

Raises clear error if name and `subset_name` conflict


fix(iterable): ensure MappedExamplesIterable supports state_dict for resume

Fixes #7630

### Problem
When calling `.map()` on an `IterableDataset`, resuming from a checkpoint skips a large number of samples. This is because `MappedExamplesIterable` did not implement `state_dict()` or `load_state_dict()`, so checkpointing was not properly delegated to the underlying iterable.

### What This PR Does
This patch adds:
```python
def state_dict(self):
    return self.ex_iterable.state_dict()

def load_state_dict(self, state):
    self.ex_iterable.load_state_dict(state)
```

to MappedExamplesIterable, so the wrapped base iterable's state can be saved and restored as expected.

Result
Using .map() no longer causes sample skipping after checkpoint resume.

Let me know if a dedicated test case is required — happy to add one!


Added specific use cases in Improve Performace

Fixes #2494


fix(load): strip deprecated use_auth_token from config_kwargs

Fixes #7504

This PR resolves a compatibility error when loading datasets via `load_dataset()` using outdated arguments like `use_auth_token`.

**What was happening:**
Users passing `use_auth_token` in `load_dataset(..., use_auth_token=...)` encountered a `ValueError`: BuilderConfig ParquetConfig(...) doesn't have a 'use_auth_token' key.

**Why:**
`use_auth_token` has been deprecated and removed from config definitions (replaced by `token`), but the `load_dataset()` function still forwarded it via `**config_kwargs` to BuilderConfigs, leading to unrecognized key errors.

**Fix:**
We now intercept and strip `use_auth_token` from `config_kwargs` inside `load_dataset`, replacing it with a warning:
```python
if "use_auth_token" in config_kwargs:
    logger.warning("The 'use_auth_token' argument is deprecated. Please use 'token' instead.")
    config_kwargs.pop("use_auth_token")

```
This ensures legacy compatibility while guiding users to switch to the token argument.

Let me know if you'd prefer a deprecation error instead of a warning. Thanks!


feat(load): fallback to `load_from_disk()` when loading a saved dataset directory

### Related Issue

Fixes #7503
Partially addresses #5044 by allowing `load_dataset()` to auto-detect and gracefully delegate to `load_from_disk()` for locally saved datasets.

---

### What does this PR do?

This PR introduces a minimal fallback mechanism in `load_dataset()` that detects when the provided `path` points to a dataset saved using `save_to_disk()`, and automatically redirects to `load_from_disk()`.

#### 🐛 Before (unexpected metadata-only rows):

```python
ds = load_dataset("/path/to/saved_dataset")
# → returns rows with only internal metadata (_data_files, _fingerprint, etc.)
````

#### ✅ After (graceful fallback):

```python
ds = load_dataset("/path/to/saved_dataset")
# → logs a warning and internally switches to load_from_disk()
```

---

### Why is this useful?

* Prevents confusion when reloading local datasets saved via `save_to_disk()`.
* Enables smoother compatibility with frameworks (e.g., TRL, `lighteval`) that rely on `load_dataset()` calls.
* Fully backward-compatible — hub-based loading, custom builders, and streaming remain untouched.


Add columns support to JSON loader for selective key filtering

Fixes #7594
This PR adds support for filtering specific columns when loading datasets from .json or .jsonl files — similar to how the columns=... argument works for Parquet.

As suggested, support for the `columns=...` argument (previously available for Parquet) has now been extended to **JSON and JSONL** loading via `load_dataset(...)`. You can now load only specific keys/columns and skip the rest — which should help in cases where some fields are unclean, inconsistent, or just unnecessary.

### Example:

```python
from datasets import load_dataset

dataset = load_dataset("json", data_files="your_data.jsonl", columns=["id", "title"])
print(dataset["train"].column_names)
# Output: ['id', 'title']
```

### Summary of changes:

* Added `columns: Optional[List[str]]` to `JsonConfig`
* Updated `_generate_tables()` to filter selected columns
* Forwarded `columns` argument from `load_dataset()` to the config
* Added test for validation(should be fine!)

Let me know if you'd like the same to be added for CSV or others as a follow-up — happy to help.


fix: Extended metadata file names for folder_based_builder

Fixes #7650.

The metadata files generated by the `DatasetDict.save_to_file` function are not included in the folder_based_builder's metadata list, causing issues when only 1 actual data file is present, as described in issue #7650.

This PR adds these filenames to the builder, allowing correct loading.


`load_dataset` defaults to json file format for datasets with 1 shard

### Describe the bug

I currently have multiple datasets (train+validation) saved as 50MB shards. For one dataset the validation pair is small enough to fit into a single shard and this apparently causes problems when loading the dataset. I created the datasets using a DatasetDict, saved them as 50MB arrow files for streaming and then load each dataset. I have no problem loading any of the other datasets with more than 1 arrow file/shard.

The error indicates the training set got loaded in arrow format (correct) and the validation set in json (incorrect). This seems to be because some of the metadata files are considered as dataset files.
```
Error loading /nfs/dataset_pt-uk: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('validation'): ('json', {})}
```

![Image](https://github.com/user-attachments/assets/f6e7596a-dd53-46a9-9a23-4e9cac2ac049)

Concretely, there is a mismatch between the metadata created by the `DatasetDict.save_to_file` and the builder for `datasets.load_dataset`:

https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/data_files.py#L107

The `folder_based_builder` lists all files and with 1 arrow file the json files (that are actually metadata) are in the majority.
https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58

### Steps to reproduce the bug

Create a dataset with metadata and 1 arrow file in validation and multiple arrow files in the training set, following the above description. In my case, I saved the files via:

```python
        dataset = DatasetDict({
            'train': train_dataset,
            'validation': val_dataset
        })

        dataset.save_to_disk(output_path, max_shard_size="50MB")
```

### Expected behavior

The dataset would get loaded.

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-6.14.0-22-generic-x86_64-with-glibc2.41
- Python version: 3.12.7
- `huggingface_hub` version: 0.31.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1


Enable parallel shard upload in push_to_hub() using num_proc

Fixes #7591

### Add num_proc support to `push_to_hub()` for parallel shard upload

This PR adds support for parallel upload of dataset shards via the `num_proc` argument in `Dataset.push_to_hub()`.

📌 While the `num_proc` parameter was already present in the `push_to_hub()` signature and correctly passed to `_push_parquet_shards_to_hub()`, it was not being used to parallelize the upload.

🔧 This PR updates the internal `_push_parquet_shards_to_hub()` function to:

- Use `multiprocessing.Pool` and `iflatmap_unordered()` for concurrent shard upload when `num_proc > 1`
- Preserve original serial upload behavior if `num_proc` is `None` or ≤ 1
- Keep tqdm progress and commit behavior unchanged

Let me know if any test coverage or further changes are needed!


Fix misleading add_column() usage example in docstring

Fixes #7611

This PR fixes the usage example in the Dataset.add_column() docstring, which previously implied that add_column() modifies the dataset in-place.

Why:
The method returns a new dataset with the additional column, and users must assign the result to a variable to preserve the change.

This should make the behavior clearer for users.
@lhoestq @davanstrien


loading mozilla-foundation--common_voice_11_0 fails

### Describe the bug

Hello everyone,

i am trying to load `mozilla-foundation--common_voice_11_0` and it fails. Reproducer

```
import datasets
datasets.load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True, trust_remote_code=True)
```
and it fails with
```
File ~/opt/envs/.../lib/python3.10/site-packages/datasets/utils/file_utils.py:827, in _add_retries_to_file_obj_read_method.<locals>.read_with_retries(*args, **kwargs)
    825 for retry in range(1, max_retries + 1):
    826     try:
--> 827         out = read(*args, **kwargs)
    828         break
    829     except (
    830         _AiohttpClientError,
    831         asyncio.TimeoutError,
    832         requests.exceptions.ConnectionError,
    833         requests.exceptions.Timeout,
    834     ) as err:

File /usr/lib/python3.10/codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
    319 def decode(self, input, final=False):
    320     # decode input (taking the buffer into account)
    321     data = self.buffer + input
--> 322     (result, consumed) = self._buffer_decode(data, self.errors, final)
    323     # keep undecoded input until the next call
    324     self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```
When i remove streaming then everything is good but i need `streaming=True`

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True, trust_remote_code=True)
```

### Expected behavior

Expected that it will download dataset

### Environment info

datasets==3.6.0
python3.10
on all platforms linux/win/mac


Introduces automatic subset-level grouping for folder-based dataset builders #7066

Fixes #7066
This PR introduces automatic **subset-level grouping** for folder-based dataset builders by:

1. Adding a utility function `group_files_by_subset()` that clusters files by root name (ignoring digits and shard suffixes).
2. Integrating this logic into `FolderBasedBuilder._split_generators()` to yield one split per subset.
3. Adding unit tests for the grouping function.
4. Updating the documentation to describe this new behavior under `docs/source/repository_structure.mdx`.

---

### Motivation

Datasets with files like:
```

train0.jsonl
train1.jsonl
animals.jsonl
metadata.jsonl

```

will now be **automatically grouped** as:
- `"train"` subset → `train0.jsonl`, `train1.jsonl`
- `"animals"` subset → `animals.jsonl`
- `"metadata"` subset → `metadata.jsonl`

This enables structured multi-subset loading even when the dataset doesn't follow traditional `train/validation/test` split conventions.

---

### Files Changed

- `src/datasets/data_files.py`: added `group_files_by_subset()` utility
- `src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py`: grouped files before yielding splits
- `tests/test_data_files.py`: added unit test `test_group_files_by_subset`
- `docs/source/repository_structure.mdx`: documented subset grouping for maintainers and users

---

### Benefits

- More flexible and robust dataset split logic
- Enables logical grouping of user-uploaded files without nested folder structure
- Backward-compatible with all existing folder-based configs

---

Ready for review ✅


`ClassLabel` docs: Correct value for unknown labels

This small change fixes the documentation to to be compliant with what happens in `encode_example`.
https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/features/features.py#L1126-L1129


fix sequence ci

fix error from https://github.com/huggingface/datasets/pull/7643


Backward compat sequence instance

useful to still get `isinstance(Sequence(Value("int64")), Sequence)`for downstream libs like evaluate


fix length for ci


update docs and docstrings


better features repr

following the addition of List in #7634

before:

```python
In [3]: ds.features
Out[3]:
{'json': {'id': Value(dtype='string', id=None),
  'metadata:transcript': [{'end': Value(dtype='float64', id=None),
    'start': Value(dtype='float64', id=None),
    'transcript': Value(dtype='string', id=None),
    'words': [{'end': Value(dtype='float64', id=None),
      'score': Value(dtype='float64', id=None),
      'start': Value(dtype='float64', id=None),
      'word': Value(dtype='string', id=None)}]}],
  'metadata:vad': [{'end': Value(dtype='float64', id=None),
    'start': Value(dtype='float64', id=None)}]},
 'mp4': Value(dtype='binary', id=None),
 'npz': {'boxes_and_keypoints:box': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'boxes_and_keypoints:is_valid_box': Sequence(feature=Value(dtype='bool', id=None), length=-1, id=None),
  'boxes_and_keypoints:keypoints': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
  'movement:EmotionArousalToken': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:EmotionValenceToken': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:FAUToken': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:FAUValue': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:alignment_head_rotation': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:alignment_translation': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
  'movement:emotion_arousal': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:emotion_scores': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:emotion_valence': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:expression': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:frame_latent': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:gaze_encodings': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:head_encodings': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:hypernet_features': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'movement:is_valid': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'smplh:body_pose': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
  'smplh:global_orient': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
  'smplh:is_valid': Sequence(feature=Value(dtype='bool', id=None), length=-1, id=None),
  'smplh:left_hand_pose': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
  'smplh:right_hand_pose': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
  'smplh:translation': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None)},
 'wav': Audio(sampling_rate=None, mono=True, decode=True, id=None),
 '__key__': Value(dtype='string', id=None),
 '__url__': Value(dtype='string', id=None)}
```

after:

```python
In [3]: ds.features
Out[3]:
{'json': {'id': Value('string'),
  'metadata:transcript': List({'end': Value('float64'), 'start': Value('float64'), 'transcript': Value('string'), 'words': List({'end': Value('float64'), 'score': Value('float64'), 'start': Value('float64'), 'word': Value('string')})}),
  'metadata:vad': List({'end': Value('float64'), 'start': Value('float64')})},
 'mp4': Value('binary'),
 'npz': {'boxes_and_keypoints:box': List(List(Value('float32'))),
  'boxes_and_keypoints:is_valid_box': List(Value('bool')),
  'boxes_and_keypoints:keypoints': List(List(List(Value('float32')))),
  'movement:EmotionArousalToken': List(List(Value('float32'))),
  'movement:EmotionValenceToken': List(List(Value('float32'))),
  'movement:FAUToken': List(List(Value('float32'))),
  'movement:FAUValue': List(List(Value('float32'))),
  'movement:alignment_head_rotation': List(List(Value('float32'))),
  'movement:alignment_translation': List(List(List(Value('float32')))),
  'movement:emotion_arousal': List(List(Value('float32'))),
  'movement:emotion_scores': List(List(Value('float32'))),
  'movement:emotion_valence': List(List(Value('float32'))),
  'movement:expression': List(List(Value('float32'))),
  'movement:frame_latent': List(List(Value('float32'))),
  'movement:gaze_encodings': List(List(Value('float32'))),
  'movement:head_encodings': List(List(Value('float32'))),
  'movement:hypernet_features': List(List(Value('float32'))),
  'movement:is_valid': List(List(Value('float32'))),
  'smplh:body_pose': List(List(List(Value('float32')))),
  'smplh:global_orient': List(List(Value('float32'))),
  'smplh:is_valid': List(Value('bool')),
  'smplh:left_hand_pose': List(List(List(Value('float32')))),
  'smplh:right_hand_pose': List(List(List(Value('float32')))),
  'smplh:translation': List(List(Value('float32')))},
 'wav': Audio(sampling_rate=None, decode=True, stream_index=None),
 '__key__': Value('string'),
 '__url__': Value('string')}
```


fix save_infos


Add ignore_decode_errors option to Image feature for robust decoding #7612

This PR implements support for robust image decoding in the `Image` feature, as discussed in issue #7612.

## 🔧 What was added
- A new boolean field: `ignore_decode_errors` (default: `False`)
- If set to `True`, any exceptions during decoding will be caught, and `None` will be returned instead of raising an error

```python
features = Features({
    "image": Image(decode=True, ignore_decode_errors=True),
})
````

This enables robust iteration over potentially corrupted datasets — especially useful when streaming datasets like WebDataset or image-heavy public sets where sample corruption is common.

## 🧪 Behavior

* If `ignore_decode_errors=False` (default), decoding behaves exactly as before
* If `True`, decoding errors are caught, and a warning is emitted:

  ```
  [Image.decode_example] Skipped corrupted image: ...
  ```

## 🧵 Linked issue

Closes #7612

Let me know if you'd like a follow-up test PR. Happy to write one!


Introduce subset_name as an alias of config_name

enhancement

New feature or request

### Feature request

Add support for `subset_name` as an alias for `config_name` in the datasets library and related tools (such as loading scripts, documentation, and metadata).

### Motivation

The Hugging Face Hub dataset viewer displays a column named **"Subset"**, which refers to what is currently technically called config_name in the datasets library. This inconsistency has caused confusion for many users, especially those unfamiliar with the internal terminology.

I have repeatedly received questions from users trying to understand what "config" means, and why it doesn’t match what they see as "subset" on the Hub. Renaming everything to `subset_name` might be too disruptive, but introducing subset_name as a clear alias for config_name could significantly improve user experience without breaking backward compatibility.

This change would:
- Align terminology across the Hub UI and datasets codebase
- Reduce user confusion, especially for newcomers
- Make documentation and examples more intuitive


docs: Add column overwrite example to batch mapping guide

This PR adds a complementary example showing the **column-overwriting** pattern, which is both more direct and more flexible for many transformations.

### Proposed Change

The original `remove_columns` example remains untouched. Below it, this PR introduces an alternative approach that overwrites an existing column during batch mapping.

This teaches users a core `.map()` capability for in-place transformations without extra intermediate steps.

**New Example:**

> ```python
> >>> from datasets import Dataset
> >>> dataset = Dataset.from_dict({"a": [0, 1, 2]})
> # Overwrite "a" directly to duplicate each value
> >>> duplicated_dataset = dataset.map(
> ...     lambda batch: {"a": [x for x in batch["a"] for _ in range(2)]},
> ...     batched=True
> ... )
> >>> duplicated_dataset
> Dataset({
>     features: ['a'],
>     num_rows: 6
> })
> >>> duplicated_dataset["a"]
> [0, 0, 1, 1, 2, 2]
> ```


Fix type hint `train_test_split`


fix largelist repr


Fixing __getitem__ of datasets which behaves inconsistent to documentation when setting _format_type to None

Setting _format_type to None, should return plain python object but as of 4.0.0 returns Column. This fails in libs such as sentencetransformers (such as in generation of hard negatives) where plain python is expected.


Dataset Repo Paths to Locally Stored Images Not Being Appended to Image Path

### Describe the bug

I’m not sure if this is a bug or a feature and I just don’t fully understand how dataset loading is to work, but it appears there may be a bug with how locally stored Image() are being accessed. I’ve uploaded a new dataset to hugging face (rmdig/rocky_mountain_snowpack) but I’ve come into a ton of trouble trying to have the images handled properly (at least in the way I’d expect them to be handled).

I find that I cannot use relative paths for loading images remotely from the Hugging Face repo or from a local repository. Any time I do it always simply appends my current working directory to the dataset. As a result to use the datasets library with my dataset I have to change my working directory to the dataset folder or abandon the dataset object structure, which I cannot imagine you intended. As a result I have to use URL’s since an absolute path on my system obviously wouldn’t work for others. The URL works ok, but despite me having it locally downloaded, it appears to be redownloading the dataset every time I train my snowGAN model on it (and often times I’m coming into HTTPS errors for over requesting the data).

Or maybe image relative paths aren't intended to be loaded directly through your datasets library as images and should be kept as strings for the user to handle? If so I feel like you’re missing out on some pretty seamless functionality

### Steps to reproduce the bug

1. Download a local copy of the dataset (rmdig/rocky_mountain_snowpack) through git or whatever you prefer.
2. Alter the README.md YAML for file_path (the relative path to each image) to be type Image instead of type string

`
---
dataset_info:
  features:
    - name: image
      dtype: Image
    - name: file_path
      dtype: Image
`

3. Initialize the dataset locally, make sure your working directory is not the dataset directory root

`dataset = datasets.load_dataset(‘path/to/local/rocky_mountain_snowpack/‘)`

4. Call to one of the samples and you’ll get an error that the image was not found in current/working/directory/preprocessed/cores/image_1.png. Showing that it’s simply looking in the current working directory + relative path

`
>>> dataset['train'][0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 2859, in __getitem__
    return self._getitem(key)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 2841, in _getitem
    formatted_output = format_table(
                       ^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/formatting/formatting.py", line 657, in format_table
    return formatter(pa_table, query_type=query_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/formatting/formatting.py", line 410, in __call__
    return self.format_row(pa_table)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/formatting/formatting.py", line 459, in format_row
    row = self.python_features_decoder.decode_row(row)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/formatting/formatting.py", line 223, in decode_row
    return self.features.decode_example(row, token_per_repo_id=self.token_per_repo_id) if self.features else row
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/features/features.py", line 2093, in decode_example
    column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/features/features.py", line 1405, in decode_nested_example
    return schema.decode_example(obj, token_per_repo_id=token_per_repo_id) if obj is not None else None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/datasets/features/image.py", line 171, in decode_example
    image = PIL.Image.open(path)
            ^^^^^^^^^^^^^^^^^^^^
  File "/Users/dennyschaedig/miniconda3/lib/python3.12/site-packages/PIL/Image.py", line 3277, in open
    fp = builtins.open(filename, "rb")
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/dennyschaedig/Datasets/preprocessed/cores/image_1.png'
`

### Expected behavior

I expect the datasets and Image() to load the locally hosted data using path/to/local/rocky_mountain_snowpack/ (that I pass in with my datasets.load_dataset() or the you all handle on the backend) call + relative path.

Instead it appears to load from my current working directory + relative path.

### Environment info

Tested on…

Windows 11, Ubuntu Linux 22.04 and Mac Sequoia 15.5 Silicone M2

datasets version 4.0.0

Python 3.12 and 3.13


webdataset: key errors when `field_name` has upper case characters

### Describe the bug

When using a webdataset each sample can be a collection of different "fields"
like this:
```
images17/image194.left.jpg
images17/image194.right.jpg
images17/image194.json
images17/image12.left.jpg
images17/image12.right.jpg
images17/image12.json
```

if the field_name contains upper case characters, the HF webdataset integration throws a key error when trying to load the dataset:
e.g. from a dataset (now updated so that it doesn't throw this error)

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[1], line 2
      1 from datasets import load_dataset
----> 2 ds = load_dataset("commaai/comma2k19", data_files={'train': ['data-00000.tar.gz']}, num_proc=1)

File ~/xx/.venv/lib/python3.11/site-packages/datasets/load.py:1412, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, keep_in_memory, save_infos, revision, token, streaming, num_proc, storage_options, **config_kwargs)
   1409     return builder_instance.as_streaming_dataset(split=split)
   1411 # Download and prepare data
-> 1412 builder_instance.download_and_prepare(
   1413     download_config=download_config,
   1414     download_mode=download_mode,
   1415     verification_mode=verification_mode,
   1416     num_proc=num_proc,
   1417     storage_options=storage_options,
   1418 )
   1420 # Build dataset for splits
   1421 keep_in_memory = (
   1422     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1423 )

File ~/xx/.venv/lib/python3.11/site-packages/datasets/builder.py:894, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, dl_manager, base_path, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    892 if num_proc is not None:
    893     prepare_split_kwargs["num_proc"] = num_proc
--> 894 self._download_and_prepare(
    895     dl_manager=dl_manager,
    896     verification_mode=verification_mode,
    897     **prepare_split_kwargs,
    898     **download_and_prepare_kwargs,
    899 )
    900 # Sync info
    901 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/xx/.venv/lib/python3.11/site-packages/datasets/builder.py:1609, in GeneratorBasedBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_splits_kwargs)
   1608 def _download_and_prepare(self, dl_manager, verification_mode, **prepare_splits_kwargs):
-> 1609     super()._download_and_prepare(
   1610         dl_manager,
   1611         verification_mode,
   1612         check_duplicate_keys=verification_mode == VerificationMode.BASIC_CHECKS
   1613         or verification_mode == VerificationMode.ALL_CHECKS,
   1614         **prepare_splits_kwargs,
   1615     )

File ~/xx/.venv/lib/python3.11/site-packages/datasets/builder.py:948, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
    946 split_dict = SplitDict(dataset_name=self.dataset_name)
    947 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 948 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    950 # Checksums verification
    951 if verification_mode == VerificationMode.ALL_CHECKS and dl_manager.record_checksums:

File ~/xx/.venv/lib/python3.11/site-packages/datasets/packaged_modules/webdataset/webdataset.py:81, in WebDataset._split_generators(self, dl_manager)
     78 if not self.info.features:
     79     # Get one example to get the feature types
     80     pipeline = self._get_pipeline_from_tar(tar_paths[0], tar_iterators[0])
---> 81     first_examples = list(islice(pipeline, self.NUM_EXAMPLES_FOR_FEATURES_INFERENCE))
     82     if any(example.keys() != first_examples[0].keys() for example in first_examples):
     83         raise ValueError(
     84             "The TAR archives of the dataset should be in WebDataset format, "
     85             "but the files in the archive don't share the same prefix or the same types."
     86         )

File ~/xx/.venv/lib/python3.11/site-packages/datasets/packaged_modules/webdataset/webdataset.py:55, in WebDataset._get_pipeline_from_tar(cls, tar_path, tar_iterator)
     53         data_extension = field_name.split(".")[-1]
     54     if data_extension in cls.DECODERS:
---> 55         current_example[field_name] = cls.DECODERS[data_extension](current_example[field_name])
     56 if current_example:
     57     yield current_example

KeyError: 'processed_log_IMU_magnetometer_value.npy'
```

### Steps to reproduce the bug

unit test was added in: https://github.com/huggingface/datasets/pull/7726
it fails without the fixed proposed in the same PR

### Expected behavior

Not throwing a key error.

### Environment info

```
- `datasets` version: 4.0.0
- Platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.39
- Python version: 3.11.4
- `huggingface_hub` version: 0.33.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.7.0
```


New feature or request

### Feature request

Add the possibility of a backend for audio decoding. Before version 4.0.0, soundfile was used, and now torchcodec is used, but the problem is that torchcodec requires ffmpeg, which is problematic to install on the same colab. Therefore, I suggest adding a decoder selection when loading the dataset.

### Motivation

I use a service for training models in which ffmpeg cannot be installed.

### Your contribution

I use a service for training models in which ffmpeg cannot be installed.


Grammar fix: correct "showed" to "shown" in fingerprint.py

This PR corrects a small grammatical issue in the outputs of fingerprint.py:

```diff
- "This warning is only showed once. Subsequent hashing failures won't be showed."
+ "This warning is only shown once. Subsequent hashing failures won't be shown."
```


OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory

> Hi  is there any solution for that eror i try to install this one
pip install torch==1.12.1+cpu torchaudio==0.12.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
this is working fine but tell me how to install pytorch version that is fit for gpu


NonMatchingSplitsSizesError and ExpectedMoreSplitsError

### Describe the bug

When loading dataset, the info specified by `data_files` did not overwrite the original info.

### Steps to reproduce the bug

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz",
                        "validation": "en/c4-validation.00000-of-00008.json.gz"},
        )
```

```log
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=828589180707, num_examples=364868892, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=809262831, num_examples=356317, shard_lengths=[223006, 133311], dataset_name='c4')}, {'expected': SplitInfo(name='validation', num_bytes=825767266, num_examples=364608, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=102199431, num_examples=45576, shard_lengths=None, dataset_name='c4')}]
```

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
            split="train"
        )
```

```log
ExpectedMoreSplitsError: {'validation'}
```

### Expected behavior

No error

### Environment info

datasets 4.0.0


config paths that start with ./ are not valid as hf:// accessed repos, but are valid when accessed locally

### Describe the bug

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - images/xyz/*.jpg
```
will correctly download but

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```
will error with `FileNotFoundError` due to improper url joining. `load_dataset` on the same directory locally works fine.

### Steps to reproduce the bug

1. create a README.md with the front matter of the form

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```

2. `touch ./images/xyz/1.jpg`
3. Observe this directory loads with `load_dataset("filesystem_path", "some_config")` correctly.
4. Observe exceptions when you load this with `load_dataset("repoid/filesystem_path", "some_config")`

### Expected behavior

`./` prefix should be interpreted correctly

### Environment info

datasets 4.0.0
datasets 3.4.0

reproduce


fix(webdataset): don't .lower() field_name

This fixes cases where keys have upper case identifiers


Can not stepinto load_dataset.py?

I set a breakpoint in "load_dataset.py" and try to debug my data load codes, but it does not stop at any breakpoints, so "load_dataset.py" can not be stepped into ?

<!-- Failed to upload "截图 2025-08-05 17-25-18.png" -->


Don't remove `trust_remote_code` arg!!!

New feature or request

### Feature request

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!

Add `trust_remote_code` arg back please!

### Motivation

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!

### Your contribution

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!


Out of memory even though using load_dataset(..., streaming=True)

### Describe the bug

I am iterating over a large dataset that I load using streaming=True to avoid running out of memory. Unfortunately, I am observing that memory usage increases over time and I'm finally running in an oom.

### Steps to reproduce the bug

```
  ds = load_dataset("openslr/librispeech_asr", split="train.clean.360", streaming=True)
  for i,sample in enumerate(tqdm(ds)):
    target_file = os.path.join(NSFW_TARGET_FOLDER, f'audio{i}.wav')
    try:
      sf.write(target_file, sample['audio']['array'], samplerate=sample['audio']['sampling_rate'])
    except Exception as e:
      print(f"Could not write audio {i} in ds: {e}")
```

### Expected behavior

I'd expect to have a small memory footprint and memory being freed after each iteration of the for loop. Instead the memory usage is increasing. I tried to remove the logic to write the sound file and just print the sample but the issue remains the same.

### Environment info

Python 3.12.11
Ubuntu 24
datasets 4.0.0 and 3.6.0


Bad split error message when using percentages

### Describe the bug

Hi, I'm trying to download a dataset. To not load the entire dataset in memory, I split it as described [here](https://huggingface.co/docs/datasets/v4.0.0/loading#slice-splits) in 10% steps.

When doing so, the library returns this error:
    raise ValueError(f"Bad split: {split}. Available splits: {list(splits_generators)}")
ValueError: Bad split: train[0%:10%]. Available splits: ['train']

Edit: Same happens with a split like _train[:90000]_
### Steps to reproduce the bug

```
  for split in range(10):
    split_str = f"train[{split*10}%:{(split+1)*10}%]"
    print(f"Processing split {split_str}...")
    ds = load_dataset("user/dataset", split=split_str, streaming=True)
```

### Expected behavior

I'd expect the library to split my dataset in 10% steps.

### Environment info

python 3.12.11
ubuntu 24
dataset 4.0.0


Datasets 4.0 map function causing column not found

### Describe the bug

Column returned after mapping is not found in new instance of the dataset.

### Steps to reproduce the bug

Code for reproduction. After running get_total_audio_length, it is errored out due to `data` not having `duration`
```
def compute_duration(x):
    return {"duration": len(x["audio"]["array"]) / x["audio"]["sampling_rate"]}

def get_total_audio_length(dataset):
    data = dataset.map(compute_duration, num_proc=NUM_PROC)
    print(data)
    durations=data["duration"]
    total_seconds = sum(durations)
    return total_seconds
```

### Expected behavior

New datasets.Dataset instance should have new columns attached.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 20.0.0
- Pandas version: 2.3.0
- `fsspec` version: 2023.12.2


Specify dataset columns types in typehint

New feature or request

### Feature request

Make dataset optionaly generic to datasets usage with type annotations like it was done in `torch.Dataloader` https://github.com/pytorch/pytorch/blob/134179474539648ba7dee1317959529fbd0e7f89/torch/utils/data/dataloader.py#L131

### Motivation

In MTEB we're using a lot of datasets objects, but they're a bit poor in typehints. E.g. we can specify this for dataloder

```python
from typing import TypedDict
from torch.utils.data import DataLoader

class CorpusInput(TypedDict):
    title: list[str]
    body: list[str]

class QueryInput(TypedDict):
    query: list[str]
    instruction: list[str]

def queries_loader() -> DataLoader[QueryInput]:
    ...

def corpus_loader() -> DataLoader[CorpusInput]:
   ...
```
But for datasets we can only specify columns in type in comments
```python
from datasets import Dataset

QueryDataset = Dataset
"""Query dataset should have `query` and `instructions` columns as `str` """
```

### Your contribution

I can create draft implementation


add support for pyarrow string view in features


Cached dataset is not used when explicitly passing the cache_dir parameter

### Describe the bug

Hi, we are pre-downloading a dataset using snapshot_download(). When loading this exact dataset with load_dataset() the cached snapshot is not used. In both calls, I provide the cache_dir parameter.

### Steps to reproduce the bug

```
from datasets import load_dataset, concatenate_datasets
from huggingface_hub import snapshot_download

def download_ds(name: str):
    snapshot_download(repo_id=name, repo_type="dataset", cache_dir="G:/Datasets/cache")

def prepare_ds():
    audio_ds = load_dataset("openslr/librispeech_asr", num_proc=4, cache_dir="G:/Datasets/cache")
    print(sfw_ds.features)

if __name__ == '__main__':
    download_ds("openslr/librispeech_asr")
    prepare_ds()
```

### Expected behavior

I'd expect that the cached version of the dataset is used. Instead, the same dataset is downloaded again to the default cache directory.

### Environment info

Windows 11
datasets==4.0.0
Python 3.12.11


typo


Docs: Use Image(mode="F") for PNG/JPEG depth maps


fix num_proc=1 ci test


Update cli.mdx to refer to the new "hf" CLI

Update to refer to `hf auth login`


Retry intermediate commits too


Update dataset_dict push_to_hub

following https://github.com/huggingface/datasets/pull/7708


Concurrent IterableDataset push_to_hub

Same as https://github.com/huggingface/datasets/pull/7708 but for `IterableDataset`


Release 4.0.0 breaks usage patterns of with_format

### Describe the bug

Previously it was possible to access a whole column that was e.g. in numpy format via `with_format` by indexing the column. Now this possibility seems to be gone with the new Column() class. As far as I see, this makes working on a whole column (in-memory) more complex, i.e. normalizing an in-memory dataset for which iterating would be too slow. Is this intended behaviour? I couldn't find much documentation on the intended usage of the new Column class yet.

### Steps to reproduce the bug

Steps to reproduce:
```
from datasets import load_dataset
dataset = load_dataset("lhoestq/demo1")
dataset = dataset.with_format("numpy")
print(dataset["star"].ndim)
```

### Expected behavior

Working on whole columns should be possible.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.8.0-63-generic-x86_64-with-glibc2.36
- Python version: 3.12.11
- `huggingface_hub` version: 0.34.3
- PyArrow version: 21.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0


Concurrent push_to_hub

Retry the step that (download + update + upload) the README.md using `create_commit(..., parent_commit=...)` if there was a commit in the meantime. This should enable concurrent `push_to_hub()` since it won't overwrite the README.md metadata anymore.

Note: we fixed an issue server side to make this work:

<details>

DO NOT MERGE FOR NOW since it seems there is one bug that prevents this logic from working:

I'm using parent_commit to enable concurrent push_to_hub() in datasets for a retry mechanism, but for some reason I always run into a weird situation.
Sometimes create_commit(.., parent_commit=...) returns error 500 but the commit did happen on the Hub side without respecting parent_commit

e.g. request id
```
huggingface_hub.errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/lhoestq/tmp/commit/main (Request ID: Root=1-6888d8af-2ce517bc60c69cb378b51526;d1b17993-c5d0-4ccd-9926-060c45f9ed61)
```

fix coming in [internal](https://github.com/huggingface-internal/moon-landing/pull/14617)

</details>

close https://github.com/huggingface/datasets/issues/7600


load_dataset() in 4.0.0 failed when decoding audio

### Describe the bug

Cannot decode audio data.

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
print(dataset[0]["audio"]["array"])
```

1st round run, got
```
  File "/usr/local/lib/python3.12/dist-packages/datasets/features/audio.py", line 172, in decode_example
    raise ImportError("To support decoding audio data, please install 'torchcodec'.")
ImportError: To support decoding audio data, please install 'torchcodec'.
```

After `pip install torchcodec` and run, got
```
  File "/usr/local/lib/python3.12/dist-packages/torchcodec/_core/_metadata.py", line 16, in <module>
    from torchcodec._core.ops import (
  File "/usr/local/lib/python3.12/dist-packages/torchcodec/_core/ops.py", line 84, in <module>
    load_torchcodec_shared_libraries()
  File "/usr/local/lib/python3.12/dist-packages/torchcodec/_core/ops.py", line 69, in load_torchcodec_shared_libraries
    raise RuntimeError(
RuntimeError: Could not load libtorchcodec. Likely causes:
          1. FFmpeg is not properly installed in your environment. We support
             versions 4, 5, 6 and 7.
          2. The PyTorch version (2.8.0a0+5228986c39.nv25.06) is not compatible with
             this version of TorchCodec. Refer to the version compatibility
             table:
             https://github.com/pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec.
          3. Another runtime dependency; see exceptions below.
        The following exceptions were raised as we tried to load libtorchcodec:

[start of libtorchcodec loading traceback]
FFmpeg version 7: libavutil.so.59: cannot open shared object file: No such file or directory
FFmpeg version 6: libavutil.so.58: cannot open shared object file: No such file or directory
FFmpeg version 5: libavutil.so.57: cannot open shared object file: No such file or directory
FFmpeg version 4: libavutil.so.56: cannot open shared object file: No such file or directory
[end of libtorchcodec loading traceback].
```

After `apt update && apt install ffmpeg -y`, got
```
Traceback (most recent call last):
  File "/workspace/jiqing/test_datasets.py", line 4, in <module>
    print(dataset[0]["audio"]["array"])
          ~~~~~~~^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/arrow_dataset.py", line 2859, in __getitem__
    return self._getitem(key)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/arrow_dataset.py", line 2841, in _getitem
    formatted_output = format_table(
                       ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/formatting/formatting.py", line 657, in format_table
    return formatter(pa_table, query_type=query_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/formatting/formatting.py", line 410, in __call__
    return self.format_row(pa_table)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/formatting/formatting.py", line 459, in format_row
    row = self.python_features_decoder.decode_row(row)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/formatting/formatting.py", line 223, in decode_row
    return self.features.decode_example(row, token_per_repo_id=self.token_per_repo_id) if self.features else row
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/features/features.py", line 2093, in decode_example
    column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/features/features.py", line 1405, in decode_nested_example
    return schema.decode_example(obj, token_per_repo_id=token_per_repo_id) if obj is not None else None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/datasets/features/audio.py", line 198, in decode_example
    audio = AudioDecoder(bytes, stream_index=self.stream_index, sample_rate=self.sampling_rate)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torchcodec/decoders/_audio_decoder.py", line 62, in __init__
    self._decoder = create_decoder(source=source, seek_mode="approximate")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torchcodec/decoders/_decoder_utils.py", line 33, in create_decoder
    return core.create_from_bytes(source, seek_mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torchcodec/_core/ops.py", line 144, in create_from_bytes
    return create_from_tensor(buffer, seek_mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 756, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
NotImplementedError: Could not run 'torchcodec_ns::create_from_tensor' with arguments from the 'CPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'torchcodec_ns::create_from_tensor' is only available for these backends: [Meta, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy, AutogradMTIA, AutogradMAIA, AutogradMeta, Tracer, AutocastCPU, AutocastMTIA, AutocastMAIA, AutocastXPU, AutocastMPS, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

Meta: registered at /dev/null:214 [kernel]
BackendSelect: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
Python: registered at /__w/torchcodec/torchcodec/pytorch/torchcodec/src/torchcodec/_core/custom_ops.cpp:694 [kernel]
FuncTorchDynamicLayerBackMode: registered at /opt/pytorch/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:479 [backend fallback]
Functionalize: registered at /opt/pytorch/pytorch/aten/src/ATen/FunctionalizeFallbackKernel.cpp:349 [backend fallback]
Named: registered at /opt/pytorch/pytorch/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
Conjugate: registered at /opt/pytorch/pytorch/aten/src/ATen/ConjugateFallback.cpp:17 [backend fallback]
Negative: registered at /opt/pytorch/pytorch/aten/src/ATen/native/NegateFallback.cpp:18 [backend fallback]
ZeroTensor: registered at /opt/pytorch/pytorch/aten/src/ATen/ZeroTensorFallback.cpp:86 [backend fallback]
ADInplaceOrView: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:104 [backend fallback]
AutogradOther: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:63 [backend fallback]
AutogradCPU: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:67 [backend fallback]
AutogradCUDA: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:75 [backend fallback]
AutogradXLA: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:87 [backend fallback]
AutogradMPS: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:95 [backend fallback]
AutogradXPU: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:71 [backend fallback]
AutogradHPU: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:108 [backend fallback]
AutogradLazy: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:91 [backend fallback]
AutogradMTIA: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:79 [backend fallback]
AutogradMAIA: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:83 [backend fallback]
AutogradMeta: registered at /opt/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:99 [backend fallback]
Tracer: registered at /opt/pytorch/pytorch/torch/csrc/autograd/TraceTypeManual.cpp:294 [backend fallback]
AutocastCPU: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:322 [backend fallback]
AutocastMTIA: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:466 [backend fallback]
AutocastMAIA: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:504 [backend fallback]
AutocastXPU: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:542 [backend fallback]
AutocastMPS: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:209 [backend fallback]
AutocastCUDA: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:165 [backend fallback]
FuncTorchBatched: registered at /opt/pytorch/pytorch/aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:731 [backend fallback]
BatchedNestedTensor: registered at /opt/pytorch/pytorch/aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:758 [backend fallback]
FuncTorchVmapMode: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/functorch/VmapModeRegistrations.cpp:27 [backend fallback]
Batched: registered at /opt/pytorch/pytorch/aten/src/ATen/LegacyBatchingRegistrations.cpp:1075 [backend fallback]
VmapMode: fallthrough registered at /opt/pytorch/pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
FuncTorchGradWrapper: registered at /opt/pytorch/pytorch/aten/src/ATen/functorch/TensorWrapper.cpp:208 [backend fallback]
PythonTLSSnapshot: registered at /opt/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:202 [backend fallback]
FuncTorchDynamicLayerFrontMode: registered at /opt/pytorch/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:475 [backend fallback]
PreDispatch: registered at /opt/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:206 [backend fallback]
PythonDispatcher: registered at /opt/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:198 [backend fallback]
```

### Expected behavior

The result is
```
[0.00238037 0.0020752  0.00198364 ... 0.00042725 0.00057983 0.0010376 ]
```
on `datasets==3.6.0`

### Environment info

[NV official docker image](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch): `nvcr.io/nvidia/pytorch:25.06-py3`

```
- `datasets` version: 4.0.0
- Platform: Linux-5.4.292-1.el8.elrepo.x86_64-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.34.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0
```


Reimplemented partial split download support (revival of #6832)

(revival of #6832)

https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

Close https://github.com/huggingface/datasets/issues/4101, and more

---
### PR under work!!!!


Can Not read installed dataset in dataset.load(.)

Hi, folks, I'm newbie in huggingface dataset api.
As title, i'm facing the issue that the dataset.load api can not connect to the installed dataset.

code snippet :

data path :
"/xxx/joseph/llava_ds/vlm_ds"
it contains all video clips i want!

i run the py script by

But bad happended, even i provide dataset path by "HF_HUB_CACHE", it still attempt to download data from remote side :

Any suggestion will be appreciated!!


Fix map() example in datasets documentation: define tokenizer before use

## Problem
The current datasets.Dataset.map() example in the documentation demonstrates batched processing using a tokenizer object without defining or importing it. This causes a NameError when users copy and run the example as-is, breaking the expected seamless experience.

## Correction
This PR fixes the issue by explicitly importing and initializing the tokenizer using the Transformers library (AutoTokenizer.from_pretrained("bert-base-uncased")), making the example self-contained and runnable without errors.
This will help new users understand the workflow and apply the method correctly.

Closes #7703


[Docs] map() example uses undefined `tokenizer` — causes NameError

## Description

The current documentation example for `datasets.Dataset.map()` demonstrates batched processing but uses a `tokenizer` object without defining or importing it. This causes an error every time it's copied.

Here is the problematic line:

```python
# process a batch of examples
>>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This assumes the user has already set up a tokenizer, which contradicts the goal of having self-contained, copy-paste-friendly examples.

## Problem
Users who copy and run the example as-is will encounter:

```python
NameError: name 'tokenizer' is not defined
```

This breaks the flow for users and violates HuggingFace's documentation principle that examples should "work as expected" when copied directly.

## Proposal
Update the example to include the required tokenizer setup using the Transformers library, like so:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
ds_tokenized = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This will help new users understand the workflow and apply the method correctly.

## Note
This PR complements ongoing improvements like #7700, which clarifies multiprocessing in .map(). My change focuses on undefined tokenizer — causes NameError


num_proc=0 behave like None, num_proc=1 uses one worker (not main process) and clarify num_proc documentation

Fixes issue #7700

This PR makes num_proc=0 behave like None in Dataset.map(), disabling multiprocessing.
It improves UX by aligning with DataLoader(num_workers=0) behavior.
The num_proc docstring is also updated to clearly explain valid values and behavior.

@SunMarc


Update fsspec max version to current release 2025.7.0

Diffusers currently asks for a max fsspec version of `2025.3.0`. This change updates it to the current latest version. This change is mainly required to resolve conflicts with other packages in an environment. In my particular case, `aider-chat` which is a part of my environment installs `2025.5.1` which is incompatible with `datasets`.


[doc] map.num_proc needs clarification

https://huggingface.co/docs/datasets/v4.0.0/en/package_reference/main_classes#datasets.Dataset.map.num_proc
```
num_proc (int, optional, defaults to None) — Max number of processes when generating cache. Already cached
shards are loaded sequentially.
```

for batch:
```
num_proc (int, optional, defaults to None): The number of processes to use for multiprocessing. If None, no
multiprocessing is used. This can significantly speed up batching for large datasets.
```

So what happens to `map.num_proc` - is it the same behavior as `batch.num_proc` - so only if `num_proc=None` then no multiprocessing is used?

Let's update the doc to be unambiguous.

**bonus**: we could make all of these behave similarly to `DataLoader.num_workers` - where `num_workers==0` implies no multiprocessing. I think that's the most intuitive, IMHO. 0 workers - the main process has to do all the work. `None` could be the same as `0`.

context: debugging a failing `map`

Thank you!


Broken link in documentation for "Create a video dataset"

The link to "the [WebDataset documentation](https://webdataset.github.io/webdataset)." is broken.
https://huggingface.co/docs/datasets/main/en/video_dataset#webdataset

<img width="2048" height="264" alt="Image" src="https://github.com/user-attachments/assets/975dd10c-aad8-42fc-9fbc-de0e2747a326" />


NotImplementedError when using streaming=True in Google Colab environment

### Describe the bug

When attempting to load a large dataset (like tiiuae/falcon-refinedweb or allenai/c4) using streaming=True in a standard Google Colab notebook, the process fails with a NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet. This issue persists even after upgrading datasets and huggingface_hub and restarting the session.

### Steps to reproduce the bug

Open a new Google Colab notebook.

(Optional but recommended) Run !pip install --upgrade datasets huggingface_hub and restart the runtime.

Run the following code:

Python

from datasets import load_dataset

try:
    print("Attempting to load a stream...")
    streaming_dataset = load_dataset('tiiuae/falcon-refinedweb', streaming=True)
    print("Success!")
except Exception as e:
    print(e)

### Expected behavior

The load_dataset command should return a StreamingDataset object without raising an error, allowing iteration over the dataset.

Actual Behavior
The code fails and prints the following error traceback:

[PASTE THE FULL ERROR TRACEBACK HERE]
(Note: Copy the entire error message you received, from Traceback... to the final error line, and paste it in this section.)

### Environment info

Platform: Google Colab

datasets version: [Run !pip show datasets in Colab and paste the version here]

huggingface_hub version: [Run !pip show huggingface_hub and paste the version here]

Python version: [Run !python --version and paste the version here]


load_dataset() in 4.0.0 returns different audio samples compared to earlier versions breaking reproducibility

### Describe the bug

In datasets 4.0.0 release, `load_dataset()` returns different audio samples compared to earlier versions, this breaks integration tests that depend on consistent sample data across different environments (first and second envs specified below).

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(24000))
sample= ds[0]["audio"]["array"]
print(sample)

# sample in 3.6.0
[0.00231914 0.00245417 0.00187414 ... 0.00061956 0.00101157 0.00076325]

# sample in 4.0.0
array([0.00238037, 0.00220794, 0.00198703, ..., 0.00057983, 0.00085863,
       0.00115309], dtype=float32)
```

### Expected behavior

The same dataset should load identical samples across versions to maintain reproducibility.

### Environment info

First env:
- datasets version: 3.6.0
- Platform: Windows-10-10.0.26100-SP0
- Python: 3.11.0

Second env:
- datasets  version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python: 3.11.13


Support downloading specific splits in load_dataset

This PR builds on #6832 by @mariosasko.

May close - #4101, #2538

Discussion - https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

---
### Note - This PR is under work and frequent changes will be pushed.


Dataset.to_json consumes excessive memory, appears to not be a streaming operation

### Describe the bug

When exporting a Dataset object to a JSON Lines file using the .to_json(lines=True) method, the process consumes a very large amount of memory. The memory usage is proportional to the size of the entire Dataset object being saved, rather than being a low, constant memory operation.

This behavior is unexpected, as the JSONL format is line-oriented and ideally suited for streaming writes. This issue can easily lead to Out-of-Memory (OOM) errors when exporting large datasets, especially in memory-constrained environments like Docker containers.

<img width="1343" height="329" alt="Image" src="https://github.com/user-attachments/assets/518b4263-ad12-422d-9672-28ffe97240ce" />

### Steps to reproduce the bug

```
import os
from datasets import load_dataset, Dataset
from loguru import logger

# A public dataset to test with
REPO_ID = "adam89/TinyStoriesChinese"
SUBSET = "default"
SPLIT = "train"
NUM_ROWS_TO_LOAD = 10  # Use a reasonably large number to see the memory spike

def run_test():
    """Loads data into memory and then saves it, triggering the memory issue."""
    logger.info("Step 1: Loading data into an in-memory Dataset object...")

    # Create an in-memory Dataset object from a stream
    # This simulates having a processed dataset ready to be saved
    iterable_dataset = load_dataset(REPO_ID, name=SUBSET, split=SPLIT, streaming=True)
    limited_stream = iterable_dataset.take(NUM_ROWS_TO_LOAD)
    in_memory_dataset = Dataset.from_generator(limited_stream.__iter__)

    logger.info(f"Dataset with {len(in_memory_dataset)} rows created in memory.")

    output_path = "./test_output.jsonl"
    logger.info(f"Step 2: Saving the dataset to {output_path} using .to_json()...")
    logger.info("Please monitor memory usage during this step.")

    # This is the step that causes the massive memory allocation
    in_memory_dataset.to_json(output_path, force_ascii=False)

    logger.info("Save operation complete.")
    os.remove(output_path)

if __name__ == "__main__":
    # To see the memory usage clearly, run this script with a memory profiler:
    # python -m memray run your_script_name.py
    # python -m memray tree xxx.bin
    run_test()

```

### Expected behavior

I would expect the .to_json(lines=True) method to be a memory-efficient, streaming operation. The memory usage should remain low and relatively constant, as data is converted and written to the file line-by-line or in small batches. The memory footprint should not be proportional to the total number of rows in the in_memory_dataset.

### Environment info

datasets version:3.6.0
Python version:3.9.18
os:macOS 15.3.1 (arm64)