`shared_datadir` fixture is missing

### Describe the bug

Running the tests for the latest release fails due to missing `shared_datadir` fixture.

### Steps to reproduce the bug

Running `pytest` while building a package for Arch Linux leads to these errors:

```
==================================== ERRORS ====================================
_________ ERROR at setup of test_pdf_feature_encode_example[<lambda>1] _________
[gw44] linux -- Python 3.13.2 /build/python-datasets/src/datasets-3.5.0/test-env/bin/python
file /build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py, line 8
  @require_pdfplumber
  @pytest.mark.parametrize(
      "build_example",
      [
          lambda pdf_path: pdf_path,
          lambda pdf_path: open(pdf_path, "rb").read(),
          lambda pdf_path: {"path": pdf_path},
          lambda pdf_path: {"path": pdf_path, "bytes": None},
          lambda pdf_path: {"path": pdf_path, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"path": None, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"bytes": open(pdf_path, "rb").read()},
      ],
  )
  def test_pdf_feature_encode_example(shared_datadir, build_example):
E       fixture 'shared_datadir' not found
>       available fixtures: _hf_gated_dataset_repo_txt_data, arrow_file, arrow_path, audio_file, bz2_csv_path, bz2_file, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, ci_hfh_hf_hub_url, ci_hub_config, cleanup_repo, csv2_path, csv_path, data_dir_with_hidden_files, dataset, dataset_dict, disable_implicit_token, disable_tqdm_output, doctest_namespace, geoparquet_path, gz_file, hf_api, hf_gated_dataset_repo_txt_data, hf_private_dataset_repo_txt_data, hf_private_dataset_repo_txt_data_, hf_private_dataset_repo_zipped_img_data, hf_private_dataset_repo_zipped_img_data_, hf_private_dataset_repo_zipped_txt_data, hf_private_dataset_repo_zipped_txt_data_, hf_token, image_file, json_dict_of_lists_path, json_list_of_dicts_path, jsonl2_path, jsonl_312_path, jsonl_gz_path, jsonl_path, jsonl_str_path, lz4_file, mock_fsspec, mockfs, monkeypatch, parquet_path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, set_ci_hub_access_token, set_sqlalchemy_silence_uber_warning, set_test_cache_config, set_update_download_counts_to_false, seven_zip_file, sqlite_path, tar_file, tar_jsonl_path, tar_nested_jsonl_path, temporary_repo, tensor_file, testrun_uid, text2_path, text_dir, text_dir_with_unsupported_extension, text_file, text_file_content, text_gz_path, text_path, text_path_with_unicode_new_lines, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, tmpfs, worker_id, xml_file, xz_file, zero_time_out_for_remote_code, zip_csv_path, zip_csv_with_dir_path, zip_file, zip_image_path, zip_jsonl_path, zip_jsonl_with_dir_path, zip_nested_jsonl_path, zip_text_path, zip_text_with_dir_path, zip_unsupported_ext_path, zip_uppercase_csv_path, zstd_file
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py:8
_________ ERROR at setup of test_pdf_feature_encode_example[<lambda>2] _________
[gw44] linux -- Python 3.13.2 /build/python-datasets/src/datasets-3.5.0/test-env/bin/python
file /build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py, line 8
  @require_pdfplumber
  @pytest.mark.parametrize(
      "build_example",
      [
          lambda pdf_path: pdf_path,
          lambda pdf_path: open(pdf_path, "rb").read(),
          lambda pdf_path: {"path": pdf_path},
          lambda pdf_path: {"path": pdf_path, "bytes": None},
          lambda pdf_path: {"path": pdf_path, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"path": None, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"bytes": open(pdf_path, "rb").read()},
      ],
  )
  def test_pdf_feature_encode_example(shared_datadir, build_example):
E       fixture 'shared_datadir' not found
>       available fixtures: _hf_gated_dataset_repo_txt_data, arrow_file, arrow_path, audio_file, bz2_csv_path, bz2_file, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, ci_hfh_hf_hub_url, ci_hub_config, cleanup_repo, csv2_path, csv_path, data_dir_with_hidden_files, dataset, dataset_dict, disable_implicit_token, disable_tqdm_output, doctest_namespace, geoparquet_path, gz_file, hf_api, hf_gated_dataset_repo_txt_data, hf_private_dataset_repo_txt_data, hf_private_dataset_repo_txt_data_, hf_private_dataset_repo_zipped_img_data, hf_private_dataset_repo_zipped_img_data_, hf_private_dataset_repo_zipped_txt_data, hf_private_dataset_repo_zipped_txt_data_, hf_token, image_file, json_dict_of_lists_path, json_list_of_dicts_path, jsonl2_path, jsonl_312_path, jsonl_gz_path, jsonl_path, jsonl_str_path, lz4_file, mock_fsspec, mockfs, monkeypatch, parquet_path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, set_ci_hub_access_token, set_sqlalchemy_silence_uber_warning, set_test_cache_config, set_update_download_counts_to_false, seven_zip_file, sqlite_path, tar_file, tar_jsonl_path, tar_nested_jsonl_path, temporary_repo, tensor_file, testrun_uid, text2_path, text_dir, text_dir_with_unsupported_extension, text_file, text_file_content, text_gz_path, text_path, text_path_with_unicode_new_lines, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, tmpfs, worker_id, xml_file, xz_file, zero_time_out_for_remote_code, zip_csv_path, zip_csv_with_dir_path, zip_file, zip_image_path, zip_jsonl_path, zip_jsonl_with_dir_path, zip_nested_jsonl_path, zip_text_path, zip_text_with_dir_path, zip_unsupported_ext_path, zip_uppercase_csv_path, zstd_file
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py:8
_________ ERROR at setup of test_pdf_feature_encode_example[<lambda>3] _________
[gw44] linux -- Python 3.13.2 /build/python-datasets/src/datasets-3.5.0/test-env/bin/python
file /build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py, line 8
  @require_pdfplumber
  @pytest.mark.parametrize(
      "build_example",
      [
          lambda pdf_path: pdf_path,
          lambda pdf_path: open(pdf_path, "rb").read(),
          lambda pdf_path: {"path": pdf_path},
          lambda pdf_path: {"path": pdf_path, "bytes": None},
          lambda pdf_path: {"path": pdf_path, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"path": None, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"bytes": open(pdf_path, "rb").read()},
      ],
  )
  def test_pdf_feature_encode_example(shared_datadir, build_example):
E       fixture 'shared_datadir' not found
>       available fixtures: _hf_gated_dataset_repo_txt_data, arrow_file, arrow_path, audio_file, bz2_csv_path, bz2_file, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, ci_hfh_hf_hub_url, ci_hub_config, cleanup_repo, csv2_path, csv_path, data_dir_with_hidden_files, dataset, dataset_dict, disable_implicit_token, disable_tqdm_output, doctest_namespace, geoparquet_path, gz_file, hf_api, hf_gated_dataset_repo_txt_data, hf_private_dataset_repo_txt_data, hf_private_dataset_repo_txt_data_, hf_private_dataset_repo_zipped_img_data, hf_private_dataset_repo_zipped_img_data_, hf_private_dataset_repo_zipped_txt_data, hf_private_dataset_repo_zipped_txt_data_, hf_token, image_file, json_dict_of_lists_path, json_list_of_dicts_path, jsonl2_path, jsonl_312_path, jsonl_gz_path, jsonl_path, jsonl_str_path, lz4_file, mock_fsspec, mockfs, monkeypatch, parquet_path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, set_ci_hub_access_token, set_sqlalchemy_silence_uber_warning, set_test_cache_config, set_update_download_counts_to_false, seven_zip_file, sqlite_path, tar_file, tar_jsonl_path, tar_nested_jsonl_path, temporary_repo, tensor_file, testrun_uid, text2_path, text_dir, text_dir_with_unsupported_extension, text_file, text_file_content, text_gz_path, text_path, text_path_with_unicode_new_lines, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, tmpfs, worker_id, xml_file, xz_file, zero_time_out_for_remote_code, zip_csv_path, zip_csv_with_dir_path, zip_file, zip_image_path, zip_jsonl_path, zip_jsonl_with_dir_path, zip_nested_jsonl_path, zip_text_path, zip_text_with_dir_path, zip_unsupported_ext_path, zip_uppercase_csv_path, zstd_file
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py:8
_________ ERROR at setup of test_pdf_feature_encode_example[<lambda>4] _________
[gw44] linux -- Python 3.13.2 /build/python-datasets/src/datasets-3.5.0/test-env/bin/python
file /build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py, line 8
  @require_pdfplumber
  @pytest.mark.parametrize(
      "build_example",
      [
          lambda pdf_path: pdf_path,
          lambda pdf_path: open(pdf_path, "rb").read(),
          lambda pdf_path: {"path": pdf_path},
          lambda pdf_path: {"path": pdf_path, "bytes": None},
          lambda pdf_path: {"path": pdf_path, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"path": None, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"bytes": open(pdf_path, "rb").read()},
      ],
  )
  def test_pdf_feature_encode_example(shared_datadir, build_example):
E       fixture 'shared_datadir' not found
>       available fixtures: _hf_gated_dataset_repo_txt_data, arrow_file, arrow_path, audio_file, bz2_csv_path, bz2_file, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, ci_hfh_hf_hub_url, ci_hub_config, cleanup_repo, csv2_path, csv_path, data_dir_with_hidden_files, dataset, dataset_dict, disable_implicit_token, disable_tqdm_output, doctest_namespace, geoparquet_path, gz_file, hf_api, hf_gated_dataset_repo_txt_data, hf_private_dataset_repo_txt_data, hf_private_dataset_repo_txt_data_, hf_private_dataset_repo_zipped_img_data, hf_private_dataset_repo_zipped_img_data_, hf_private_dataset_repo_zipped_txt_data, hf_private_dataset_repo_zipped_txt_data_, hf_token, image_file, json_dict_of_lists_path, json_list_of_dicts_path, jsonl2_path, jsonl_312_path, jsonl_gz_path, jsonl_path, jsonl_str_path, lz4_file, mock_fsspec, mockfs, monkeypatch, parquet_path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, set_ci_hub_access_token, set_sqlalchemy_silence_uber_warning, set_test_cache_config, set_update_download_counts_to_false, seven_zip_file, sqlite_path, tar_file, tar_jsonl_path, tar_nested_jsonl_path, temporary_repo, tensor_file, testrun_uid, text2_path, text_dir, text_dir_with_unsupported_extension, text_file, text_file_content, text_gz_path, text_path, text_path_with_unicode_new_lines, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, tmpfs, worker_id, xml_file, xz_file, zero_time_out_for_remote_code, zip_csv_path, zip_csv_with_dir_path, zip_file, zip_image_path, zip_jsonl_path, zip_jsonl_with_dir_path, zip_nested_jsonl_path, zip_text_path, zip_text_with_dir_path, zip_unsupported_ext_path, zip_uppercase_csv_path, zstd_file
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py:8
_________ ERROR at setup of test_pdf_feature_encode_example[<lambda>5] _________
[gw44] linux -- Python 3.13.2 /build/python-datasets/src/datasets-3.5.0/test-env/bin/python
file /build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py, line 8
  @require_pdfplumber
  @pytest.mark.parametrize(
      "build_example",
      [
          lambda pdf_path: pdf_path,
          lambda pdf_path: open(pdf_path, "rb").read(),
          lambda pdf_path: {"path": pdf_path},
          lambda pdf_path: {"path": pdf_path, "bytes": None},
          lambda pdf_path: {"path": pdf_path, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"path": None, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"bytes": open(pdf_path, "rb").read()},
      ],
  )
  def test_pdf_feature_encode_example(shared_datadir, build_example):
E       fixture 'shared_datadir' not found
>       available fixtures: _hf_gated_dataset_repo_txt_data, arrow_file, arrow_path, audio_file, bz2_csv_path, bz2_file, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, ci_hfh_hf_hub_url, ci_hub_config, cleanup_repo, csv2_path, csv_path, data_dir_with_hidden_files, dataset, dataset_dict, disable_implicit_token, disable_tqdm_output, doctest_namespace, geoparquet_path, gz_file, hf_api, hf_gated_dataset_repo_txt_data, hf_private_dataset_repo_txt_data, hf_private_dataset_repo_txt_data_, hf_private_dataset_repo_zipped_img_data, hf_private_dataset_repo_zipped_img_data_, hf_private_dataset_repo_zipped_txt_data, hf_private_dataset_repo_zipped_txt_data_, hf_token, image_file, json_dict_of_lists_path, json_list_of_dicts_path, jsonl2_path, jsonl_312_path, jsonl_gz_path, jsonl_path, jsonl_str_path, lz4_file, mock_fsspec, mockfs, monkeypatch, parquet_path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, set_ci_hub_access_token, set_sqlalchemy_silence_uber_warning, set_test_cache_config, set_update_download_counts_to_false, seven_zip_file, sqlite_path, tar_file, tar_jsonl_path, tar_nested_jsonl_path, temporary_repo, tensor_file, testrun_uid, text2_path, text_dir, text_dir_with_unsupported_extension, text_file, text_file_content, text_gz_path, text_path, text_path_with_unicode_new_lines, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, tmpfs, worker_id, xml_file, xz_file, zero_time_out_for_remote_code, zip_csv_path, zip_csv_with_dir_path, zip_file, zip_image_path, zip_jsonl_path, zip_jsonl_with_dir_path, zip_nested_jsonl_path, zip_text_path, zip_text_with_dir_path, zip_unsupported_ext_path, zip_uppercase_csv_path, zstd_file
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py:8
_________ ERROR at setup of test_pdf_feature_encode_example[<lambda>6] _________
[gw44] linux -- Python 3.13.2 /build/python-datasets/src/datasets-3.5.0/test-env/bin/python
file /build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py, line 8
  @require_pdfplumber
  @pytest.mark.parametrize(
      "build_example",
      [
          lambda pdf_path: pdf_path,
          lambda pdf_path: open(pdf_path, "rb").read(),
          lambda pdf_path: {"path": pdf_path},
          lambda pdf_path: {"path": pdf_path, "bytes": None},
          lambda pdf_path: {"path": pdf_path, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"path": None, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"bytes": open(pdf_path, "rb").read()},
      ],
  )
  def test_pdf_feature_encode_example(shared_datadir, build_example):
E       fixture 'shared_datadir' not found
>       available fixtures: _hf_gated_dataset_repo_txt_data, arrow_file, arrow_path, audio_file, bz2_csv_path, bz2_file, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, ci_hfh_hf_hub_url, ci_hub_config, cleanup_repo, csv2_path, csv_path, data_dir_with_hidden_files, dataset, dataset_dict, disable_implicit_token, disable_tqdm_output, doctest_namespace, geoparquet_path, gz_file, hf_api, hf_gated_dataset_repo_txt_data, hf_private_dataset_repo_txt_data, hf_private_dataset_repo_txt_data_, hf_private_dataset_repo_zipped_img_data, hf_private_dataset_repo_zipped_img_data_, hf_private_dataset_repo_zipped_txt_data, hf_private_dataset_repo_zipped_txt_data_, hf_token, image_file, json_dict_of_lists_path, json_list_of_dicts_path, jsonl2_path, jsonl_312_path, jsonl_gz_path, jsonl_path, jsonl_str_path, lz4_file, mock_fsspec, mockfs, monkeypatch, parquet_path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, set_ci_hub_access_token, set_sqlalchemy_silence_uber_warning, set_test_cache_config, set_update_download_counts_to_false, seven_zip_file, sqlite_path, tar_file, tar_jsonl_path, tar_nested_jsonl_path, temporary_repo, tensor_file, testrun_uid, text2_path, text_dir, text_dir_with_unsupported_extension, text_file, text_file_content, text_gz_path, text_path, text_path_with_unicode_new_lines, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, tmpfs, worker_id, xml_file, xz_file, zero_time_out_for_remote_code, zip_csv_path, zip_csv_with_dir_path, zip_file, zip_image_path, zip_jsonl_path, zip_jsonl_with_dir_path, zip_nested_jsonl_path, zip_text_path, zip_text_with_dir_path, zip_unsupported_ext_path, zip_uppercase_csv_path, zstd_file
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py:8
_______________ ERROR at setup of test_dataset_with_pdf_feature ________________
[gw44] linux -- Python 3.13.2 /build/python-datasets/src/datasets-3.5.0/test-env/bin/python
file /build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py, line 34
  @require_pdfplumber
  def test_dataset_with_pdf_feature(shared_datadir):
E       fixture 'shared_datadir' not found
>       available fixtures: _hf_gated_dataset_repo_txt_data, arrow_file, arrow_path, audio_file, bz2_csv_path, bz2_file, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, ci_hfh_hf_hub_url, ci_hub_config, cleanup_repo, csv2_path, csv_path, data_dir_with_hidden_files, dataset, dataset_dict, disable_implicit_token, disable_tqdm_output, doctest_namespace, geoparquet_path, gz_file, hf_api, hf_gated_dataset_repo_txt_data, hf_private_dataset_repo_txt_data, hf_private_dataset_repo_txt_data_, hf_private_dataset_repo_zipped_img_data, hf_private_dataset_repo_zipped_img_data_, hf_private_dataset_repo_zipped_txt_data, hf_private_dataset_repo_zipped_txt_data_, hf_token, image_file, json_dict_of_lists_path, json_list_of_dicts_path, jsonl2_path, jsonl_312_path, jsonl_gz_path, jsonl_path, jsonl_str_path, lz4_file, mock_fsspec, mockfs, monkeypatch, parquet_path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, set_ci_hub_access_token, set_sqlalchemy_silence_uber_warning, set_test_cache_config, set_update_download_counts_to_false, seven_zip_file, sqlite_path, tar_file, tar_jsonl_path, tar_nested_jsonl_path, temporary_repo, tensor_file, testrun_uid, text2_path, text_dir, text_dir_with_unsupported_extension, text_file, text_file_content, text_gz_path, text_path, text_path_with_unicode_new_lines, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, tmpfs, worker_id, xml_file, xz_file, zero_time_out_for_remote_code, zip_csv_path, zip_csv_with_dir_path, zip_file, zip_image_path, zip_jsonl_path, zip_jsonl_with_dir_path, zip_nested_jsonl_path, zip_text_path, zip_text_with_dir_path, zip_unsupported_ext_path, zip_uppercase_csv_path, zstd_file
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py:34
_________ ERROR at setup of test_pdf_feature_encode_example[<lambda>0] _________
[gw46] linux -- Python 3.13.2 /build/python-datasets/src/datasets-3.5.0/test-env/bin/python
file /build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py, line 8
  @require_pdfplumber
  @pytest.mark.parametrize(
      "build_example",
      [
          lambda pdf_path: pdf_path,
          lambda pdf_path: open(pdf_path, "rb").read(),
          lambda pdf_path: {"path": pdf_path},
          lambda pdf_path: {"path": pdf_path, "bytes": None},
          lambda pdf_path: {"path": pdf_path, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"path": None, "bytes": open(pdf_path, "rb").read()},
          lambda pdf_path: {"bytes": open(pdf_path, "rb").read()},
      ],
  )
  def test_pdf_feature_encode_example(shared_datadir, build_example):
E       fixture 'shared_datadir' not found
>       available fixtures: _hf_gated_dataset_repo_txt_data, arrow_file, arrow_path, audio_file, bz2_csv_path, bz2_file, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, ci_hfh_hf_hub_url, ci_hub_config, cleanup_repo, csv2_path, csv_path, data_dir_with_hidden_files, dataset, dataset_dict, disable_implicit_token, disable_tqdm_output, doctest_namespace, geoparquet_path, gz_file, hf_api, hf_gated_dataset_repo_txt_data, hf_private_dataset_repo_txt_data, hf_private_dataset_repo_txt_data_, hf_private_dataset_repo_zipped_img_data, hf_private_dataset_repo_zipped_img_data_, hf_private_dataset_repo_zipped_txt_data, hf_private_dataset_repo_zipped_txt_data_, hf_token, image_file, json_dict_of_lists_path, json_list_of_dicts_path, jsonl2_path, jsonl_312_path, jsonl_gz_path, jsonl_path, jsonl_str_path, lz4_file, mock_fsspec, mockfs, monkeypatch, parquet_path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, set_ci_hub_access_token, set_sqlalchemy_silence_uber_warning, set_test_cache_config, set_update_download_counts_to_false, seven_zip_file, sqlite_path, tar_file, tar_jsonl_path, tar_nested_jsonl_path, temporary_repo, tensor_file, testrun_uid, text2_path, text_dir, text_dir_with_unsupported_extension, text_file, text_file_content, text_gz_path, text_path, text_path_with_unicode_new_lines, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, tmpfs, worker_id, xml_file, xz_file, zero_time_out_for_remote_code, zip_csv_path, zip_csv_with_dir_path, zip_file, zip_image_path, zip_jsonl_path, zip_jsonl_with_dir_path, zip_nested_jsonl_path, zip_text_path, zip_text_with_dir_path, zip_unsupported_ext_path, zip_uppercase_csv_path, zstd_file
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-datasets/src/datasets-3.5.0/tests/features/test_pdf.py:8
```

### Expected behavior

All fixtures used in tests should be available.

### Environment info

Arch Linux build system, building the [python-datasets](https://gitlab.archlinux.org/archlinux/packaging/packages/python-datasets) package.

There are actually [many deselected tests](https://gitlab.archlinux.org/archlinux/packaging/packages/python-datasets/-/blob/6f97957f0c326cc7b3da6b7f12326305bcaef374/PKGBUILD#L66-148) which were failing on previous releases, but these errors popped up in 3.5.0.


set dev version


release: 3.5.0


Support skip_trying_type

This PR addresses Issue #7472

cc: @lhoestq


Implement capability to restore non-nullability in Features

This PR attempts to keep track of non_nullable pyarrow fields when converting a `pa.Schema` to `Features`. At the same time, when outputting the `arrow_schema`, the original non-nullable fields are restored. This allows for more consistent behavior and avoids breaking behavior as illustrated in #7479.

I am by no means a pyarrow expert so some logic in `find_non_nullable_fields` may not perfect. Not sure if more logic (type checks) are needed for deep-checking a given schema. Maybe there are other pyarrow structures that need to be covered?

Tests are added, but again, these may not have sufficient coverage in terms of pyarrow structure types.

closes #7479


deal with python `10_000` legal number in slice syntax

enhancement

New feature or request

### Feature request

```
In [6]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1000]")

In [7]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1_000]")
[dozens of frames skipped]
File /usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py:444, in _str_to_read_instruction(spec)
    442 res = _SUB_SPEC_RE.match(spec)
    443 if not res:
--> 444     raise ValueError(f"Unrecognized instruction format: {spec}")
ValueError: Unrecognized instruction format: train_sft[:1_000]
```

It took me a while to understand what the problem was. But apparently `pyarrow` doesn't allow python numbers that may include `_` as in `1_000`. The `_` aids readability since `10_000_000` vs `10000000` is obviously easier to grasp of what the actual number is.

Feature request:

ideally `datasets` being a python module will do the right thing and convert python numbers into whatever pyarrow supports - in this case stripping `_`s.

Second best it'd err and tell the user that using numbers with `_` in split slices is not acceptible, so that the user won't have to deal with a huge pyarrow assert they know nothing about.

Thank you!


HF_DATASETS_CACHE ignored?

### Describe the bug

I'm struggling to get things to respect HF_DATASETS_CACHE.

Rationale: I'm on a system that uses NFS for homedir, so downloading to NFS is expensive, slow, and wastes valuable quota compared to local disk. Instead, it seems to rely mostly on HF_HUB_CACHE.

Current version: 3.2.1dev. In the process of testing 3.4.0

### Steps to reproduce the bug

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]

dump.py:
```python
from datasets import load_dataset
dataset = load_dataset("HuggingFaceFW/fineweb", name="sample-100BT", split="train")
```

Repro steps
```bash
# ensure no cache
$ mv ~/.cache/huggingface ~/.cache/huggingface.bak

$ export HF_DATASETS_CACHE=/tmp/roller/datasets
$ rm -rf ${HF_DATASETS_CACHE}
$ env | grep HF | grep -v TOKEN
HF_DATASETS_CACHE=/tmp/roller/datasets

$ python dump.py
# (omitted for brevity)

# (while downloading)
$ du -hcs ~/.cache/huggingface/hub
18G     hub
18G     total

# (after downloading)
$ du -hcs ~/.cache/huggingface/hub
```

It's a shame because datasets supports s3 (which I could really use right now) but hub does not.

### Expected behavior

* ~/.cache/huggingface/hub stays empty
* /tmp/roller/datasets becomes full of stuff

### Environment info

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]


Features.from_arrow_schema is destructive

### Describe the bug

I came across this, perhaps niche, bug where `Features` does not/cannot account for pyarrow's `nullable=False` option in Fields. Interestingly, I found that in regular "flat" fields this does not necessarily lead to conflicts, but when a non-nullable field is in a struct, an incompatibility arises.

It's not easy to explain in words, so the minimal example below should help I hope.

Note that I suggest a solution in the comments in the code, simply allowing `Dataset.to_parquet` to allow for a `schema` argument which, when provided, will override the default ds.features.arrow_schema.

### Steps to reproduce the bug

```python
import os
from datasets import Dataset, Features

import pyarrow as pa
import pyarrow.parquet as pq

# HF datasets is destructive when you call Features.from_arrow_schema(schema) on a schema
# because it will not account for nullable and non-nullable fields in structs (it will always allow nullable)
# Reloading the same dataset with the original schema will raise an error because the schema is not the same anymore
non_nullable_schema = pa.schema(
    [
        pa.field("text", pa.string(), nullable=False),
        pa.field("meta",
            pa.struct(
                [
                    pa.field("date", pa.list_(pa.string()), nullable=False),
                ],
            ),
        ),

    ]
)
print("ORIGINAL SCHEMA")
print(non_nullable_schema)
print()

feats = Features.from_arrow_schema(non_nullable_schema)

print("FEATUR-IZED SCHEMA (nullable-restrictions are gone)")
print(feats.arrow_schema)
print()

ds = Dataset.from_dict(
    {
        "text": ["a", "b", "c"],
        "meta": [{"date": ["2021-01-01"]}, {"date": ["2021-01-02"]}, {"date": ["2021-01-03"]}],
    },
    features=feats,
)

fname = "tmp.parquet"

# This is not possible: TypeError: pyarrow.parquet.core.ParquetWriter() got multiple values for keyword argument 'schema'
# Though I believe this would be the easiest fix: allow schema to be passed to to_parquet and overwrite the schema in the dataset
# ds.to_parquet(fname, schema=non_nullable_schema)

ds.to_parquet(fname)

try:
    _ = pq.read_table(fname, schema=non_nullable_schema)
finally:
    os.unlink(fname)
```

### Expected behavior

- Non-destructive behavior when converting an arrow schema to Features; or
- the ability to override the default arrow schema with a custom one

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.20.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.10
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0


update fsspec 2025.3.0

It appears there have been two releases of fsspec since this dependency was last updated, it would be great if Datasets could be updated so that it didn't hold back the usage of newer fsspec versions in consuming projects.

PR based on https://github.com/huggingface/datasets/pull/7352


What is the canonical way to compress a Dataset?

Given that Arrow is the preferred backend for a Dataset, what is a user supposed to do if they want concurrent reads, concurrent writes AND on-disk compression for a larger dataset?

Parquet would be the obvious answer except that there is no native support for writing sharded, parquet datasets concurrently [[1](https://github.com/huggingface/datasets/issues/7047)].

Am I missing something?

And if so, why is this not the standard/default way that `Dataset`'s work as they do in Xarray, Ray Data, Composer, etc.?


Priotitize json

`datasets` should load the JSON data in https://huggingface.co/datasets/facebook/natural_reasoning, not the PDF


IterableDataset's state_dict shard_example_idx is always equal to the number of samples in a shard

### Describe the bug

I've noticed a strange behaviour with Iterable state_dict: the value of shard_example_idx is always equal to the amount of samples in a shard.

### Steps to reproduce the bug

I am reusing the example from the doc
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=1)
state_dict = None
# Iterate through the dataset and print examples
for idx, example in enumerate(ds):
    print(example)
    if idx == 2:
        state_dict = ds.state_dict()
        print("checkpoint")
        break
print(state_dict)
```
Returns:
```
{'a': 0}
{'a': 1}
checkpoint
{'examples_iterable': {'shard_idx': 0, 'shard_example_idx': 6, 'type': 'ArrowExamplesIterable'}, 'epoch': 0}
```

### Expected behavior

shard_example_idx should be 2 instead of 6
If we run with num_shards=2, then shard_example_idx is 3 instead of 2 and so on.

### Environment info

- `datasets` version: 3.4.1
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.9
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Remove conditions for Python < 3.9

This PR remove conditions for Python < 3.9.


Webdataset data format problem

### Describe the bug

Please see https://huggingface.co/datasets/ejschwartz/idioms/discussions/1

Error code: FileFormatMismatchBetweenSplitsError

All three splits, train, test, and validation, use webdataset. But only the train split has more than one file. How can I force the other two splits to also be interpreted as being the webdataset format?  (I don't think there is currently a way, but happy to be told that I am wrong.)

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("ejschwartz/idioms")

### Expected behavior

The dataset loads.  Alternatively, there is a YAML syntax for manually specifying the format.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0


Label casting during `map` process is canceled after the `map` process

### Describe the bug

When preprocessing a multi-label dataset, I introduced a step to convert int labels to float labels as [BCEWithLogitsLoss](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html) expects float labels and forward function of models in transformers package internally use `BCEWithLogitsLoss`

However, the casting was canceled after `.map` process and the label values still use int values, which leads to an error
```
  File "/home/yoshitomo/anaconda3/envs/torchdistill/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1711, in forward
    loss = loss_fct(logits, labels)
  File "/home/yoshitomo/anaconda3/envs/torchdistill/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/yoshitomo/anaconda3/envs/torchdistill/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/yoshitomo/anaconda3/envs/torchdistill/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 819, in forward
    return F.binary_cross_entropy_with_logits(
  File "/home/yoshitomo/anaconda3/envs/torchdistill/lib/python3.10/site-packages/torch/nn/functional.py", line 3628, in binary_cross_entropy_with_logits
    return torch.binary_cross_entropy_with_logits(
RuntimeError: result type Float can't be cast to the desired output type Long
```

This seems like happening only when the original labels are int values (see examples below)

### Steps to reproduce the bug

If the original dataset uses a list of int labels, it will cancel the int->float casting

```python
from datasets import Dataset

data = {
    'text': ['text1', 'text2', 'text3', 'text4'],
    'labels': [[0, 1, 2], [3], [3, 4], [3]]
}

dataset = Dataset.from_dict(data)

label_set = set([label for labels in data['labels'] for label in labels])
label2idx = {label: idx for idx, label in enumerate(sorted(label_set))}

def multi_labels_to_ids(labels):
    ids = [0.0] * len(label2idx)
    for label in labels:
        ids[label2idx[label]] = 1.0
    return ids

def preprocess(examples):
    result = {'sentence': [[0, 3, 4] for _ in range(len(examples['labels']))]}
    print('"labels" are int', examples['labels'])
    result['labels'] = [multi_labels_to_ids(l) for l in examples['labels']]
    print('"labels" were converted to multi-label format with float values', result['labels'])
    return result

preprocessed_dataset = dataset.map(preprocess, batched=True, remove_columns=['labels', 'text'])
print(preprocessed_dataset[0]['labels'])
# Output: "[1, 1, 1, 0, 0]"
# Expected: "[1.0, 1.0, 1.0, 0.0, 0.0]"
```

If the original dataset uses non-int labels, it works as expected.

```python
from datasets import Dataset

data = {
    'text': ['text1', 'text2', 'text3', 'text4'],
    'labels': [['label1', 'label2', 'label3'], ['label4'], ['label4', 'label5'], ['label4']]
}

dataset = Dataset.from_dict(data)

label_set = set([label for labels in data['labels'] for label in labels])
label2idx = {label: idx for idx, label in enumerate(sorted(label_set))}

def multi_labels_to_ids(labels):
    ids = [0.0] * len(label2idx)
    for label in labels:
        ids[label2idx[label]] = 1.0
    return ids

def preprocess(examples):
    result = {'sentence': [[0, 3, 4] for _ in range(len(examples['labels']))]}
    print('"labels" are int', examples['labels'])
    result['labels'] = [multi_labels_to_ids(l) for l in examples['labels']]
    print('"labels" were converted to multi-label format with float values', result['labels'])
    return result

preprocessed_dataset = dataset.map(preprocess, batched=True, remove_columns=['labels', 'text'])
print(preprocessed_dataset[0]['labels'])
# Output: "[1.0, 1.0, 1.0, 0.0, 0.0]"
# Expected: "[1.0, 1.0, 1.0, 0.0, 0.0]"
```

Note that the only difference between these two examples is
>    'labels': [[0, 1, 2], [3], [3, 4], [3]]

v.s

>    'labels': [['label1', 'label2', 'label3'], ['label4'], ['label4', 'label5'], ['label4']]

### Expected behavior

Even if the original dataset uses a list of int labels, the int->float casting during `.map` process should not be canceled as shown in the above example

### Environment info

OS Ubuntu 22.04 LTS
Python 3.10.11
datasets v3.4.1


Adding argument to `_get_data_files_patterns`

New feature or request

### Feature request

How about adding if the user already know about the pattern?

https://github.com/huggingface/datasets/blob/a256b85cbc67aa3f0e75d32d6586afc507cf535b/src/datasets/data_files.py#L252

### Motivation

While using this load_dataset people might use 10M of images for the local files.
However, due to searching all the appropriate file pattern in fsspec, purely searching this pattern takes more than 10 hours (real use-case).

### Your contribution

Yeah I can make this happen if this seems valid. @lhoestq WDYT?
such like
```
def _get_data_files_patterns(pattern_resolver: Callable[[str], list[str]], patterns: PATTERNS) -> dict[str, list[str]]:
```


Is it possible to shard a single-sharded IterableDataset?

I thought https://github.com/huggingface/datasets/pull/7252 might be applicable but looking at it maybe not.

Say we have a process, eg. a database query, that can return data in slightly different order each time. So, the initial query needs to be run by a single thread (not to mention running multiple times incurs more cost too). But the results are also big enough that we don't want to materialize it entirely and instead stream it with an IterableDataset.

But after we have the results we want to split it up across workers to parallelize processing.

Is something like this possible to do?

Here's a failed attempt. The end result should be that each of the shards has unique data, but unfortunately with this attempt the generator gets run once in each shard and the results end up with duplicates...

```
import random
import datasets

def gen():
  print('RUNNING GENERATOR!')
  items = list(range(10))
  random.shuffle(items)
  yield from items

ds = datasets.IterableDataset.from_generator(gen)

print('dataset contents:')
for item in ds:
  print(item)
print()

print('dataset contents (2):')
for item in ds:
  print(item)
print()

num_shards = 3

def sharded(shard_id):
  for i, example in enumerate(ds):
    if i % num_shards in shard_id:
      yield example

ds1 = datasets.IterableDataset.from_generator(
  sharded, gen_kwargs={'shard_id': list(range(num_shards))}
)

for shard in range(num_shards):
  print('shard', shard)
  for item in ds1.shard(num_shards, shard):
    print(item)
```


Custom split name with the web interface

### Describe the bug

According the doc here: https://huggingface.co/docs/hub/datasets-file-names-and-splits#custom-split-name
it should infer the split name from the subdir of data or the beg of the name of the files in data.
When doing this manually through web upload it does not work. it uses "train" as a unique split.
example: https://huggingface.co/datasets/eole-nlp/estimator_chatml

### Steps to reproduce the bug

follow the link above

### Expected behavior

there should be two splits "mlqe" and "1720_da"

### Environment info

website


function `load_dataset`  can't solve folder path with regex characters like "[]"

### Describe the bug

When using the `load_dataset` function with a folder path containing regex special characters (such as "[]"), the issue occurs due to how the path is handled in the `resolve_pattern` function. This function passes the unprocessed path directly to `AbstractFileSystem.glob`, which supports regular expressions. As a result, the globbing mechanism interprets these characters as regex patterns, leading to a traversal of the entire disk partition instead of confining the search to the intended directory.

### Steps to reproduce the bug

just create a folder like `E:\[D_DATA]\koch_test`, then `load_dataset("parquet", data_dir="E:\[D_DATA]\\test", split="train")`
it will keep searching the whole disk.

I add two `print` in `glob` and `resolve_pattern` to see the path

### Expected behavior

it should load the dataset as in normal folders

### Environment info

- `datasets` version: 3.3.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


load_dataset with streaming hangs on parquet datasets

### Describe the bug

When I try to load a dataset with parquet files (e.g. "bigcode/the-stack") the dataset loads, but python interpreter can't exit and hangs

### Steps to reproduce the bug

```python3
import datasets

print('Start')
dataset = datasets.load_dataset("bigcode/the-stack", data_dir="data/yaml", streaming=True, split="train")
it = iter(dataset)
next(it)
print('Finish')
```

The program prints finish but doesn't exit and hangs indefinitely.
I tried this on two different machines and several datasets.

### Expected behavior

The program exits successfully

### Environment info

datasets==3.4.1
Python 3.12.9.
MacOS and Ubuntu Linux


Fix local pdf loading

fir this error when accessing a local pdf

```
File ~/.pyenv/versions/3.12.2/envs/hf-datasets/lib/python3.12/site-packages/pdfminer/psparser.py:220, in PSBaseParser.seek(self, pos)
    218 """Seeks the parser to the given position."""
    219 log.debug("seek: %r", pos)
--> 220 self.fp.seek(pos)
    221 # reset the status for nextline()
    222 self.bufpos = pos

ValueError: seek of closed file
```


Minor fix for metadata files in extension counter


Adds EXR format to store depth images in float32

This PR adds the EXR feature to store depth images (or can be normals, etc) in float32.
It relies on [openexr_numpy](https://github.com/martinResearch/openexr_numpy/tree/main) to manipulate EXR images.


set dev version


List of images behave differently on IterableDataset and Dataset

### Describe the bug

This code:

```python
def train_iterable_gen():
        images = np.array(load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg").resize((128, 128)))
        yield {
            "images": np.expand_dims(images, axis=0),
            "messages": [
                {
                    "role": "user",
                    "content": [{"type": "image", "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" }]
                },
                {
                    "role": "assistant",
                    "content": [{"type": "text", "text": "duck" }]
                }
            ]
        }
    train_ds = Dataset.from_generator(train_iterable_gen,
                                      features=Features({
                                           'images': [datasets.Image(mode=None, decode=True, id=None)],
                                           'messages': [{'content': [{'text': datasets.Value(dtype='string', id=None), 'type': datasets.Value(dtype='string', id=None) }], 'role': datasets.Value(dtype='string', id=None)}]
                                           } )
                                           )
```

works as I'd expect; if I iterate the dataset then the `images` column returns a `List[PIL.Image.Image]`, i.e. `'images': [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=128x128 at 0x77EFB7EF4680>]`.

But if I change `Dataset` to `IterableDataset`, the `images` column changes into `'images': [{'path': None, 'bytes': ..]`

### Steps to reproduce the bug

The code above +

```python
def load_image(url):
        response = requests.get(url)
        image = Image.open(io.BytesIO(response.content))
        return image
```

I'm feeding it to SFTTrainer

### Expected behavior

Dataset and IterableDataset would behave the same

### Environment info

```yaml
requires-python = ">=3.12"
dependencies = [
    "av>=14.1.0",
    "boto3>=1.36.7",
    "datasets>=3.3.2",
    "docker>=7.1.0",
    "google-cloud-storage>=2.19.0",
    "grpcio>=1.70.0",
    "grpcio-tools>=1.70.0",
    "moviepy>=2.1.2",
    "open-clip-torch>=2.31.0",
    "opencv-python>=4.11.0.86; sys_platform == 'darwin'",
    "opencv-python-headless>=4.11.0.86; sys_platform == 'linux'",
    "pandas>=2.2.3",
    "pillow>=10.4.0",
    "plotly>=6.0.0",
    "py-spy>=0.4.0",
    "pydantic>=2.10.6",
    "pydantic-settings>=2.7.1",
    "pymysql>=1.1.1",
    "ray[data,default,serve,train,tune]>=2.43.0",
    "torch>=2.6.0",
    "torchmetrics>=1.6.1",
    "torchvision>=0.21.0",
    "transformers[torch]@git+https://github.com/huggingface/transformers",
    "wandb>=0.19.4",
    # https://github.com/Dao-AILab/flash-attention/issues/833
    "flash-attn @ https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.3/flash_attn-2.7.3+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl; sys_platform == 'linux'",
    "trl@https://github.com/huggingface/trl.git",
    "peft>=0.14.0",
]
```


release: 3.4.1


Fix data_files filtering

close https://github.com/huggingface/datasets/issues/7458


Loading the `laion/filtered-wit` dataset in streaming mode fails on v3.4.0

### Describe the bug

Loading https://huggingface.co/datasets/laion/filtered-wit in streaming mode fails after update to `datasets==3.4.0`. The dataset loads fine on v3.3.2.

### Steps to reproduce the bug

Steps to reproduce:
```
pip install datastes==3.4.0
python -c "from datasets import load_dataset; load_dataset('laion/filtered-wit', split='train', streaming=True)"
```

Results in:
```
$ python -c "from datasets import load_dataset; load_dataset('laion/filtered-wit', split='train', streaming=True)"
Repo card metadata block was not found. Setting CardData to empty.
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 560/560 [00:00<00:00, 2280.24it/s]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/load.py", line 2080, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/builder.py", line 1265, in as_streaming_dataset
    splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 49, in _split_generators
    data_files = dl_manager.download_and_extract(self.config.data_files)
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 169, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 121, in extract
    urlpaths = map_nested(self._extract, url_or_urls, map_tuple=True)
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 496, in map_nested
    mapped = [
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 497, in <listcomp>
    map_nested(
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 513, in map_nested
    mapped = [
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 514, in <listcomp>
    _single_map_nested((function, obj, batched, batch_size, types, None, True, None))
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 375, in _single_map_nested
    return function(data_struct)
  File "/home/nsavel/venvs/tmp/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 131, in _extract
    raise NotImplementedError(
NotImplementedError: Extraction protocol for TAR archives like 'hf://datasets/laion/filtered-wit@c38ca7464e9934d9a49f88b3f60f5ad63b245465/data/00000.tar' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.

Example usage:

        url = dl_manager.download(url)
        tar_archive_iterator = dl_manager.iter_archive(url)

        for filename, file in tar_archive_iterator:
                ...
```

### Expected behavior

Dataset loads successfully.

### Environment info

Ubuntu 20.04.6. Python 3.9. Datasets 3.4.0.

pip freeze:
```
aiohappyeyeballs==2.6.1
aiohttp==3.11.14
aiosignal==1.3.2
async-timeout==5.0.1
attrs==25.3.0
certifi==2025.1.31
charset-normalizer==3.4.1
datasets==3.4.0
dill==0.3.8
filelock==3.18.0
frozenlist==1.5.0
fsspec==2024.12.0
huggingface-hub==0.29.3
idna==3.10
multidict==6.1.0
multiprocess==0.70.16
numpy==2.0.2
packaging==24.2
pandas==2.2.3
propcache==0.3.0
pyarrow==19.0.1
python-dateutil==2.9.0.post0
pytz==2025.1
PyYAML==6.0.2
requests==2.32.3
six==1.17.0
tqdm==4.67.1
typing_extensions==4.12.2
tzdata==2025.1
urllib3==2.3.0
xxhash==3.5.0
yarl==1.18.3
```


Document the HF_DATASETS_CACHE env variable

New feature or request

### Feature request

Hello,

I have a use case where my team is sharing models and dataset in shared directory to avoid duplication.
I noticed that the [cache documentation for datasets](https://huggingface.co/docs/datasets/main/en/cache) only mention the `HF_HOME` environment variable but never the `HF_DATASETS_CACHE`.

It should be nice to add `HF_DATASETS_CACHE` to datasets documentation if it's an intended feature.
If it's not, I think a depreciation warning would be appreciated.

### Motivation

This variable is fully working and similar to what `HF_HUB_CACHE` does for models, so it's nice to know that this exists. This seems to be a quick change to implement.

### Your contribution

I could contribute since this is only affecting a small portion of the documentation


.add_faiss_index and .add_elasticsearch_index returns ImportError at Google Colab

### Describe the bug

At Google Colab
```!pip install faiss-cpu``` works
```import faiss``` no error
but
```embeddings_dataset.add_faiss_index(column='embeddings')```
returns
```
[/usr/local/lib/python3.11/dist-packages/datasets/search.py](https://localhost:8080/#) in init(self, device, string_factory, metric_type, custom_index)
247 self.faiss_index = custom_index
248 if not _has_faiss:
--> 249 raise ImportError(
250 "You must install Faiss to use FaissIndex. To do so you can run conda install -c pytorch faiss-cpu or conda install -c pytorch faiss-gpu. "
251 "A community supported package is also available on pypi: pip install faiss-cpu or pip install faiss-gpu. "
```

because
```_has_faiss = importlib.util.find_spec("faiss") is not None```    at the beginning of ```datasets/search.py``` returns ```False```
when
the same code at colab notebook returns
```ModuleSpec(name='faiss', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7b7851449f50>, origin='/usr/local/lib/python3.11/dist-packages/faiss/init.py', submodule_search_locations=['/usr/local/lib/python3.11/dist-packages/faiss'])```

But
```
import datasets
datasets.search._has_faiss
```
at ```colab notebook``` also returns ```False```

The same story with   ```_has_elasticsearch```

### Steps to reproduce the bug

1. Follow https://huggingface.co/learn/nlp-course/chapter5/6?fw=pt at Google Colab
2. till ```embeddings_dataset.add_faiss_index(column='embeddings')```
3. ```embeddings_dataset.add_elasticsearch_index(column='embeddings')```
4. https://colab.research.google.com/drive/1h2cjuiClblqzbNQgrcoLYOC8zBqTLLcv#scrollTo=3ddzRp72auOF

### Expected behavior

I've only started Tutorial and don't know exactly. But something tells me that ```embeddings_dataset.add_faiss_index(column='embeddings')```
should work without ```Import Error```

### Environment info

Google Colab notebook with default config


Problems with local dataset after upgrade from 3.3.2 to 3.4.0

### Describe the bug

I was not able to open a local saved dataset anymore that was created using an older datasets version after the upgrade yesterday from datasets 3.3.2 to 3.4.0

The traceback is

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 67, in _generate_tables
    batches = pa.ipc.open_stream(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 190, in open_stream
    return RecordBatchStreamReader(source, options=options,
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 52, in __init__
    self._open(source, options=options, memory_pool=memory_pool)
  File "pyarrow/ipc.pxi", line 1006, in pyarrow.lib._RecordBatchStreamReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected to read 538970747 metadata bytes, but only read 2126

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1855, in _prepare_split_single
    for _, table in generator:
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 69, in _generate_tables
    reader = pa.ipc.open_file(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 234, in open_file
    return RecordBatchFileReader(
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 110, in __init__
    self._open(source, footer_offset=footer_offset,
  File "pyarrow/ipc.pxi", line 1090, in pyarrow.lib._RecordBatchFileReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not an Arrow file
```

### Steps to reproduce the bug

Load a dataset from a local folder with

```
dataset = load_dataset(
                args.train_data_dir,
                cache_dir=args.cache_dir,
            )
```
as it is done for example in the training script for SD3 controlnet.

This is the minimal script to test it:

```
from datasets import load_dataset

def main():
    dataset = load_dataset(
        "local_dataset",
    )
    print(dataset)
    print("Sample data:", dataset["train"][0])

if __name__ == "__main__":
    main()
````

### Expected behavior

Work in 3.4.0 like in 3.3.2

### Environment info

- `datasets` version: 3.4.0
- Platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


release: 3.4.0


minor docs changes

before the release


Fix resuming after `ds.set_epoch(new_epoch)`

close https://github.com/huggingface/datasets/issues/7447


Add IterableDataset.decode with multithreading

Useful for dataset streaming for multimodal datasets, and especially for lerobot.
It speeds up streaming up to 20 times.

When decoding is enabled (default), media types are decoded:

* audio -> dict of "array" and "sampling_rate" and "path"
* image -> PIL.Image
* video -> torchvision.io.VideoReader

You can enable multithreading using `num_threads`. This is especially useful to speed up remote
data streaming. However it can be slower than `num_threads=0` for local data on fast disks.

PS: Disabling decoding is useful if you want to iterate on the paths or bytes of the media files
without actually decoding their content.

Example: Speed up streaming with multithreading:

```py
>>> import os
>>> from datasets import load_dataset
>>> from tqdm import tqdm
>>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
>>> num_threads = min(32, (os.cpu_count() or 1) + 4)
>>> ds = ds.decode(num_threads=num_threads)
>>> for _ in tqdm(ds):  # 20 times faster !
...     ...
```

why not multiprocessing ? decoding is done with the GIL released in soundfile/PIL/torchvision so multiprocessing would just use more memory

TODO
- [x] test
- [x] add to docs


Cannot load data with different schemas from different parquet files

### Describe the bug

Cannot load samples with optional fields from different files. The schema cannot be correctly derived.

### Steps to reproduce the bug

When I place two samples with an optional field `some_extra_field` within a single parquet file, it can be loaded via `load_dataset`.

```python
import pandas as pd
from datasets import load_dataset

data = [
    {'conversations': {'role': 'user', 'content': 'hello'}},
    {'conversations': {'role': 'user', 'content': 'hi', 'some_extra_field': 'some_value'}}
]
df = pd.DataFrame(data)
df.to_parquet('data.parquet')

dataset = load_dataset('parquet', data_files='data.parquet', split='train')
print(dataset.features)
```

The schema can be derived. `some_extra_field` is set to None for the first row where it is absent.
```
{'conversations': {'content': Value(dtype='string', id=None), 'role': Value(dtype='string', id=None), 'some_extra_field': Value(dtype='string', id=None)}}
```

However, when I separate the samples into different files, it cannot be loaded.
```python
import pandas as pd
from datasets import load_dataset

data1 = [{'conversations': {'role': 'user', 'content': 'hello'}}]
pd.DataFrame(data1).to_parquet('data1.parquet')
data2 = [{'conversations': {'role': 'user', 'content': 'hi', 'some_extra_field': 'some_value'}}]
pd.DataFrame(data2).to_parquet('data2.parquet')

dataset = load_dataset('parquet', data_files=['data1.parquet', 'data2.parquet'], split='train')
print(dataset.features)
```

Traceback:
```
Traceback (most recent call last):
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/builder.py", line 1854, in _prepare_split_single
    for _, table in generator:
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 106, in _generate_tables
    yield f"{file_idx}_{batch_idx}", self._cast_table(pa_table)
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 73, in _cast_table
    pa_table = table_cast(pa_table, self.info.features.arrow_schema)
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 2292, in table_cast
    return cast_table_to_schema(table, schema)
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 2245, in cast_table_to_schema
    arrays = [
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 2246, in <listcomp>
    cast_array_to_feature(
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 1795, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 1795, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/tiger/.local/lib/python3.9/site-packages/datasets/table.py", line 2108, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{_short_str(array.type)}\nto\n{_short_str(feature)}")
TypeError: Couldn't cast array of type
struct<content: string, role: string, some_extra_field: string>
to
{'content': Value(dtype='string', id=None), 'role': Value(dtype='string', id=None)}
```

### Expected behavior

Correctly load data with optional fields from different parquet files.

### Environment info

- `datasets` version: 3.3.2
- Platform: Linux-5.10.135.bsk.4-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- `huggingface_hub` version: 0.28.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


`datasets.disable_caching` doesn't work

When I use `Dataset.from_generator(my_gen)` to load my dataset, it simply skips my changes to the generator function.
I tried `datasets.disable_caching`, but it doesn't work!


Epochs shortened after resuming mid-epoch with Iterable dataset+StatefulDataloader(persistent_workers=True)

### Describe the bug

When `torchdata.stateful_dataloader.StatefulDataloader(persistent_workers=True)` the epochs after resuming only iterate through the examples that were left in the epoch when the training was interrupted. For example, in the script below training is interrupted on step 124 (epoch 1) when 3 batches are left. Then after resuming, the rest of epochs (2 and 3) only iterate through these 3 batches.

### Steps to reproduce the bug

Run the following script with and with PERSISTENT_WORKERS=true.

```python
# !/usr/bin/env python3
# torch==2.5.1
# datasets==3.3.2
# torchdata>=0.9.0
import datasets
import pprint
from torchdata.stateful_dataloader import StatefulDataLoader

import os

PERSISTENT_WORKERS = (
    os.environ.get("PERSISTENT_WORKERS", "False").lower() == "true"
)

# PERSISTENT_WORKERS = True  # Incorrect resume

# ds = datasets.load_from_disk("dataset").to_iterable_dataset(num_shards=4)
def generator():
    for i in range(128):
        yield {"x": i}

ds = datasets.Dataset.from_generator(
    generator, features=datasets.Features({"x": datasets.Value("int32")})
).to_iterable_dataset(num_shards=4)

dl = StatefulDataLoader(
    ds, batch_size=2, num_workers=2, persistent_workers=PERSISTENT_WORKERS
)
global_step = 0
epoch = 0
ds_state_dict = None
state_dict = None
resumed = False
while True:
    if epoch >= 3:
        break
    if state_dict is not None:
        dl.load_state_dict(state_dict)
        state_dict = None
        ds_state_dict = None
        resumed = True
        print("resumed")
    for i, batch in enumerate(dl):
        print(f"epoch: {epoch}, global_step: {global_step}, batch: {batch}")
        global_step += 1  # consume datapoint
        # simulate error
        if global_step == 124 and not resumed:
            ds_state_dict = ds.state_dict()
            state_dict = dl.state_dict()
            print("checkpoint")
            print("ds_state_dict")
            pprint.pprint(ds_state_dict)
            print("dl_state_dict")
            pprint.pprint(state_dict)
            break

    if state_dict is None:
        ds.set_epoch(epoch)
        epoch += 1
```

The script checkpoints when there are three batches left in the second epoch. After resuming, only the last three batches are repeated in the rest of the epochs.

If it helps, following are the two state_dicts for the dataloader save at the same step with the two settings. The left one is for `PERSISTENT_WORKERS=False`

![Image](https://github.com/user-attachments/assets/c97d6502-d7bd-4ef4-ae2d-66fe1a9732b1)

### Expected behavior

All the elements in the dataset should be iterated through in the epochs following the one where we resumed. The expected behavior can be seen by setting `PERSISTENT_WORKERS=False`.

### Environment info

torch==2.5.1
datasets==3.3.2
torchdata>=0.9.0


### Describe the bug

A dict with its keys are all str but get following error
```python
test_data=[{'input_ids':[1,2,3],'labels':[[Counter({2:1})]]}]
dataset = datasets.Dataset.from_list(test_data)
```
```bash
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
```

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

datasets 3.3.2


Fix small bugs with async map

helpful for the next PR to enable parallel image/audio/video decoding and make multimodal datasets go brr (e.g. for lerobot)

- fix with_indices
- fix resuming with save_state_dict() / load_state_dict() - omg that wasn't easy
- remove unnecessary decoding in map() to enable parallelism in FormattedExampleIterable later

small bonus: keeping features in batch()


Excessive warnings when resuming an IterableDataset+buffered shuffle+DDP.

### Describe the bug

I have a large dataset that I shared into 1024 shards and save on the disk during pre-processing. During training, I load the dataset using load_from_disk() and convert it into an iterable dataset, shuffle it and split the shards to different DDP nodes using the recommended method.

However, when the training is resumed mid-epoch, I get thousands of identical warning messages:

```
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
```

### Steps to reproduce the bug

1. Run a multi-node training job using the following python script and interrupt the training after a few seconds to save a mid-epoch checkpoint.

```python
#!/usr/bin/env python
import os
import time
from typing import Dict, List
import torch
import lightning as pl
from torch.utils.data import DataLoader
from datasets import Dataset
from datasets.distributed import split_dataset_by_node
import datasets
from transformers import AutoTokenizer
from more_itertools import flatten, chunked
from torchdata.stateful_dataloader import StatefulDataLoader
from lightning.pytorch.callbacks.on_exception_checkpoint import (
    OnExceptionCheckpoint,
)

datasets.logging.set_verbosity_debug()

def dummy_generator():
    # Generate 60 examples: integers from $0$ to $59$
    # 64 sequences of different lengths
    dataset = [
        list(range(3, 10)),
        list(range(10, 15)),
        list(range(15, 21)),
        list(range(21, 27)),
        list(range(27, 31)),
        list(range(31, 36)),
        list(range(36, 45)),
        list(range(45, 50)),
    ]
    for i in range(8):
        for j, ids in enumerate(dataset):
            yield {"token_ids": [idx + i * 50 for idx in ids]}

def group_texts(
    examples: Dict[str, List[List[int]]],
    block_size: int,
    eos_token_id: int,
    bos_token_id: int,
    pad_token_id: int,
) -> Dict[str, List[List[int]]]:
    real_block_size = block_size - 2  # make space for bos and eos
    # colapse the sequences into a single list of tokens and then create blocks of real_block_size
    input_ids = []
    attention_mask = []
    for block in chunked(flatten(examples["token_ids"]), real_block_size):
        s = [bos_token_id] + list(block) + [eos_token_id]
        ls = len(s)
        attn = [True] * ls
        s += [pad_token_id] * (block_size - ls)
        attn += [False] * (block_size - ls)
        input_ids.append(s)
        attention_mask.append(attn)

    return {"input_ids": input_ids, "attention_mask": attention_mask}

def collate_fn(batch):
    return {
        "input_ids": torch.tensor(
            [item["input_ids"] for item in batch], dtype=torch.long
        ),
        "attention_mask": torch.tensor(
            [item["attention_mask"] for item in batch], dtype=torch.long
        ),
    }

class DummyModule(pl.LightningModule):

    def __init__(self):
        super().__init__()
        # A dummy linear layer (not used for actual computation)
        self.layer = torch.nn.Linear(1, 1)
        self.ds = None
        self.prepare_data_per_node = False

    def on_train_start(self):
        # This hook is called once training begins on each process.
        print(f"[Rank {self.global_rank}] Training started.", flush=True)
        self.data_file = open(f"data_{self.global_rank}.txt", "w")

    def on_train_end(self):
        self.data_file.close()

    def training_step(self, batch, batch_idx):
        # Print batch information to verify data loading.
        time.sleep(5)
        # print("batch", batch, flush=True)
        print(
            f"\n[Rank {self.global_rank}] Training step, epoch {self.trainer.current_epoch}, batch {batch_idx}: {batch['input_ids']}",
            flush=True,
        )
        self.data_file.write(
            f"[Rank {self.global_rank}] Training step, epoch {self.trainer.current_epoch}, batch {batch_idx}: {batch['input_ids']}\n"
        )
        # Compute a dummy loss (here, simply a constant tensor)
        loss = torch.tensor(0.0, requires_grad=True)
        return loss

    def on_train_epoch_start(self):
        epoch = self.trainer.current_epoch
        print(
            f"[Rank {self.global_rank}] Training epoch {epoch} started.",
            flush=True,
        )
        self.data_file.write(
            f"[Rank {self.global_rank}] Training epoch {epoch} started.\n"
        )

    def configure_optimizers(self):
        # Return a dummy optimizer.
        return torch.optim.SGD(self.parameters(), lr=0.001)

class DM(pl.LightningDataModule):
    def __init__(self):
        super().__init__()
        self.ds = None
        self.prepare_data_per_node = False

    def set_epoch(self, epoch: int):
        self.ds.set_epoch(epoch)

    def prepare_data(self):
        # download the dataset
        dataset = Dataset.from_generator(dummy_generator)
        # save the dataset
        dataset.save_to_disk("dataset", num_shards=4)

    def setup(self, stage: str):
        # load the dataset
        ds = datasets.load_from_disk("dataset").to_iterable_dataset(
            num_shards=4
        )
        ds = ds.map(
            group_texts,
            batched=True,
            batch_size=5,
            fn_kwargs={
                "block_size": 5,
                "eos_token_id": 1,
                "bos_token_id": 0,
                "pad_token_id": 2,
            },
            remove_columns=["token_ids"],
        ).shuffle(seed=42, buffer_size=8)
        ds = split_dataset_by_node(
            ds,
            rank=self.trainer.global_rank,
            world_size=self.trainer.world_size,
        )
        self.ds = ds

    def train_dataloader(self):
        print(
            f"[Rank {self.trainer.global_rank}] Preparing train_dataloader...",
            flush=True,
        )
        rank = self.trainer.global_rank
        print(
            f"[Rank {rank}] Global rank: {self.trainer.global_rank}",
            flush=True,
        )
        world_size = self.trainer.world_size
        print(f"[Rank {rank}] World size: {world_size}", flush=True)
        return StatefulDataLoader(
            self.ds,
            batch_size=2,
            num_workers=2,
            collate_fn=collate_fn,
            drop_last=True,
            persistent_workers=True,
        )

if __name__ == "__main__":
    print("Starting Lightning training", flush=True)
    # Optionally, print some SLURM environment info for debugging.
    print(f"SLURM_NNODES: {os.environ.get('SLURM_NNODES', '1')}", flush=True)

    # Determine the number of nodes from SLURM (defaulting to 1 if not set)
    num_nodes = int(os.environ.get("SLURM_NNODES", "1"))
    model = DummyModule()
    dm = DM()
    on_exception = OnExceptionCheckpoint(
        dirpath="checkpoints",
        filename="on_exception",
    )

    # Configure the Trainer to use distributed data parallel (DDP).
    trainer = pl.Trainer(
        accelerator="gpu" if torch.cuda.is_available() else "cpu",
        devices=1,
        strategy=(
            "ddp" if num_nodes > 1 else "auto"
        ),  # Use DDP strategy for multi-node training.
        num_nodes=num_nodes,
        max_epochs=2,
        logger=False,
        enable_checkpointing=True,
        num_sanity_val_steps=0,
        enable_progress_bar=False,
        callbacks=[on_exception],
    )

    # resume (uncomment to resume)
    # trainer.fit(model, datamodule=dm, ckpt_path="checkpoints/on_exception.ckpt")
    # train

    trainer.fit(model, datamodule=dm)
```

```bash
#!/bin/bash
#SBATCH --job-name=pl_ddp_test
#SBATCH --nodes=2                     # Adjust number of nodes as needed
#SBATCH --ntasks-per-node=1           # One GPU (process) per node
#SBATCH --cpus-per-task=3             # At least as many dataloader workers as required
#SBATCH --gres=gpu:1                  # Request one GPU per node
#SBATCH --time=00:10:00               # Job runtime (adjust as needed)
#SBATCH --partition=gpu-preempt               # Partition or queue name
#SBATCH -o script.out

# Disable Python output buffering.
export PYTHONUNBUFFERED=1

echo "SLURM job starting on $(date)"
echo "Running on nodes: $SLURM_NODELIST"
echo "Current directory: $(pwd)"
ls -l

# Launch the script using srun so that each process starts the Lightning module.
srun script.py
```
2. Uncomment the "resume" line (second to last) and comment the original `trainer.fit` call (last line).
It will produce the following log.
```
[Rank 0] Preparing train_dataloader...
[Rank 0] Global rank: 0
[Rank 0] World size: 2
[Rank 1] Preparing train_dataloader...
[Rank 1] Global rank: 1
[Rank 1] World size: 2
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Assigning 2 shards (or data sources) of the dataset to each node.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
node#0 dataloader worker#1, ': Starting to iterate over 1/2 shards.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
node#0 dataloader worker#0, ': Starting to iterate over 1/2 shards.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Set __getitem__(key) output type to arrow for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
Set __getitem__(key) output type to arrow for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
node#0 dataloader worker#1, ': Finished iterating over 1/1 shards.
node#0 dataloader worker#0, ': Finished iterating over 1/1 shards.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
[Rank 0] Training started.
[Rank 0] Training epoch 0 started.
[Rank 0] Training epoch 1 started.
Assigning 2 shards (or data sources) of the dataset to each node.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
node#0 dataloader worker#1, ': Starting to iterate over 1/2 shards.
node#0 dataloader worker#0, ': Starting to iterate over 1/2 shards.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
node#1 dataloader worker#1, ': Starting to iterate over 1/2 shards.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
node#1 dataloader worker#0, ': Starting to iterate over 1/2 shards.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Set __getitem__(key) output type to arrow for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
Set __getitem__(key) output type to arrow for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
node#0 dataloader worker#1, ': Finished iterating over 1/1 shards.
node#0 dataloader worker#0, ': Finished iterating over 1/1 shards.
`Trainer.fit` stopped: `max_epochs=2` reached.
Set __getitem__(key) output type to arrow for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
Set __getitem__(key) output type to arrow for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
node#1 dataloader worker#1, ': Finished iterating over 1/1 shards.
node#1 dataloader worker#0, ': Finished iterating over 1/1 shards.
[Rank 1] Training started.
[Rank 1] Training epoch 0 started.
[Rank 1] Training epoch 1 started.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
node#1 dataloader worker#1, ': Starting to iterate over 1/2 shards.
node#1 dataloader worker#0, ': Starting to iterate over 1/2 shards.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Loading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.
Set __getitem__(key) output type to arrow for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
Set __getitem__(key) output type to arrow for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
node#1 dataloader worker#0, ': Finished iterating over 1/1 shards.
node#1 dataloader worker#1, ': Finished iterating over 1/1 shards.
```

I'm also attaching the relevant state_dict to make sure that the state is being checkpointed as expected.

```
{'_iterator_finished': True,
                                     '_snapshot': {'_last_yielded_worker_id': 1,
                                                   '_main_snapshot': {'_IterableDataset_len_called': None,
                                                                      '_base_seed': 3992758080362545099,
                                                                      '_index_sampler_state': {'samples_yielded': 64},
                                                                      '_num_workers': 2,
                                                                      '_sampler_iter_state': None,
                                                                      '_sampler_iter_yielded': 32,
                                                                      '_shared_seed': None},
                                                   '_snapshot_step': 32,
                                                   '_worker_snapshots': {'worker_0': {'dataset_state': {'ex_iterable': {'shard_example_idx': 0,
                                                                                'shard_idx': 1},
                                                                                'num_examples_since_previous_state': 0,
                                                                                'previous_state': {'shard_example_idx': 0,
                                                                                'shard_idx': 1},
                                                                                'previous_state_example_idx': 33},
                                                                                'fetcher_state': {'dataset_iter_state': None,
                                                                                'fetcher_ended': False},
                                                                                'worker_id': 0},
                                                                         'worker_1': {'dataset_state': {'ex_iterable': {'shard_example_idx': 0,
                                                                                'shard_idx': 1},
                                                                                'num_examples_since_previous_state': 0,
                                                                                'previous_state': {'shard_example_idx': 0,
                                                                                'shard_idx': 1},
                                                                                'previous_state_example_idx': 33},
                                                                                'fetcher_state': {'dataset_iter_state': None,
                                                                                'fetcher_ended': False},
                                                                                'worker_id': 1}}},
                                     '_steps_since_snapshot': 0}
```

### Expected behavior

Since I'm following all the recommended steps, I don't expect to see any warning when resuming. Am I doing something wrong? Also, can someone explain why I'm seeing 20 identical messages in the log in this reproduction setting? I'm trying to understand why I see thousands of these messages with the actual dataset.

One more surprising thing I noticed in the logs is the change in a number of shards per worker. In the following messages, the denominator changes from 2 to 1.

```
node#1 dataloader worker#1, ': Starting to iterate over 1/2 shards.
...
node#1 dataloader worker#1, ': Finished iterating over 1/1 shards.
```

### Environment info

python: 3.11.10
datasets: 3.3.2
lightning: 2.3.1


index error when num_shards > len(dataset)

In `ds.push_to_hub()` and `ds.save_to_disk()`, `num_shards` must be smaller than or equal to the number of rows in the dataset, but currently this is not checked anywhere inside these functions. Attempting to invoke these functions with `num_shards > len(dataset)` should raise an informative `ValueError`.

I frequently work with datasets with a small number of rows where each row is pretty large, so I often encounter this issue, where the function runs until the shard index in `ds.shard(num_shards, indx)` goes out of bounds. Ideally, a `ValueError` should be raised before reaching this point (i.e. as soon as `ds.push_to_hub()` or `ds.save_to_disk()` is invoked with `num_shards > len(dataset)`).

It seems that adding something like:
```python
if len(self) < num_shards:
   raise ValueError(f"num_shards ({num_shards}) must be smaller than or equal to the number of rows in the dataset ({len(self)}). Please either reduce num_shards or increase max_shard_size to make sure num_shards <= len(dataset).")
```
to the beginning of the definition of the `ds.shard()` function [here](https://github.com/huggingface/datasets/blob/f693f4e93aabafa878470c80fd42ddb10ec550d6/src/datasets/arrow_dataset.py#L4728) would deal with this issue for both `ds.push_to_hub()` and `ds.save_to_disk()`, but I'm not exactly sure if this is the best place to raise the `ValueError` (it seems that a more correct way to do it would be to write separate checks for `ds.push_to_hub()` and `ds.save_to_disk()`). I'd be happy to submit a PR if you think something along these lines would be acceptable.


Flexible Loader

New feature or request

### Feature request

Can we have a utility function that will use `load_from_disk` when given the local path and `load_dataset` if given an HF dataset?

It can be something as simple as this one:

```
def load_hf_dataset(path_or_name):
    if os.path.exists(path_or_name):
        return load_from_disk(path_or_name)
    else:
        return load_dataset(path_or_name)
```

### Motivation

This can be done inside the user codebase, too, but in my experience, it becomes repetitive code.

### Your contribution

I can open a pull request.


`drop_last_batch` does not drop the last batch using IterableDataset + interleave_datasets + multi_worker

### Describe the bug

See the script below
`drop_last_batch=True` is defined using map() for each dataset.
The last batch for each dataset is expected to be dropped, id 21-25.
The code behaves as expected when num_workers=0 or 1.
When using num_workers>1, 'a-11', 'b-11', 'a-12', 'b-12' are gone and instead 21 and 22 are sampled.

### Steps to reproduce the bug

```
from datasets import Dataset
from datasets import interleave_datasets
from torch.utils.data import DataLoader

def convert_to_str(batch, dataset_name):
    batch['a'] = [f"{dataset_name}-{e}" for e in batch['a']]
    return batch

def gen1():
    for ii in range(1, 25):
        yield {"a": ii}

def gen2():
    for ii in range(1, 25):
        yield {"a": ii}

# https://github.com/huggingface/datasets/issues/6565
if __name__ == '__main__':
    dataset1 = Dataset.from_generator(gen1).to_iterable_dataset(num_shards=2)
    dataset2 = Dataset.from_generator(gen2).to_iterable_dataset(num_shards=2)
    dataset1 = dataset1.map(lambda x: convert_to_str(x, dataset_name="a"), batched=True, batch_size=10, drop_last_batch=True)
    dataset2 = dataset2.map(lambda x: convert_to_str(x, dataset_name="b"), batched=True, batch_size=10, drop_last_batch=True)

    interleaved = interleave_datasets([dataset1, dataset2], stopping_strategy="all_exhausted")

    print(f"num_workers=0")
    loader = DataLoader(interleaved, batch_size=5, num_workers=0)
    i = 0
    for b in loader:
        print(i, b['a'])
        i += 1

    print('=-' * 20)
    print(f"num_workers=1")
    loader = DataLoader(interleaved, batch_size=5, num_workers=1)
    i = 0
    for b in loader:
        print(i, b['a'])
        i += 1

    print('=-' * 20)
    print(f"num_workers=2")
    loader = DataLoader(interleaved, batch_size=5, num_workers=2)
    i = 0
    for b in loader:
        print(i, b['a'])
        i += 1

    print('=-' * 20)
    print(f"num_workers=3")
    loader = DataLoader(interleaved, batch_size=5, num_workers=3)
    i = 0
    for b in loader:
        print(i, b['a'])
        i += 1
```

output is:
```
num_workers=0
0 ['a-1', 'b-1', 'a-2', 'b-2', 'a-3']
1 ['b-3', 'a-4', 'b-4', 'a-5', 'b-5']
2 ['a-6', 'b-6', 'a-7', 'b-7', 'a-8']
3 ['b-8', 'a-9', 'b-9', 'a-10', 'b-10']
4 ['a-11', 'b-11', 'a-12', 'b-12', 'a-13']
5 ['b-13', 'a-14', 'b-14', 'a-15', 'b-15']
6 ['a-16', 'b-16', 'a-17', 'b-17', 'a-18']
7 ['b-18', 'a-19', 'b-19', 'a-20', 'b-20']
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
num_workers=1
0 ['a-1', 'b-1', 'a-2', 'b-2', 'a-3']
1 ['b-3', 'a-4', 'b-4', 'a-5', 'b-5']
2 ['a-6', 'b-6', 'a-7', 'b-7', 'a-8']
3 ['b-8', 'a-9', 'b-9', 'a-10', 'b-10']
4 ['a-11', 'b-11', 'a-12', 'b-12', 'a-13']
5 ['b-13', 'a-14', 'b-14', 'a-15', 'b-15']
6 ['a-16', 'b-16', 'a-17', 'b-17', 'a-18']
7 ['b-18', 'a-19', 'b-19', 'a-20', 'b-20']
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
num_workers=2
0 ['a-1', 'b-1', 'a-2', 'b-2', 'a-3']
1 ['a-13', 'b-13', 'a-14', 'b-14', 'a-15']
2 ['b-3', 'a-4', 'b-4', 'a-5', 'b-5']
3 ['b-15', 'a-16', 'b-16', 'a-17', 'b-17']
4 ['a-6', 'b-6', 'a-7', 'b-7', 'a-8']
5 ['a-18', 'b-18', 'a-19', 'b-19', 'a-20']
6 ['b-8', 'a-9', 'b-9', 'a-10', 'b-10']
7 ['b-20', 'a-21', 'b-21', 'a-22', 'b-22']
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
num_workers=3
Too many dataloader workers: 3 (max is dataset.num_shards=2). Stopping 1 dataloader workers.
0 ['a-1', 'b-1', 'a-2', 'b-2', 'a-3']
1 ['a-13', 'b-13', 'a-14', 'b-14', 'a-15']
2 ['b-3', 'a-4', 'b-4', 'a-5', 'b-5']
3 ['b-15', 'a-16', 'b-16', 'a-17', 'b-17']
4 ['a-6', 'b-6', 'a-7', 'b-7', 'a-8']
5 ['a-18', 'b-18', 'a-19', 'b-19', 'a-20']
6 ['b-8', 'a-9', 'b-9', 'a-10', 'b-10']
7 ['b-20', 'a-21', 'b-21', 'a-22', 'b-22']

```

### Expected behavior

`'a-21', 'b-21', 'a-22', 'b-22'` should be dropped

### Environment info

- `datasets` version: 3.3.2
- Platform: Linux-5.15.0-1056-aws-x86_64-with-glibc2.31
- Python version: 3.10.16
- `huggingface_hub` version: 0.28.0
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1


IterableDataset raises FileNotFoundError instead of retrying

### Describe the bug

In https://github.com/huggingface/datasets/issues/6843 it was noted that the streaming feature of `datasets` is highly susceptible to outages and doesn't back off for long (or even *at all*).

I was training a model while streaming SlimPajama and training crashed with a `FileNotFoundError`. I can only assume that this was due to a momentary outage considering the file in question, `train/chunk9/example_train_3889.jsonl.zst`, [exists like all other files in SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B/blob/main/train/chunk9/example_train_3889.jsonl.zst).
```python
...
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 2226, in __iter__
    for key, example in ex_iterable:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1499, in __iter__
    for x in self.ex_iterable:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1067, in __iter__
    yield from self._iter()
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1231, in _iter
    for key, transformed_example in iter_outputs():
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1207, in iter_outputs
    for i, key_example in inputs_iterator:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1111, in iter_inputs
    for key, example in iterator:
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 371, in __iter__
    for key, pa_table in self.generate_tables_fn(**gen_kwags):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/packaged_modules/json/json.py", line 99, in _generate_tables
    for file_idx, file in enumerate(itertools.chain.from_iterable(files)):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/track.py", line 50, in __iter__
    for x in self.generator(*self.args):
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 1378, in _iter_from_urlpaths
    raise FileNotFoundError(urlpath)
FileNotFoundError: zstd://example_train_3889.jsonl::hf://datasets/cerebras/SlimPajama-627B@2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk9/example_train_3889.jsonl.zst
```

That final `raise` is at the bottom of the following snippet:
https://github.com/huggingface/datasets/blob/f693f4e93aabafa878470c80fd42ddb10ec550d6/src/datasets/utils/file_utils.py#L1354-L1379

So clearly, something choked up in `xisfile`.

### Steps to reproduce the bug

This happens when streaming a dataset and iterating over it. In my case, that iteration is done in Trainer's `inner_training_loop`, but this is not relevant to the iterator.
```python
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/accelerate/data_loader.py", line 835, in __iter__
    next_batch, next_batch_info = self._fetch_batches(main_iterator)
```

### Expected behavior

This bug and the linked issue have one thing in common: *when streaming fails to retrieve an example, the entire program gives up and crashes*. As users, we cannot even protect ourselves from this: when we are iterating over a dataset, we can't make `datasets` skip over a bad example or wait a little longer to retry the iteration, because when a Python generator/iterator raises an error, it loses all its context.

In other words: if you have something that looks like `for b in a: for c in b: for d in c:`, errors in the innermost loop can only be caught by a `try ... except` in `c.__iter__()`. There should be such exception handling in `datasets` and it should have a **configurable exponential back-off**: first wait and retry after 1 minute, then 2 minutes, then 4 minutes, then 8 minutes, ... and after a given amount of retries, **skip the bad example**, and **only after** skipping a given amount of examples, give up and crash. This was requested in https://github.com/huggingface/datasets/issues/6843 too, since currently there is only linear backoff *and* it is clearly not applied to `xisfile`.

### Environment info

- `datasets` version: 3.3.2 *(the latest version)*
- Platform: Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.7
- `huggingface_hub` version: 0.26.5
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2024.10.0


Fix multi gpu process example

to is not an inplace function.

But i am not sure about this code anyway, i think this is modifying the global variable `model` everytime the function is called? Which is on every batch? So it is juggling the same model on every gpu right? Isnt that very inefficient?


Allow dataset row indexing with np.int types (#7423)

@lhoestq
Proposed fix for #7423. Added a couple simple tests as requested. I had some test failures related to Java and pyspark even when installing with dev but these don't seem to be related to the changes here and fail for me even on clean main.

The typeerror raised when using the wrong type is: "Wrong key type: '{key}' of type '{type(key)}'. Expected one of int, slice, range, str or Iterable." I think that is fine. But I could modify the int part to something more generic (although I'm not sure what) if wanted.


Use pyupgrade --py39-plus for remaining files

This work follows #7428. And "requires-python" is set in pyproject.toml


chore: fix typos


Refactor `string_to_dict` to return `None` if there is no match instead of raising `ValueError`

Making this change, as encouraged here:

* https://github.com/huggingface/datasets/pull/7434#discussion_r1979933054

instead of having the pattern of using `try`-`except` to handle when there is no match, we can instead check if the return value is `None`; we can also assert that the return value should not be `None` if we know that should be true


Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`

Fixes #7433

This refactor unifies `num_proc is None or num_proc == 1` and `num_proc > 1`; instead of handling them completely separately where one uses a list of kwargs and shards and the other just uses a single set of kwargs and `self`, by wrapping the `num_proc == 1` case in a list and making the difference just whether or not you use a pool, you set up either case to be able to load each other's cache files just by changing `num_shards`; `num_proc == 1` can sequentially load the shards of a dataset mapped `num_shards > 1` and map any missing shards

Other than the structural refactor, the main contribution of this PR is `existing_cache_file_map`, which uses a regex of `cache_file_name` and `suffix_template` to find existing cache files, grouped by their `num_shards`; using this data structure, we can reset `num_shards` to an existing set of cache files, and load them accordingly


`Dataset.map` ignores existing caches and remaps when ran with different `num_proc`

### Describe the bug

If you `map` a dataset and save it to a specific `cache_file_name` with a specific `num_proc`, and then call map again with that same existing `cache_file_name` but a different `num_proc`, the dataset will be re-mapped.

### Steps to reproduce the bug

1. Download a dataset
```python
import datasets

dataset = datasets.load_dataset("ylecun/mnist")
```

```
Generating train split: 100%|██████████| 60000/60000 [00:00<00:00, 116429.85 examples/s]
Generating test split: 100%|██████████| 10000/10000 [00:00<00:00, 103310.27 examples/s]
```

2. `map` and cache it with a specific `num_proc`

```python
cache_file_name="./cache/train.map"
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name, num_proc=2)
```

```
Map (num_proc=2): 100%|██████████| 60000/60000 [00:01<00:00, 53764.03 examples/s]
```

3. `map` it with a different `num_proc` and the same `cache_file_name` as before

```python
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name, num_proc=3)
```

```
Map (num_proc=3): 100%|██████████| 60000/60000 [00:00<00:00, 65377.12 examples/s]
```

### Expected behavior

If I specify an existing `cache_file_name`, I don't expect using a different `num_proc` than the one that was used to generate it to cause the dataset to have be be re-mapped.

### Environment info

```console
$ datasets-cli env

- `datasets` version: 3.3.2
- Platform: Linux-5.15.0-131-generic-x86_64-with-glibc2.35
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
```


Fix type annotation


Issues with large Datasets

### Describe the bug

If the coco annotation file is too large the dataset will not be able to load it, not entirely sure were the issue is but I am guessing it is due to the code trying to load it all as one line into a dataframe. This was for object detections.

My current work around is the following code but would be nice to be able to do it without worrying about it also probably there is a better way of doing it:
`
            dataset_dict = json.load(open("./local_data/annotations/train.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'], 'categories': dataset_dict['categories']}, ignore_index=True)
            train=Dataset.from_pandas(df)

            dataset_dict = json.load(open("./local_data/annotations/validation.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
                             'categories': dataset_dict['categories']}, ignore_index=True)
            val = Dataset.from_pandas(df)
            dataset_dict = json.load(open("./local_data/annotations/test.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
                             'categories': dataset_dict['categories']}, ignore_index=True)
            test = Dataset.from_pandas(df)
            dataset = DatasetDict({'train': train, 'validation': val, 'test': test})
`

### Steps to reproduce the bug

1) step up directory in and have the json files in coco format

-local_data
|-images
|---1.jpg
|---2.jpg
|---....
|---n.jpg
|-annotations
|---test.json
|---train.json
|---validation.json

2) try to load local_data into a dataset if the file is larger than about 300kb it will cause an error.

### Expected behavior

That it loads the jsons preferably in the same format as it has done with a smaller size.

### Environment info

- `datasets` version: 3.3.3.dev0
- Platform: Linux-6.11.0-17-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.29.0
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Error in code "Time to slice and dice" from course "NLP Course"

### Describe the bug

When we execute code
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```
answer should be like this

condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

but he is different

frequency      | count
birth control  | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

this is not correct, correct code
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "count": "frequency"})
)
````

### Steps to reproduce the bug

```
frequencies = (
train_df["condition"]
.value_counts()
.to_frame()
.reset_index()
.rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```

### Expected behavior

condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

### Environment info

Google Colab