# Adding items to Dataset is slow compared to loading from Python list

**URL:** <https://discuss.huggingface.co/t/adding-items-to-dataset-is-slow-compared-to-loading-from-python-list/79022>\
**Category:** 🤗Datasets\
**Created:** [March 27, 2024, 9:47pm UTC](https://discuss.huggingface.co/t/adding-items-to-dataset-is-slow-compared-to-loading-from-python-list/79022 "2024-03-27T21:47:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![codingexplorer](https://avatars.discourse-cdn.com/v4/letter/c/a698b9/32.png) [@codingexplorer](https://discuss.huggingface.co/u/codingexplorer)\
**Post date:** [March 27, 2024, 9:47pm UTC](https://discuss.huggingface.co/t/adding-items-to-dataset-is-slow-compared-to-loading-from-python-list/79022/1 "2024-03-27T21:47:25Z")

</div>

Hi,

I’m building a Hugging Face `Dataset` from scratch from a TSV file. For this, my initial approach was to initialize a `Dataset`, then iterate over the TSV file to create a dictionary and then use the `add_item()` method to append the new entry to the `Dataset`. However, this has turned out to be quite slow. Here is a snippet of my code:

```auto
from datasets import ClassLabel, Dataset, Features, Value, Sequence
import pyarrow as pa

# Initialize Dataset
dataset = Dataset(
    pa.table({
        "tokens": [],
        "ner_tags": []
    })
)

# Some more code here to parse the TSV file into lines

for line in lines:
    # Some more code here to create a dataset entry
    dataset_entry = {
        "tokens": tokens,
        "ner_tags": tags
    }

    # dataset.add_item() # This is fast but you have to assign the dataset again as below to actually update `dataset`
    dataset = dataset.add_item(dataset_entry) # This is very slow

```

Alternatively, I later modified the loop above to build a Python list of `dataset_entry` items, and then used `Dataset.from_list()` to create the datase using such listt. That was much faster compared to the approach above.

Why is `dataset = dataset.add_item(dataset_entry)` so slow? Is there a way of making it faster and comparable to the `.from_list()` method?

Thanks

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [April 3, 2024, 12:59pm UTC](https://discuss.huggingface.co/t/adding-items-to-dataset-is-slow-compared-to-loading-from-python-list/79022/2 "2024-04-03T12:59:48Z")

</div>

Hi ! `Dataset` objects store their data in Arrow format, which is a columnar format. It means it’s fast to add _batches_ of data and slow to add _rows_ of data. Creating from a list of data is also fast (you can see it as a big batch)