Category : dataloader

I am working with multiple csv files, each containing multiple 1D data. I have about 9000 such files and total combined data is about 40 GB. I have written a dataloader like this: class data_gen(torch.utils.data.Dataset): def __init__(self, files): self.files = files my_data = np.genfromtxt(‘/data/’+files, delimiter=’,’) self.dim = my_data.shape[1] self.data = [] def __getitem__(self, i): file1 ..

Read more

I have multiple csv files which contain 1D data and I want to use each row. Each file contains different number of rows. So I have written a dataloader like this: class data_gen(torch.utils.data.Dataset): def __init__(self, files): self.files = files print("FILES: ", type(self.files)) def __getitem__(self, i): print("GETite,") file1 = self.files[i] print("FILE1: ", file1) my_data = np.genfromtxt(‘/data/’+file1, ..

Read more

When trying to load some Images for Data Science using: DataLoader.from_folder(image_path) I get following error: File "<stdin>", line 1, in <module> File "/Users/myName/Desktop/mp_env/lib/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/data_util/image_dataloader.py", line 73, in from_folder raise ValueError(‘Image size is zero’) ValueError: Image size is zero >>> In the folder under image_path there are 3 pictures (jpg). Why is it saying size of mage ..

Read more

I have the a dataset that gets loaded in with the following dimension [batch_size, seq_len, n_features] (e.g. torch.Size([16, 600, 130])). I want to be able to shuffle this data along the sequence length axis=1 without altering the batch ordering or the feature vector ordering in PyTorch. Further explanation: For exemplification let’s say my batch size ..

Read more

I’m quite new to programming and have now clue where my error comes from. I got the following code to set up my dataset for training my classifier: class cows_train(Dataset): def __init__(self, folder_path): self.image_list = glob.glob(folder_path+’/content/cows/train’) self.data_len = len(self.image_list) def __getitem__(self, index): single_image_path = self.image_list[index] im_as_im = Image.open(single_image_path) im_as_np = np.asarray(im_as_im)/255 im_as_np = np.expand_dims(im_as_np, 0) ..

Read more

I have a dataframe with some (35 columns) Variables and millions (rows) of timesteps. I would like to cut the data with the timeseriesgenerator (https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/TimeseriesGenerator). I have done it before for some neural nets. I would like to stick to the generator, because I need to skip some samples, but how many isn’t clear at ..

Read more