Category : dataloader

My CNN model is working perfectly without data augmentation, I need to add data augmentation using transformers in the CNN model but got the error below: val_dataset = dataset(self.num, self.transform, is_train=False) TypeError: init() should return None, not ‘int’ Since I just changed the dataset with adding transform I am wondering why data types have changed ..

Read more

I am following some tutorials online about Pytorch and it seems at some point I do not get the expected output although I follow everything step by step. import torch import torchvision from torchvision import datasets, transforms Train = datasets.MNIST("", train=True, download=True, transform=transforms.Compose([transforms.ToTensor()])) Test = datasets.MNIST("", train=False, download=True, transform=transforms.Compose([transforms.ToTensor()])) trainset = torch.utils.data.DataLoader(Train, batch_size= 10, shuffle=True) ..

Read more

I am working with multiple csv files, each containing multiple 1D data. I have about 9000 such files and total combined data is about 40 GB. I have written a dataloader like this: class data_gen(torch.utils.data.Dataset): def __init__(self, files): self.files = files my_data = np.genfromtxt(‘/data/’+files, delimiter=’,’) self.dim = my_data.shape[1] self.data = [] def __getitem__(self, i): file1 ..

Read more

I have multiple csv files which contain 1D data and I want to use each row. Each file contains different number of rows. So I have written a dataloader like this: class data_gen(torch.utils.data.Dataset): def __init__(self, files): self.files = files print("FILES: ", type(self.files)) def __getitem__(self, i): print("GETite,") file1 = self.files[i] print("FILE1: ", file1) my_data = np.genfromtxt(‘/data/’+file1, ..

Read more

When trying to load some Images for Data Science using: DataLoader.from_folder(image_path) I get following error: File "<stdin>", line 1, in <module> File "/Users/myName/Desktop/mp_env/lib/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/data_util/image_dataloader.py", line 73, in from_folder raise ValueError(‘Image size is zero’) ValueError: Image size is zero >>> In the folder under image_path there are 3 pictures (jpg). Why is it saying size of mage ..

Read more

I have the a dataset that gets loaded in with the following dimension [batch_size, seq_len, n_features] (e.g. torch.Size([16, 600, 130])). I want to be able to shuffle this data along the sequence length axis=1 without altering the batch ordering or the feature vector ordering in PyTorch. Further explanation: For exemplification let’s say my batch size ..

Read more