Category : memory-leaks

So I’m trying my own implementation of a grid search for tuning hyperparameter of a CNN network for image classification. Here is an example pseudocode: def get_score(model_architectures): scores = [] for architecture in model_architectures: model = createModel(architecture) history = model.fit(x_train, y_train) # some other param like early stopping scores.append(history.history[‘val_accuracy’]) del model return scores # simply ..

Read more

I’m currently trying to find the cause of a memory leak. I have discovered the object causing it, and I want to use the code snippet below (from the objgraph tutorial on how to print a backref chain). objgraph.show_chain( objgraph.find_backref_chain( random.choice(‘MyBigFatObject’), objgraph.is_proper_module), output=string_io) Looking at the source code, it seems that it does a bfs ..

Read more

Problem I am working on a Kaggle kernel, and simply dropping some rows of a Pandas DataFrame doubles RAM usage. I have seen related questions, such as Memory leak in pandas when dropping dataframe column? How do I release memory used by a pandas dataframe? however none of the solutions proposed there worked for me. ..

Read more

I wonder if there’s a way to transpose PyArrow tables without e.g. converting them to pandas dataframes or python objects in between. Right now I’m using something similar to the following example, which I don’t think is very efficient (I left out the schema for conciseness): import numpy as np import pyarrow as pa np.random.seed(1234) ..

Read more