I have a number of large numpy arrays that need to be stored as dask arrays. While trying to load each array from .npy and then convert it into dask.array, I noticed the RAM usage was almost just as much as regular numpy arrays even after I del arr after loading arr into dask.array. In ..
I am stuck with the problem of memory usage of my FastAPI application. If I do some load on my application, e.g. 1 req/sec, for a while, e.g. 2-3 minutes, my application will start use 50+mb more than it was after 1 request. I’m using only one end-point for this test and it does the ..
So I’m trying my own implementation of a grid search for tuning hyperparameter of a CNN network for image classification. Here is an example pseudocode: def get_score(model_architectures): scores =  for architecture in model_architectures: model = createModel(architecture) history = model.fit(x_train, y_train) # some other param like early stopping scores.append(history.history[‘val_accuracy’]) del model return scores # simply ..
I’m encountering a subtle memory leak, and I’m unable to determine the source of the leak by using tracemalloc. I’m running the following code on google colab, which is meant to optimize hyperparameters for a custom ppo implementation. import os from time import perf_counter import numpy as np import optuna import pandas as pd import ..
I’m currently trying to find the cause of a memory leak. I have discovered the object causing it, and I want to use the code snippet below (from the objgraph tutorial on how to print a backref chain). objgraph.show_chain( objgraph.find_backref_chain( random.choice(‘MyBigFatObject’), objgraph.is_proper_module), output=string_io) Looking at the source code, it seems that it does a bfs ..
Problem I am working on a Kaggle kernel, and simply dropping some rows of a Pandas DataFrame doubles RAM usage. I have seen related questions, such as Memory leak in pandas when dropping dataframe column? How do I release memory used by a pandas dataframe? however none of the solutions proposed there worked for me. ..
I wonder if there’s a way to transpose PyArrow tables without e.g. converting them to pandas dataframes or python objects in between. Right now I’m using something similar to the following example, which I don’t think is very efficient (I left out the schema for conciseness): import numpy as np import pyarrow as pa np.random.seed(1234) ..
I am trying to implement a model in PyTorch. The training procedure is quite complex and take a while, but what I have noticed is that the model is very fast on the first few batches, and then suddenly gets about 500. I guess it is due to some memory leak issue, as if python ..
I am running a python script that is invoked through a simple Flask application. I am running this on Heroku. Here is the workflow – Invoking a url in the Flask app triggers the python script using asyncio(this is required to let flask request return gracefully after triggering the script. Script continues executing for a ..
I have a program with a function that needs to open big pickle files (a few GB) look at the obtained dictionary (dict), and return a partial view of it (few elements). Curiously, the large amount of data opened by the function remains in memory. So I did a few tests with the following code: ..