I have a simple python script that I would like to run thousands of it’s instances on GCP (at the same time). This script is triggered by the $Universe scheduler, something like "python main.py –date ‘2022_01’". What architecture and technology I have to use to achieve this. PS: I cannot drop $Universe but I’m not ..
We are in the processing of setting up a system with front end Dashboard in React, Python Flask API hosted in Azure Kubernetes as Docker containers as Middle Layer and Azure SQL Database as backend. The SQL Tables are loaded from Databricks Delta Tables and have millions of records and potentially big size. The Business ..
Consider the following (very simplified) project structure: project/ src/ collections/ validators/ foo/ bar/ The "collections" sub-directory name shadows the Python’s "collections" module. The "validators" sub-directory name shadows "validators" 3rd-party package name. Question: Is it an anti-pattern to have sub-directories with the same names as Python modules or 3rd-party packages? Note, the real project structure has ..
I’m working on a project involving a lot of api calls and heavy calculation. here is an image of the actual architecture I have a total of 4 thread pools and some queues to allow communication between these threads. The problem is that one of these ThreadPool (cf_worker in the image) should actually perform CPU-bound ..
When I design a new application including interaction with a user, I always start by the data analysis and the relation between the different pieces of this data. There is most often a need to keep the different pieces of this data coherent, e.g. if you change a parameter 1, then the parameters 2, and ..
I’m receiving a bunch of data via. osc in my application and the ‘osc-receiver’ part runs in a different process. Assuming the applications consists of: Process A (GUI) Process B (Osc-Receiver) Now, since i need to handle multiple values that are send to different addresses, i’d ideally like to have a datastructure that the retrieved ..
I’m currently quite undecided on what is actually the best approach to tackle this problem. Assuming the program only consists of: GUI using imgui and glfw OSC Server that listens for incoming messages The gui cannot block and the osc server constantly needs to be able receive new messages. So first of all, would it ..
I have a scraping module on my application that uses Beautiful Soup and Selenium to get website info with this function: def get_page(user: str) -> Optional[BeautifulSoup]: """Get a Beautiful Soup object that represents the user profile page in some website""" try: browser = webdriver.Chrome(options=options) wait = WebDriverWait(browser, 10) browser.get(‘https://somewebsite.com/’ + user) wait.until(EC.presence_of_element_located((By.TAG_NAME, ‘article’))) except TimeoutException: ..
I’ve started working with Raspberry Pi and I’ve faced with architecture problem. I want to make python app which blink the LED using GPIO and have Web interface to start and stop blinking. In the Web there are a lot of examples of how to use Flask to make one action (e.g. enable LED, disable ..
I am working on an application that turns client’s geographic polygon files (geojson, shapefile, kml) and do some hard processing (temporal and spatial interpolation) using these data. The hard processing should be done on the application server side and when it is finished, the interpolation results should be available to the user as maps, charts ..