Socket Programming between Django REST API, VUE and Scrapy

Write now my code is able to successfully send a post request to the Django REST API which in result triggers the spider respectively and stores its output to the database successfully. For this cause I am using scrapyd API as you can see in the below code snippet

@api_view(['POST'])
@permission_classes((permissions.AllowAny,))
def crawlRottenTomatoes(request):
    print("in crawl tottentomatoes method")
    url = request.data["url"]

    if not url:
        return JsonResponse({'error': 'Missing  args'})

    if not is_valid_url(url):
        return JsonResponse({'error': 'URL is invalid'})
    domain = urlparse(url).netloc
    msg_dic = {}
    try:
        scrapyd.schedule(project="movie_spider", spider="rottentomatoes", url=url, domain=domain)
        msg_dic['msg'] = "Spider RottenTomatoes for given url is up and running"
    except Exception as e:
        print("exception")
        msg_dic['error'] = "Error running spider for RottenTomatoes"
    return JsonResponse(msg_dic, safe=False)

But now what I want. Is to have some kind of response back from the scrapyd when it’s done crawling and parsing the website/channel for that I came across the WebSockets. I tried to use WebSockets but the problem is scrapyd is a demon itself. I am not really able to send a message to the WebSocket client from the scrappy. Does anyone have any idea of how do to do or can share some resources which can help me with this?

Source: Python Questions

LEAVE A COMMENT