Is it possible to return the accumulated results from a celery chain?

  celery, python

I’m trying to make a word counter in celery, however it doesn’t seem to return the results of the final task.

I’ve tried many things, but it seems I’m at a loss here.

I would expect to receive the output from collect_results, but it doesn’t show when I collect or get from the word_counter.delay() task.

simple-input.txt:

cat dog dog car
dog car cat
car wut hello

Tasks:

@app.task(trail=True)
def collect_results(data):
    result = Counter()

    for d in data:
        print(d)
        for k, v in d.items():
            result[k] += v

    return result


@app.task(trail=True)
def word_counter():
    document = open('simple-input.txt', 'r').read()
    lines = document.splitlines()

    job = (group(process_chunk.s(l) for l in lines) | collect_results.s())()

    return job

Test script:

from pprint import pprint
from tasks import word_counter
result = word_counter.delay()
pprint([v for v in result.collect()])

Expected output:
Something containing the output of "collect_results()"

Actual output:

[(<AsyncResult: 9f515c1b-7538-494a-bd9d-59aaf9697d66>,
  [['e5941f90-6d17-4cca-a9d1-6d498b962f7f',
    [['eaf56b9c-60ff-417b-9c93-c79ed98379b6', None],
     [[['27524ce6-bb91-4f11-a141-f6bd14987bb1', None], None],
      [['4795eef4-b7cc-4a25-ace6-8bde0466d23f', None], None],
      [['0b5175b3-d94f-455a-ad58-3590babfe983', None], None]]]],
   None]),
 (<GroupResult: eaf56b9c-60ff-417b-9c93-c79ed98379b6 [27524ce6-bb91-4f11-a141-f6bd14987bb1, 4795eef4-b7cc-4a25-ace6-8bde0466d23f, 0b5175b3-d94f-455a-ad58-3590babfe983]>,
  [{'car': 1, 'cat': 1, 'dog': 2},
   {'car': 1, 'cat': 1, 'dog': 1},
   {'car': 1, 'hello': 1, 'wut': 1}]),
 (<AsyncResult: 27524ce6-bb91-4f11-a141-f6bd14987bb1>,
  {'car': 1, 'cat': 1, 'dog': 2}),
 (<AsyncResult: 4795eef4-b7cc-4a25-ace6-8bde0466d23f>,
  {'car': 1, 'cat': 1, 'dog': 1}),
 (<AsyncResult: 0b5175b3-d94f-455a-ad58-3590babfe983>,
  {'car': 1, 'hello': 1, 'wut': 1})]

Source: Python Questions

LEAVE A COMMENT