Some of you may have seen “The ‘<name>’ task processor queue reached <number> scheduled tasks.” warning messages, wondered what they mean, and if there is anything you can do about them.
Task processors have been in Asterisk for a long time. Since Asterisk v12, they have become more important because the PJSIP, sorcery, and stasis modules heavily rely on task processors. As a result, the symptoms of overloading your Asterisk installation have changed. An early symptom of an overloaded system is when task processors start queueing an excessive number of tasks to process. If the overload conditions are not relieved then later symptoms can include high CPU utilization and/or high memory consumption due to extremely large queues.
You can see some task processor statistics by executing the CLI command “core show taskprocessors”. Starting in Asterisk v13.8.0 task processors created by PJSIP, sorcery, and stasis have meaningful names instead of an opaque UUID string. Meaningful task processor names help identify modules in Asterisk that are bottle necks on your system.
pjsip/aor/<aor-name>
Task processor created by res_pjsip_registrar.so for incoming endpoint registrations. Asterisk v13.10.0 eliminated these task processors to continue using the pjsip/distributor task processor the REGISTER request came in on.
pjsip/default
Task processor created by res_pjsip.so. Miscellaneous PJSIP tasks that don’t have any other association. These task processors are assigned tasks in a round robin sequence. Before Asterisk v13.10.0 these task processors also handled the initial PJSIP message distribution if the message was not associated with any other task processor.
pjsip/distributor
Task processor created by res_pjsip.so. PJSIP distributes incoming messages to these task processors if the message is not associated with any other task processor. The task processor assignment is determined by hashing the SIP request’s Call-ID and From tag. These task processors are new since Asterisk v13.10.0.
pjsip/messaging
Task processor created by res_pjsip_messaging.so to send all out-of-dialog MESSAGE requests for PJSIP.
pjsip/outreg/<endpoint>
Task processor created by res_pjsip_outbound_registration.so to handle the endpoint’s outbound registration messages.
pjsip/session/<endpoint>
Task processor created by res_pjsip_session.so to handle incoming and outgoing sessions and their associated dialog messages. Asterisk v13.10.0 renamed the task processors for outgoing sessions to pjsip/outsess/<endpoint>. The task processors for incoming sessions were eliminated to continue using the pjsip/distributor task processor the initial INVITE came in on.
pjsip/outsess/<endpoint>
Task processor created by res_pjsip_session.so to handle messages associated with the endpoint’s outgoing session (Outgoing channels). This includes any messages associated with the session’s dialog.
pjsip/pubsub/<endpoint>
Task processor created by res_pjsip_pubsub.so to handle the endpoint’s outbound PUBLISH and SUBSCRIBE dialog messages. Asterisk v13.10.0 stopped using these task processors for incoming PUBLISH and SUBSCRIBE dialogs to continue using the pjsip/distributor task processor the original request came in on.
pjsip/refer/<endpoint>
Task processor created by res_pjsip_refer.so to handle the endpoint’s outbound REFER requests.
pjsip/websocket
Task processor created by res_pjsip_transport_websocket.so to process WebRTC HTTP requests for PJSIP.
sorcery/<type>
Used to process sorcery observer callbacks. Sorcery task processors use a self managed thread pool.
subm:<topic>
Stasis topic subscription mailbox. The task processor has a dedicated thread to process the task queue.
subp:<topic>
Stasis topic subscription thread pool mailbox. The task processor uses an available thread from the stasis thread pool to process the task queue.
What can you do?
* Disable features you do not need. It is important that if you do not need features like Homer(HEP), CDR, CEL, or AMI that you do not enable them and for HEP you shouldn’t load those modules either.
* You can adjust the thread pool parameters for PJSIP and stasis task processors. The PJSIP thread pool is configured in pjsip.conf. The stasis thread pool is configured in stasis.conf.
* If the task processor name happens to be of the ‘subm:<topic>’ form where the topic is your stasis application name then that task processor is servicing the stasis message bus communication with your application. One thing you can try is to register multiple copies of your stasis application and randomly spread calls to the different copies of the application. e.g., my_stasis_app1, my_stasis_app2,…
There may be other things you could do but they likely depend more on your call patterns and the services your Asterisk installation is performing.
5 Responses
I get the following warning on asterisk log
taskprocessor.c: The ‘subm:cdr_engine-00000003’ task processor queue reached 5000 scheduled tasks again. Asterisk is configured with real time sip and we use fastagi (in JAVA) to read dialplan from an mysql database. With 20 cps if it reaches to 400 calls the calls start to retransmit, If a try to make a new call from zoiper the status will be trying,after 40-60 second everything goes back to normal. I have the same configuration on another server, as far as I know because it’s not me who has configured it and it works ok even with 30-40 cps, No retransmission. If I disable the cdr module everything is ok.Also I removed fastagi and used the extensional.conf file and still the same problem.
But what if we are not using pjsip in Asterisk 16.5 and still we are getting
taskprocessor.c: The ‘subm:cdr_engine-00000003’ task processor queue reached 5000 scheduled tasks again
That message means that the CDR handling has backed up in some manner. This can be due to, for example, a connected database responding slowly or the over all load of the system. One option is to enable batching in the cdr.conf file.
What does this mean then and is there a way to fix it? This is during non-peak time (i.e. there are times that the system is way more busy)
[2020-08-19 08:37:50] WARNING[15135] taskprocessor.c: The ‘stasis/m:cache_pattern:0/endpoint:all-00000008’ task processor queue reached 500 scheduled tasks again.
[2020-08-19 08:37:50] WARNING[22700] taskprocessor.c: The ‘stasis/m:manager:core-00000007’ task processor queue reached 3000 scheduled tasks again.
[2020-08-19 08:37:51] WARNING[15135] taskprocessor.c: The ‘stasis/m:cache_pattern:0/endpoint:all-00000008’ task processor queue reached 500 scheduled tasks again.
[2020-08-19 08:37:51] WARNING[22700] taskprocessor.c: The ‘stasis/m:manager:core-00000007’ task processor queue reached 3000 scheduled tasks again.
I would suggest reaching out on the community site[1] for help. Generally speaking, though, it does mean that a lot of stuff is happening at the time – including some manager activity.
[1] https://community.asterisk.org/