You may be finding yourself in a situation whereby the batch server status shows as "Running" but there are no batch jobs being processed and want to know how to track this down!

This issue has been reported to Sage Support in the past and there have been attempts to reproduce this issue, but sadly the issue has always been very intermittent so its been difficult to ascertain root cause.  Even worse, due to the weeks or months between occurrences it is also difficult to see if any changes or updates introduced have made any difference.

UPDATE - 29th January 2025 :  I am pleased to report there is now a fix made to Syracuse which is designed to overcome this issue.  See alert https://communityhub.sage.com/gb/sage-x3/f/announcements/241696/alert-syracuse-server-12-20-13-sage-x3-2024-r1-and-syracuse-server-12-21-4-sage-x3-2024-r2-are-now-available

Only if this fix does not help your situation should the remaining steps be considered

This article will provide the current troubleshooting steps and any other recommendations to confirm the symptoms and mitigate the issue.  I will be keeping this article up to date when the situation changes.

NOTE: this article is most relevant for X3 2024 R1 (12.0.35) and 2023 R2 (12.0.34) but may be useful for other versions also.

First steps are to confirm you have the same issue.  The current bug number being investigated by Sage Support is X3-322141 for which the symptoms are:

• The batch server status is "Running"
• No batch jobs are currently being processed, even though jobs are eligible to have been run
• No relevant errors showing in the Syracuse log files

The immediate workaround to restore service is to:

• Stop then restart the batch server process.  You can do this in Administration, Administration, End Points, Batch Server function.
• If this does not restart the batch jobs processing, you will then need to Stop and Restart the Syracuse service.  This will need an outage for your users of course.

Then setup additional logging.  The messages that are generated when this issue occurs are unfortunately considered warnings rather than errors, and the default log level is errors only.  

• Navigate to Administration, Administration, Settings, Global settings
  • Set "Maximum file size (MB)" to 200 and "Maximum number of days" to 15
  • Under code "batch" set "server" and "x3query" to "Info" level
  • Under code "memory" set this to "Debug" level

When you save these changes they take effect immediately

...

With the above logging enabled, when the problem occurs you will then see the info level message "Batch_X3APPL_running previous loop not ended !!!" to confirm the issue.  

The next steps could also be done straightaway, especially if the issue is a relatively rare event but you may decide to wait until you see the above message before continuing.   Of course you will be testing these steps on your TEST server to confirm they work as expected, but also take full backups before implementing on LIVE as normal.

1.  Eliminate the possibility of the Batch Server record being corrupted in the Mongo database

These steps should only be attempted by a suitably experienced consultant. 

Recreate the batch server record.  
- Stop the batch server in Sage X3
- Export "BatchServer" collection from MongoDB as a backup.   You can use mongo command line or GUI tool such as Studio 3T (https://studio3t.com/download/)
- Manually remove "BatchServer" collection from MongoDB with the command:
db.BatchServer.drop()
- In Sage X3, query back the X3 solution in Administration, Administration, Endpoints, X3 Solutions.  Then click the link in "Description" column to view the details.  Check the BatchServer collection has been automatically recreated in MongoDB.
- Restart batch server

2. Change the Batch Server configuration

Navigate to Administration, Administration, Endpoints, Batch server

- Re-implement any changes you may have lost from the above step, such as "X3 runtime tags", etc.
- Change "Time between two searches (sec)" to be at least 5 seconds greater than "Timeout search time (sec)" parameter, preferably by changing "Time between two searches (sec)" to be 35 seconds assuming it is currently set to the default 30 seconds.
- Ensure "Maximum delay to launch a query (minutes)" is set to 0 (Zero)

At the time of writing, it is felt the above two steps may possibly provide a long term solution but have not yet had confirmation this is the case.

Hopefully this article will guide you to a solution if you are experiencing the problem described.

Let me know any comments below...