x Want to participate? Join the group to interact
  • State Verified Answer
  • Replies 7 replies
  • Subscribers 137 subscribers
  • Views 838 views
  • Users 0 members are here
Browse Forums
Announcements
237
topics
29
replies
General Discussion
5k
topics
13k
replies
Sage Data & Analytics
56
topics
95
replies
Sage Enterprise Intelligence
211
topics
418
replies
Sage Intelligence Reporting
121
topics
322
replies
chunheng

Need help on Batch task being canceled in 2 hours

SOLVED
Posted By

Hi all,

I am having issue with Batch task consistently entering Canceled state by NodeJS server after 2 hours.

How do I resolve this?

Additionally, I have `write ECONNRESET` error while writing this post crosschecking details on an interactive web session.

Environment context:

Sage X3 v12 p33.
Windows Server 2022

AWS ALB, Duration-based stickiness of 1 hour.

The runtime is located at a runtime-dedicated server instance. One runtime only at the moment; no Application Clustering. 

Syracuse is a cluster of three servers of 2 syracuse web nodes each.

Nodelocal.js is using default settings; no changes.

Windows Registry is default install; no changes, yet.

  • +1
    verified answer

    https://us-kb.sage.com/portal/app/portlets/results/viewsolution.jsp?solutionid=220924460105883

    https://www.tenable.com/audits/items/CIS_Microsoft_Windows_Server_2019_STIG_v1.0.1_L2_DC.audit:a12e8348f03f169cc6141a1e21cf9583

    https://portal.perforce.com/s/article/1208

    https://us-kb.sage.com/portal/app/portlets/results/viewsolution.jsp?solutionid=221924360114800&hypermediatext=null

    NodeJS:

    online-help.sageerpx3.com/.../

    Additional reference found while looking at communityhub search: 

    -  Syracuse Node.js keepalive settings 

    - communityhub.sage.com/.../econnreset-error-in-batch-server

    - Sage ref:  Performance tuning your Sage X3 system: Network 

    -  Syracuse Load balancing and Node.JS (V9U4) 

  • 0 in reply to chunheng

    Rename this `.txt` to `.reg`.

    Fullscreen keepAlive.txt Download 
    Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"KeepAliveTime"=dword:05265c00
"KeepAliveInterval"=dword:00007530
"TcpMaxDataRetransmissions"=dword:0000000a

    All Sage X3 servers, run this registry edit.

    Restart all Sage X3 servers.

    For reference, registry path at: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

    P.S. The keepalive value (24 hours) I set on the registry code is terrible practice (connections that are not disposed by program are kept there for 24 hours by windows) but I need to run a batch task on a Windows process to idle that may need up to 24 hours.

    EDIT: This is not the solution to the issue. It is simply a red herring and a way to delay for the resolution. Some batch tasks were noted to not respond at all.

  • 0 in reply to chunheng

    Progress details:



    - OS registry change at all servers except DB server where keepAliveTimeout is 24 hours, no NodeJS changes: Success. Completed in 8 hours

    - OS registry change at all servers except DB server where keepAliveTimeout is 10 minutes instead, no NodeJS changes: Killed in 10 minutes flat.

    - OS registry change at Application Server where keepAliveTimeout is 24 hours, no NodeJS changes: OS kill process.

    - No OS registry changes, no NodeJS changes: OS kill process.
    - No OS registry changes, no NodeJS changes: OS kill process.

  • 0 in reply to chunheng

    From Syracuse log:

    2023-10-29T14:52:44.217Z | cbddab2c3554 |   |      | x3Com.client | error | [stream cvgClient-a82cb916a8d7 pid=2020] stream error: graceful=false: Error: read ECONNRESET
2023-10-29T14:52:44.219Z | cbddab2c3554 |   |      | x3Com.client | error | [stream cvgClient-a82cb916a8d7 pid=2020] read Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-10-29T14:52:44.219Z | cbddab2c3554 |   |      | classic.protocol | error | x3Automata.process null Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-10-29T14:52:44.220Z | cbddab2c3554 |   |      | classic.action | error | cvgClient-a82cb916a8d7 x3Pid=2020 used classicPage  func=AREQUETE  processAction {
  "cvgClient": "501217bb-1fed-4326-9d6f-2979830620c2",
  "isClassicPage": true
} read ECONNRESET Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-10-29T14:52:44.220Z | cbddab2c3554 |   |      | x3Com.client | error | cvgClient-a82cb916a8d7 x3Pid=2020 clientClosing classicPage  func=AREQUETE  closeClient reason 102 isAlive false Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-10-29T14:52:44.220Z | cbddab2c3554 |   |      | classic.action | error | cvgClient-a82cb916a8d7 x3Pid=2020 clientClosed classicPage  func=AREQUETE  processRequest Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-10-29T14:52:44.221Z | cbddab2c3554 |   |      | x3Com.client | error | [stream cvgClient-a82cb916a8d7 pid=2020] stream close: graceful=false 
2023-10-29T14:52:48.217Z | cbddab2c3554 |   |      | classic.tracker | error | cvgClient-a82cb916a8d7 x3Pid=2020 stopped classicPage  func=AREQUETE -cvgTracker-87c798fd-f0aa-410e-9a79-03db4cde487d-requestSvc track called ontimeout Cannot read properties of null (reading 'sendContext') TypeError: Cannot read properties of null (reading 'sendContext')
    at CvgClient.generateResponse (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\cvgPage\cvgClient.jsc:1203:40)
    at CvgTracker.track (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\cvgPage\tracker\cvgTracker.jsc:58:43)
    at D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\cvgPage\tracker\cvgTracker.jsc:42:28
    at AsyncLocalStorage.run (node:async_hooks:330:14)
    at Globals.withContext (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\streamlineLib\streamline-runtime\globals.es5:22:39)
    at Timeout.<anonymous> (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\cvgPage\tracker\cvgTracker.jsc:41:42)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7)

  • 0 in reply to chunheng

    Additional reading: https://connectreport.com/blog/tuning-http-keep-alive-in-node-js/

    From syracuse log (ignore first 13 lines, it was a password reset mistake).

    2023-11-02T07:33:35.237Z | 88064700201b |   | 26521 | login.auth | error | 307 Your password has expired,you have to change it null /auth/changePassword/page
2023-11-02T07:33:35.240Z | 88064700201b |   | 26521 | session | error | ensureAuthenticated login=admin /auth/login/submit Your password has expired,you have to change it
2023-11-02T07:33:45.182Z | 88064700201b |   | 26528 | login.auth | error | 400 You cannot reuse the same password null /auth/changePassword/page
2023-11-02T07:33:45.183Z | 88064700201b |   | 26528 | login.auth | error | dispatcher /auth/changePassword/submit You cannot reuse the same password Error: You cannot reuse the same password
    at error (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\auth\helpers.jsc:14:17)
    at submit (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\auth\changePassword.jsc:59:30)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\auth\helpers.jsc:196:21
    at async D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\auth\helpers.jsc:196:21
    at async D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\http\httpRequestOverrider.jsc:34:20
    at async D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\streamlineLib\ez-streams\node-wrappers.es5:605:13
    at async WebServerRouter.dispatchRequest (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\webServer\webServerRouter.jsc:630:21)
    at async D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\webServer\webServerProcess.jsc:143:21
2023-11-02T07:45:23.135Z | 88064700201b |   |      | x3Com.client | error | [stream pid=3120] stream error: graceful=false: Error: read ECONNRESET
2023-11-02T07:45:23.136Z | 88064700201b |   |      | x3Com.client | error | [stream pid=3120] stream close: graceful=false 
2023-11-02T07:45:40.263Z | 88064700201b |   |      | x3Com.client | error | [stream pid=1960] stream error: graceful=false: Error: read ECONNRESET
2023-11-02T07:45:40.263Z | 88064700201b |   |      | x3Com.client | error | [stream pid=1960] stream close: graceful=false 
2023-11-02T07:47:47.607Z | 88064700201b |   |      | x3Com.client | error | [stream cvgClient-c963a6931f14 pid=6080] stream error: graceful=false: Error: read ECONNRESET
2023-11-02T07:47:47.608Z | 88064700201b |   |      | x3Com.client | error | [stream cvgClient-c963a6931f14 pid=6080] read Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:47:47.609Z | 88064700201b |   |      | classic.protocol | error | x3Automata.process null Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:47:47.609Z | 88064700201b |   |      | classic.action | error | cvgClient-c963a6931f14 x3Pid=6080 used classicPage  func=AREQUETE  processAction {
  "cvgClient": "dde27f70-f480-4739-9db8-88d02efd574e",
  "isClassicPage": true
} read ECONNRESET Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:47:47.609Z | 88064700201b |   |      | x3Com.client | error | cvgClient-c963a6931f14 x3Pid=6080 clientClosing classicPage  func=AREQUETE  closeClient reason 102 isAlive false Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:47:47.609Z | 88064700201b |   |      | classic.action | error | cvgClient-c963a6931f14 x3Pid=6080 clientClosed classicPage  func=AREQUETE  processRequest Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:47:47.609Z | 88064700201b |   |      | x3Com.client | error | [stream cvgClient-c963a6931f14 pid=6080] stream close: graceful=false 
2023-11-02T07:47:51.606Z | 88064700201b |   |      | classic.tracker | error | cvgClient-c963a6931f14 x3Pid=6080 stopped classicPage  func=AREQUETE -cvgTracker-2840c85a-1494-4087-8668-177fec548692-requestSvc track called ontimeout Cannot read properties of null (reading 'sendContext') TypeError: Cannot read properties of null (reading 'sendContext')
    at CvgClient.generateResponse (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\cvgPage\cvgClient.jsc:1203:40)
    at CvgTracker.track (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\cvgPage\tracker\cvgTracker.jsc:58:43)
    at D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\cvgPage\tracker\cvgTracker.jsc:42:28
    at AsyncLocalStorage.run (node:async_hooks:330:14)
    at Globals.withContext (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\streamlineLib\streamline-runtime\globals.es5:22:39)
    at Timeout.<anonymous> (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\cvgPage\tracker\cvgTracker.jsc:41:42)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7)
2023-11-02T07:52:12.739Z | 88064700201b |   |      | x3Com.client | error | [stream cvgClient-7d47156dad7b pid=1640] stream error: graceful=false: Error: read ECONNRESET
2023-11-02T07:52:12.739Z | 88064700201b |   |      | x3Com.client | error | [stream cvgClient-7d47156dad7b pid=1640] read Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:52:12.739Z | 88064700201b |   |      | classic.protocol | error | x3Automata.process null Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:52:12.739Z | 88064700201b |   |      | classic.action | error | cvgClient-7d47156dad7b x3Pid=1640 used classicPage  func=DOSEXTRA  processAction {
  "cvgClient": "aea35491-ae67-4313-a7e9-2dfd317b8e23",
  "isClassicPage": true
} read ECONNRESET Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:52:12.739Z | 88064700201b |   |      | x3Com.client | error | cvgClient-7d47156dad7b x3Pid=1640 clientClosing classicPage  func=DOSEXTRA  closeClient reason 102 isAlive false Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:52:12.739Z | 88064700201b |   |      | classic.action | error | cvgClient-7d47156dad7b x3Pid=1640 clientClosed classicPage  func=DOSEXTRA  processRequest Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
2023-11-02T07:52:12.739Z | 88064700201b |   |      | x3Com.client | error | [stream cvgClient-7d47156dad7b pid=1640] stream close: graceful=false 
2023-11-02T07:52:16.000Z | 88064700201b |   |      | x3Com.dispatch | error | cvgProxy classicSessionNotFound 94a583dc-c2e4-43a7-a8e6-7d47156dad7b
2023-11-02T07:52:16.011Z | 88064700201b |   |      | x3Com.dispatch | error | doIt Classic session not found on this server (94a583dc-c2e4-43a7-a8e6-7d47156dad7b). 
This can be due to an unscheduled server restart. Error: Classic session not found on this server (94a583dc-c2e4-43a7-a8e6-7d47156dad7b). 
This can be due to an unscheduled server restart.
    at ConnectRoute.cvgProxy (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\connectRoute.jsc:79:27)
    at ConnectRoute.walk (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\connectRoute.jsc:16:21)
    at DatasetRoute.dispatch (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\routeDispatcher.jsc:33:37)
    at DatasetRoute.sessions (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\datasetRoute.jsc:61:28)
    at ContractRoute.dispatch (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\routeDispatcher.jsc:29:46)
    at ContractRoute.walk (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\contractRoute.jsc:29:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ApplicationRoute.dispatch (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\routeDispatcher.jsc:33:25)
    at async ApplicationRoute.walk (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\applicationRoute.jsc:21:17)
    at async FusionRoute.dispatch (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\routeDispatcher.jsc:33:25)
    at async FusionRoute.walk (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\fusionRoute.jsc:61:17)
    at async FusionRoute.dispatch (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\routeDispatcher.jsc:33:25)
    at async FusionRoute.dispatchRequest (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\x3Gateway\route\fusion\fusionRoute.jsc:25:18)
    at async authenticatedRoutes.trans (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\webServer\webServerRouter.jsc:101:13)
    at async D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\http\httpRequestOverrider.jsc:34:20
    at async D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\streamlineLib\ez-streams\node-wrappers.es5:605:13
    at async WebServerRouter.dispatchRequest (D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\webServer\webServerRouter.jsc:630:21)
    at async D:\Sage\SyracuseComponent\syracuse\bin\node_modules\@sage\syracuse-lib\src\webServer\webServerProcess.jsc:143:21
2023-11-02T08:13:08.680Z | ea5bd50c8eb3 |   |      | x3Com.client | error | [stream pid=4692] stream error: graceful=false: Error: read ECONNRESET
2023-11-02T08:13:08.680Z | ea5bd50c8eb3 |   |      | x3Com.client | error | [stream pid=4692] stream close: graceful=false 
2023-11-02T08:14:05.999Z | ea5bd50c8eb3 |   |      | x3Com.client | error | [stream cvgClient-9acaf9674c2e pid=7072] stream error: graceful=false: Error: read ECONNRESET
2023-11-02T08:14:05.999Z | ea5bd50c8eb3 |   |      | x3Com.client | error | [stream cvgClient-9acaf9674c2e pid=7072] stream close: graceful=false 
2023-11-02T08:14:47.040Z | 88064700201b |   |      | x3Com.client | error | [stream pid=4800] stream error: graceful=false: Error: read ECONNRESET
2023-11-02T08:14:47.040Z | 88064700201b |   |      | x3Com.client | error | [stream pid=4800] stream close: graceful=false

  • 0 in reply to chunheng

    Another thing I noticed:

    I cannot log out after a read ECONNRESET or Convergence failure after waiting for 10 minutes and pressing the Update button at Usage > Query Management.

    When I checked the user monitor list, syracuse started a new process and the old processes are still there.

*Community Hub is the new name for Sage City