Hello guys,
killing the processes solved the issue and the new dev file wasn't growing rapidly anymore. But just about an hour ago, the system got completely stuck and it was not possible to open the login window in SAP GUI. It looked exactly as a system with a stuck archiver.
I was thinking what to do before restarting everything, so I checked with dpmon - all the dialog processes were taken by RFC connections from the Solution Manager, there was just one employee logged on:
-->
Workprocess Table (long) Wed May 13 09:25:37 2015
========================
No Type Pid Status Cause Start Rstr Err Sem Time Program Cl User Action Table
-------------------------------------------------------------------------------------------------------------------------------
0 DIA 23290 Run yes no 0 0 2695 SAPLSICM 100 SM_SMP NO_ACTION
1 DIA 23291 Run yes no 0 0 5043 SAPLSHTTP 100 SM_SMP NO_ACTION
2 DIA 23292 Run yes no 0 0 8699 SAPLSHTTP 100 SM_SMP NO_ACTION
3 DIA 23293 Run yes no 0 0 2313 SAPLSICM 100 SM_SMP NO_ACTION
4 DIA 23294 Run yes no 0 0 3238 SAPLSICM 100 SM_SMP NO_ACTION
5 DIA 21566 Run yes no 1 0 5342 SAPLSICM 100 SM_SMP NO_ACTION
6 DIA 23296 Run yes no 0 0 4024 SAPLSHTTP 100 SM_SMP NO_ACTION
7 DIA 23297 Run yes no 0 0 5702 SAPLSICM 100 SM_SMP NO_ACTION
8 DIA 23298 Run yes no 0 0 6058 SAPLSICM 100 SM_SMP NO_ACTION
9 DIA 23299 Run yes no 0 0 4979 SAPLSICM 100 SM_SMP NO_ACTION
10 DIA 23300 Run yes no 0 0 8876 SAPLSICM 100 SM_SMP NO_ACTION
11 DIA 23301 Run yes no 0 0 410 100 <SOME EMPLOYEE> NO_ACTION
12 DIA 23302 Run yes no 0 0 4620 SAPLSICM 100 SM_SMP NO_ACTION
13 DIA 23303 Run yes no 0 0 4262 SAPLSICM 100 SM_SMP NO_ACTION
14 DIA 23304 Run yes no 0 0 6422 SAPLSICM 100 SM_SMP NO_ACTION
15 DIA 23305 Run yes no 0 0 7684 SAPLSHTTP 100 SM_SMP NO_ACTION
After killing a few processes, suddenly I could login. SM04 was showing a LOT of SM_SMP processes logged on:
![]()
I wanted to disconnect them, but it wasn't possible, so I had to kill all the dialog WP using kill -9. I had locked the SM_SMP user as a precaution, in order to avoid additional connections and so I could restore normal operations on the system. But now the question is what went wrong... A buggy diagnostics agent or what... ?
Does anyone know the issue ? Or I should probably open an OSS message and get this analyzed by SAP. Btw. here is also an excerpt from the dev_w4 file:
M Wed May 13 10:23:14 2015
M ***LOG R49=> ThReceive, CPIC-Error (020223) [thxxhead.c 7927]
M ***LOG R5A=> ThReceive, CPIC-Error (25554178) [thxxhead.c 7933]
M ***LOG R64=> ThReceive, CPIC-Error ( CMSEND(SAP)) [thxxhead.c 7938]
A RFC 3710 CONVID 25554178
A * CMRC=20 DATA=1 STATUS=1 SAPRC=223 ThSAPCMRCV
A RFC> ABAP Programm: SAPMSSY1 (Transaction: )
A RFC> User: CSERKO20 (Client: 100)
A RFC> Destination: wsps450_E (handle: 1, DtConId: 0E40F9E4E3AFF160A520402CF4CB710E, DtConCnt: 0, ConvId: 25554178,{0E40F9E4-E3AF-F
A RFC> Called function module: RSWAD_URL_GET
A RFC SERVER> RFC Server Session (handle: 1, 25554178, {0E40F9E4-E3AF-F160-A520-402CF4CB710E})
A RFC SERVER> Caller host:
A RFC SERVER> Caller transaction code: (Caller Program: BExQueryDesignerStarter)
A RFC SERVER> Called function module: RSWAD_URL_GET
A *** ERROR => RFC ======> CPIC-CALL: 'ThSAPCMRCV' : cmRc=20 thRc=223
CPIC program connection ended (read error)
[abrfcio.c 9213]
A {root-id=35353532464137303535353246413730}_{conn-id=00000000000000000000000000000000}_0
A *** ERROR => RFC Error RFCIO_ERROR_SYSERROR in abrfcpic.c : 3712
CPIC-CALL: 'ThSAPCMRCV' : cmRc=20 thRc=223
CPIC program connection ended (read error)
[abrfcio.c 9213]
A {root-id=35353532464137303535353246413730}_{conn-id=00000000000000000000000000000000}_0
A *** ERROR => RFC Error RFCIO_ERROR_MESSAGE in abrfcio.c : 1987
[abrfcio.c 9213]
A {root-id=35353532464137303535353246413730}_{conn-id=00000000000000000000000000000000}_0
M
M ThAlarmHandler: first alarm, just set controls
M
M ThAlarmHandler: (2)
M ThAlarmHandler: inside critical section after 2 tries
M C-STACK
[0] DoStack2, at 0xc57ead
[1] CTrcStack2, at 0xc57b43
[2] CTrcStack, at 0xc57aec
[3] ThAlarmHandler, at 0xa9924e
[4] DpSigAlrm, at 0xa2fa44
[5] __sighndlr, at 0xfffffd7ff998ddd6
[6] call_user_handler, at 0xfffffd7ff99826a2
[7] sigacthandler, at 0xfffffd7ff99828ce
[8] ????????, at 0xffffffffffffffff
[9] fast_process_lock, at 0xfffffd7ff9986690
[10] mutex_lock_impl, at 0xfffffd7ff9986842
[11] mutex_lock, at 0xfffffd7ff998687b
[12] MtxILock, at 0xa63d29
[13] MtxLock_SPIN, at 0xa6483f
[14] MpiIEvtOpen, at 0x28e6e23
[15] MpiICreate, at 0x28dc123
[16] ThPlgCreate2, at 0xae366d
[17] ThICMGetStatus, at 0xbec526
[18] ThHdlICMOpcode, at 0xbebb64
[19] ThSysInfo, at 0xbdfc96
[20] __1cIab_jcaly6F_v_, at 0x102739b
[21] __1cIab_extri6F_i_, at 0xe8bf60
[22] __1cJab_xevent6FpkH_i_, at 0xf2e47c
[23] ab_dstep, at 0xe808b9
[24] dynpmcal, at 0xc83ae7
[25] dynppbo0, at 0xc80416
[26] dynprctl, at 0xc7fccb
[27] dynpen00, at 0xc7c117
[28] TskhLoop, at 0xa6f128
[29] ThStart, at 0xa674b2
[30] DpMain, at 0x9b6528
M
M ThAlarmHandler: return for next chance
M
M ThAlarmHandler: (3)
M ThAlarmHandler: inside critical section after 3 tries
M C-STACK
[0] DoStack2, at 0xc57ead
[1] CTrcStack2, at 0xc57b43
[2] CTrcStack, at 0xc57aec
[3] ThAlarmHandler, at 0xa9924e
[4] DpSigAlrm, at 0xa2fa44
[5] __sighndlr, at 0xfffffd7ff998ddd6
[6] call_user_handler, at 0xfffffd7ff99826a2
[7] sigacthandler, at 0xfffffd7ff99828ce
[8] ????????, at 0xffffffffffffffff
[9] fast_process_lock, at 0xfffffd7ff9986690
[10] mutex_lock_impl, at 0xfffffd7ff9986842
[11] mutex_lock, at 0xfffffd7ff998687b
[12] MtxILock, at 0xa63d29
[13] MtxLock_SPIN, at 0xa6483f
[14] MpiIEvtOpen, at 0x28e6e23
[15] MpiICreate, at 0x28dc123
[16] ThPlgCreate2, at 0xae366d
[17] ThICMGetStatus, at 0xbec526
[18] ThHdlICMOpcode, at 0xbebb64
[19] ThSysInfo, at 0xbdfc96
[20] __1cIab_jcaly6F_v_, at 0x102739b
[21] __1cIab_extri6F_i_, at 0xe8bf60
[22] __1cJab_xevent6FpkH_i_, at 0xf2e47c
[23] ab_dstep, at 0xe808b9
[24] dynpmcal, at 0xc83ae7
[25] dynppbo0, at 0xc80416
[26] dynprctl, at 0xc7fccb
[27] dynpen00, at 0xc7c117
[28] TskhLoop, at 0xa6f128
[29] ThStart, at 0xa674b2
[30] DpMain, at 0x9b6528
M
M ThAlarmHandler: return for next chance
M
M ThAlarmHandler: (4)
M ThAlarmHandler: inside critical section after 4 tries
M C-STACK
[0] DoStack2, at 0xc57ead
[1] CTrcStack2, at 0xc57b43
[2] CTrcStack, at 0xc57aec
[3] ThAlarmHandler, at 0xa9924e
[4] DpSigAlrm, at 0xa2fa44
[5] __sighndlr, at 0xfffffd7ff998ddd6
[6] call_user_handler, at 0xfffffd7ff99826a2
[7] sigacthandler, at 0xfffffd7ff99828ce
[8] ????????, at 0xffffffffffffffff
[9] fast_process_lock, at 0xfffffd7ff9986690
[10] mutex_lock_impl, at 0xfffffd7ff9986842
[11] mutex_lock, at 0xfffffd7ff998687b
[12] MtxILock, at 0xa63d29
[13] MtxLock_SPIN, at 0xa6483f
[14] MpiIEvtOpen, at 0x28e6e23
[15] MpiICreate, at 0x28dc123
[16] ThPlgCreate2, at 0xae366d
[17] ThICMGetStatus, at 0xbec526
[18] ThHdlICMOpcode, at 0xbebb64
[19] ThSysInfo, at 0xbdfc96
[20] __1cIab_jcaly6F_v_, at 0x102739b
[21] __1cIab_extri6F_i_, at 0xe8bf60
[22] __1cJab_xevent6FpkH_i_, at 0xf2e47c
[23] ab_dstep, at 0xe808b9
[24] dynpmcal, at 0xc83ae7
[25] dynppbo0, at 0xc80416
[26] dynprctl, at 0xc7fccb
[27] dynpen00, at 0xc7c117
[28] TskhLoop, at 0xa6f128
[29] ThStart, at 0xa674b2
[30] DpMain, at 0x9b6528
M
M ThAlarmHandler: return for next chance
Thank you!