SlickEdit Community
SlickEdit Product Discussion => SlickEdit® => Topic started by: rowbearto on November 27, 2019, 02:53:25 PM
-
At a Windows command prompt when I run a Java program, its behavior is different when pressing Ctrl-C vs pressing Ctrl-Break. Ctrl-Break generates a "thread dump" (https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr019.html), while Ctrl-C allows my shutdown handler to run and for my java code to gracefully exit according to my Java code.
When I execute my gradle java program inside of SlickEdit and I do stop_build or press Ctrl-C in the process buffer, it seems to be sending a Ctrl-Break to my Java program which is not what I want as I'm seeing a thread dump instead of my shutdown handler executed. I would rather have it send a Ctrl-C, as my desired behavior is to make my code terminate early and cleanup, not get a thread dump.
How to get SE to send a Ctrl-C and not Ctrl-Break?
-
ntpinit.exe is using this call:
GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT,process_id);
When I use this:
GenerateConsoleCtrlEvent(CTRL_C_EVENT,process_id);
Nothing gets stopped. Seems to have no effect at all. I don't know a work around for this. That's why SlickEdit uses Ctrl_BREAK_EVENT. It could be that this event doesn't get passed along to the child process (only CMD.EXE). Also could be it just doesn't work.
-
Is it possible that SlickEdit inside of ntpinit.exe has disabled (or never enabled) the passing of Ctrl-C down to child processes?
See: https://docs.microsoft.com/en-us/windows/console/setconsolectrlhandler
If ntpinit.exe did SetConsoleCtrlHandler(null, FALSE), it would ensure that Ctrl-C gets sent to child processes?
Maybe there is a reason why ntpinit.exe disables the sending of Ctrl-C to child processes?
I found a utility that allows me to send Ctrl-C in a similar way one can send SIGINT in Linux, "win-kill": https://github.com/alirdn/windows-kill .
When I open a command window outside of SlickEdit and run my java code, I am able to send it a Ctrl-C via win-kill from another window and it is indeed received and aborts my Java code in the way I expect (and I'm able to catch this in my java code).
But when I run the java program in the process buffer of SE by directly invoking "java ...." at the prompt in the process buffer, using win-kill to the PID of that java code doesn't seem to work.
When I look at the process tree in Process Explorer/Task Manager, I see SlickEdit invokes ntpinit.exe, which the invokes cmd.exe, and under that is java.exe.
So I suspect that it is ntpinit.exe that is blocking the sending of Ctrl-C to child processes somehow.
Also very interesting: https://stackoverflow.com/questions/813086/can-i-send-a-ctrl-c-sigint-to-an-application-on-windows
I implemented a java version of "win-kill" based on the "In Java, using JNA ..." solution on that java page, and it also is able to interrupt my java code in my own cmd, however that does not work when the Java code is launched inside SlickEdit.
So I suspect it is SlickEdit itself, possibly ntpinit.exe, that is blocking the Ctrl-C from getting to child processes.
-
Another question, when ntpinit.exe its child cmd.exe is spawned, is either one of them made the root of a new process group?
https://docs.microsoft.com/en-us/windows/console/console-process-groups
I remember a while back when there were similar issues in Linux it was required to create a new sessionid/process group. Maybe similar issue needed in windows too?
https://community.slickedit.com/index.php?topic=15406.0
Also interesting, from here: https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa?redirectedfrom=MSDN
When a process is created with CREATE_NEW_PROCESS_GROUP specified, an implicit call to SetConsoleCtrlHandler(NULL,TRUE) is made on behalf of the new process; this means that the new process has CTRL+C disabled. This lets shells handle CTRL+C themselves, and selectively pass that signal on to sub-processes. CTRL+BREAK is not disabled, and may be used to interrupt the process/process group.
-
Clark: Another question, you said ntpinit.exe is doing:
GenerateConsoleCtrlEvent(CTRL_C_EVENT,process_id);
What is the process_id that it is using? I'm assuming it is the process id of cmd.exe?
According to generateconsolectrlevent documentation (https://docs.microsoft.com/en-us/windows/console/generateconsolectrlevent):
dwProcessGroupId [in]
The identifier of the process group to receive the signal. A process group is created when the CREATE_NEW_PROCESS_GROUP flag is specified in a call to the CreateProcess function.
So I think it means that when ntpinit.exe spawns cmd.exe that cmd.exe is in a new process group? Otherwise you would need to send 0 as the process group as per the docs:
If this parameter is zero, the signal is generated in all processes that share the console of the calling process.
But then in my last post, we also saw:
When a process is created with CREATE_NEW_PROCESS_GROUP specified, an implicit call to SetConsoleCtrlHandler(NULL,TRUE) is made on behalf of the new process; this means that the new process has CTRL+C disabled. This lets shells handle CTRL+C themselves, and selectively pass that signal on to sub-processes. CTRL+BREAK is not disabled, and may be used to interrupt the process/process group.
So it is very possible that this cmd.exe is getting its ctrl-c disabled implicitly?
Maybe what is needed is a new process between ntpinit.exe and cmd.exe that is the process group leader, and this inbetween process calls
SetConsoleCtrlHandler(NULL,FALSE) to enable Ctrl-C handling and then spawns cmd.exe?
Or without this intermediate process, perhaps ntpinit.exe can be the process group leader but then when it sends the Ctrl-C event it can ignore it somehow while all other child processes get it?
This particular stack overflow answer also talks about an intermediate process: https://stackoverflow.com/questions/813086/can-i-send-a-ctrl-c-sigint-to-an-application-on-windows/2445728#2445728
-
Thanks for all this info. I'm still looking some of it over.
ntpinit.exe spawns CMD.EXE with the CREATE_NEW_PROCESS_GROUP flag.
I've trying messing with a number of things but no luck. I've even tried traversing the processes invoked by CMD.exe with no luck. TerminateProcess works but GenerateConsoleCtrlEvent doesn't work. Doesn't even work for CTRL_BREAK_EVENT. There must be some weird stuff going on here since it works when it's sent to CMD.EXE
-
Thanks Clark!
I just stumbled upon a few more interesting tidbits of info.
https://ss64.com/nt/start.html
When ntpinit spawns cmd.exe, is it using start /b? If /b flag used then:
Start application without creating a new window. In this case Ctrl-C will be ignored - leaving Ctrl-Break as the only way to interrupt the application.
I discovered about /b flag from the 2015-12-05 19:13 entry on this post: https://bugs.python.org/issue11361
Are you using start /b for cmd.exe?
-
ntpinit.exe doesn't use the start command. That way, a new window is not created. Also, if a new window is created, then ntpinit.exe can't get redirected output.
-
OK, that is good to know.
Based on all I've written and seen then, the most promising approach could be to put an intermediate process between ntpinit.exe and cmd.exe, where this intermediate process is the process group leader, it enables control-c for all children - SetConsoleCtrlHandler(null, FALSE), and then launches cmd.exe (not as new process group).
Only caveat is whether cmd.exe will create children that ignore Ctrl-C, then these children would be required to perform SetConsoleCtrlHandler(null, FALSE). If that is the case then would need another (or move) intermediate process between cmd.exe and the child that will call SetConsoleCtrlHandler(null, FALSE) and then the desired child.
-
Or if ntpinit.exe could be spawned as a new process group, and it does SetConsoleCtrlHandler(null, FALSE), and does not make cmd.exe a process group leader. Then need a way for ntpinit.exe to ignore Ctrl-C (possibly by installing some other handler via SetConsoleCtrlHandler() and returning TRUE from it), see: https://docs.microsoft.com/en-us/windows/console/registering-a-control-handler-function
-
I tried the above. That didn't work well. Added an actual signal handler for ntpinit.exe did have an effect but didn't work for Ctrl+C. It's pretty clear to me that Ctrl+C can't work unless there's a terminal window (at a minimum).
I'm tying to get a hybrid solution to work.
Send Ctrl+Break signal
Wait .1 seconds.
Terminate all child process
right now, the terminate all child processes isn't working quite right. Works on a simple program but not on a complex one. Hopefully, I'm just not traversing the tree correctly.
-
OK, well it seems that using ntpinit as the process leader (or a new intermediate process) you may need to do many special things as in this post: https://stackoverflow.com/questions/813086/can-i-send-a-ctrl-c-sigint-to-an-application-on-windows/2445728#2445728
Create a new helper application "Helper.exe". This application will sit between your application (parent) and the child process you want to be able to close. It will also create the actual child process. You must have this "middle man" process or GenerateConsoleCtrlEvent() will fail.
Use some kind of IPC mechanism to communicate from the parent to the helper process that the helper should close the child process. When the helper get this event it calls "GenerateConsoleCtrlEvent(CTRL_BREAK, 0)" which closes down itself and the child process. I used an event object for this myself which the parent completes when it wants to cancel the child process.
To create your Helper.exe create it with CREATE_NO_WINDOW and CREATE_NEW_PROCESS_GROUP. And when creating the child process create it with no flags (0) meaning it will derive the console from its parent. Failing to do this will cause it to ignore the event.
It is very important that each step is done like this. I've been trying all different kinds of combinations but this combination is the only one that works. You can't send a CTRL_C event. It will return success but will be ignored by the process. CTRL_BREAK is the only one that works. Doesn't really matter since they will both call ExitProcess() in the end.
You also can't call GenerateConsoleCtrlEvent() with a process groupd id of the child process id directly allowing the helper process to continue living. This will fail as well.
Now it does say that they failed to get Ctrl-C to work, but I suspect is that he didn't call SetConsoleCtrlHandler(null, FALSE), as that is not mentioned.
-
I tried every combination of SetConsoleCtrlHandler. It didn't help enough. Only adding an actual event handler allowed the BREAK signal to work.
I've got my hybrid solution working. Had to gather the child_pid list before sending the break signal. Works for java apps.
-
Did you try making a new process inbetween ntpinit.exe and cmd.exe? Or did you just try making ntpinit.exe as process leader and cmd.exe as not process leader?
-
I just made ntpinit.exe the leader and CMD.EXE not the leader. Anything that could be done in an additional process can be done in ntpinit. Also, ntpinit can not be allowed to terminated on Ctrl+BREAK or Ctrl+C since even CMD.EXE would end up terminating. That would be true for an additional process under ntpinit.
-
OK, with ntpinit as the leader, when you call GenerateConsoleCtrlEvent() you can only provide either the pid of ntpinit (as it is process leader) or 0. Using the pid of cmd would not work. What pid were you sending in GenerateConsoleCtrlEvent() ?
-
The pid for ntpinit
-
The hybrid solution isn't perfect for Java. The only other solution is just to always call TerminateProcess on all the child processes. Then there is no output.
You still get the following output:
2019-11-29 21:53:19
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode):
"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x0000000019e2c800 nid=0xb40 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread3" #9 daemon prio=9 os_prio=2 tid=0x0000000019d9f800 nid=0x6bc waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x0000000019d93800 nid=0xa30 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x0000000019d8d800 nid=0x2778 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x0000000019d87000 nid=0x246c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000019d85800 nid=0x1f4c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000019d84000 nid=0xf1c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000000002e8d800 nid=0x29dc in Object.wait() [0x000000001a24f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d5a08ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x00000000d5a08ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000000002e8c000 nid=0x2364 in Object.wait() [0x0000000019d4e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d5a06c00> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Unknown Source)
at java.lang.ref.Reference.tryHandlePending(Unknown Source)
- locked <0x00000000d5a06c00> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
"main" #1 prio=5 os_prio=0 tid=0x0000000002d96000 nid=0x20e8 waiting on condition [0x0000000002d8f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at javamain1.main(javamain1.java:7)
"VM Thread" os_prio=2 tid=0x0000000017e78800 nid=0x2634 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000002dac000 nid=0x28b8 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000002dad800 nid=0x1070 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002daf000 nid=0x1d9c runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002db1800 nid=0x2054 runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000000002db2800 nid=0x270c runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x0000000002db3800 nid=0x1778 runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x0000000002db7000 nid=0x1440 runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x0000000002db8000 nid=0xa68 runnable
"VM Periodic Task Thread" os_prio=2 tid=0x0000000019e48000 nid=0x2c4 waiting on condition
JNI global references: 4
Heap
PSYoungGen total 38400K, used 1996K [0x00000000d5a00000, 0x00000000d8480000, 0x0000000100000000)
eden space 33280K, 6% used [0x00000000d5a00000,0x00000000d5bf33b8,0x00000000d7a80000)
from space 5120K, 0% used [0x00000000d7f80000,0x00000000d7f80000,0x00000000d8480000)
to space 5120K, 0% used [0x00000000d7a80000,0x00000000d7a80000,0x00000000d7f80000)
ParOldGen total 87552K, used 0K [0x0000000080e00000, 0x0000000086380000, 0x00000000d5a00000)
object space 87552K, 0% used [0x0000000080e00000,0x0000000080e00000,0x0000000086380000)
Metaspace used 2464K, capacity 4486K, committed 4864K, reserved 1056768K
class space used 273K, capacity 386K, committed 512K, reserved 1048576K
-
I thought of some other things to try, just for curiosity (and maybe final product).
What if ntpinit/cmd is started in its own window? Would Ctrl-C work then? Visual C++ debugger does run programs in a dedicated console window, maybe this is why? While I would much rather run in the process buffer and not in a separate window, I wonder if this is the true reason why Visual C++ uses dedicated window? If this does work, I wonder if you can also view the input/output of the cmd in the process buffer as well - then maybe that dedicated window could be "hidden"?
Also, just to make sure - when you made ntpinit as process leader you also took away process leader for cmd? Did you also try 0 as pid in the GenerateConsoleCtrlEvent()? Did you remember to call SetConsoleCtrlHandler(null, FALSE) in ntpinit?
The hybrid solution that you did is certainly better than what we have now and I would gladly enjoy it in the next point release of SE. But it is not ideal for 2 reasons:
1) It does not do a "graceful shutdown". My java programs handle Ctrl-C and do cleanup, this cleanup will not be there. I would also not be able to debug the cleanup code in the SE debugger.
2) The output of the thread dump is not ideal, but I still think it is better than just terminating with no output.
-
FYI: I was able to send a SIGINT to my java program running in the process buffer and have it gracefully exit by doing the following:
1) Open process buffer
2) Start cygwin: c:\cygwin\cygwin
3) Start bash under a pty (required for winpty): script -eqfc "bash -i" /dev/null
4) Run my java code under winpty: "winpty java ...."
Then when I sent a Kernel32.CTRL_C_EVENT from an external program that does AttachConsole(pid) and then GenerateConsoleCtrlEvent(Kernel32.CTRL_C_EVENT, 0);, it did gracefully exit!
winpty: https://github.com/rprichard/winpty
I was using winpty previously when launching my java code from a cygwin prompt outside of SlickEdit in order for Ctrl-C to work.
Notice in the README of winpty:
The software works by starting the winpty-agent.exe process with a new, hidden console window, which bridges between the console API and terminal input/output escape codes. It polls the hidden console's screen buffer for changes and generates a corresponding stream of output.
Perhapse SlickEdit could do something similar as winpty, but instead of bridging cygwin's pty, it can bridge SlickEdit?
-
I was wondering if I could create a hidden console window. I’ll have to look into that.
-
What are you running in SlickEdit's process buffer?
Are you running winpty in the process buffer?
I guess I'm pretty confused now.
-
Yes I am running winpty in the process buffer.
I do steps 2-4 in the process buffer. Step 4 is where I run winpty, and yes I run it inside the process buffer, but that is inside of a cygwin bash shell, and the cygwin bash shell is running in the process buffer.
And I can send Ctrl-C (from external program) via GenerateConsoleCtrlEvent(Kernel32.CTRL_C_EVENT, 0) and my java code does gracefully shut down - it sees the Ctrl-C (not Ctrl-Break) and does all my cleanup, and log prints I have in the cleanup code, before exiting.
winpty is not available from the standard cygwin setup, I had to download the .exe and .dll from github and place it into c:\cygwin\bin
-
Are you sending the Ctrl-C event to winpty or java?
What does the java program run with winpty do?
Are you running another external java program to send the Ctrl+C event besides the one thats the argument to winpty?
-
Are you running another external java program to send the Ctrl+C event besides the one thats the argument to winpty?
Yes I am.
I am working on cleaning it up so you can run it yourself.
What does the java program run with winpty do?
It basically runs "sleep 10000", and if Ctrl-C is received, it will do "cleanup" and print a few things.
I'm also working on providing this to you.
Are you sending the Ctrl-C event to winpty or java?
I send it to the console that is attached to the java program's pid.
It does the below:
// Get the console of the pid running java
Kernel32.INSTANCE.AttachConsole(pid);
// Send CTRL_C event to pid 0, which corresponds to the console that this
// program is running in. Since I used AttachConsole previously, this is the same
// console that "java" is running in.
Kernel32.INSTANCE.GenerateConsoleCtrlEvent(Kernel32.CTRL_C_EVENT, 0);
I based it on: https://stackoverflow.com/questions/813086/can-i-send-a-ctrl-c-sigint-to-an-application-on-windows/42839731#42839731
-
I just made some progress with something. Didn't need all the complexity you've mentioned recently. Went back to some docs you posted earlier and tried something. Got Ctrl+C event to work in the process buffer but not exactly with the process tree I want. Now I need to figure a better process tree.
-
I've narrowed down the problems for sure. It's looking like there were two issues
Issue #1 (simple to solve)
ntpinit needed to turn off the ENABLE_PROCESS_INPUT console mode. Actually it turns it off, starts cmd.exe, and then turns it back on.
Issue#2 (the harder problem)
There seems to be a problem with SetConsoleCtrlHandler. It's looking like CMD.EXE is messing up what the child process inherits. ntpinit.exe does SetConsoleCtrlHandler(NULL,0). This doesn't help.
I modified my "cmtime" app which runs other applications to do a SetConsoleCtrlHandler(NULL,0) BEFORE running the child process. When I do this, I can send a Ctrl+C event in the process. (i.e start process buffer (new version), run "cmtime waitforkeypress", send Ctrl+C event, Zapola!!)
This unfortunately isn't a great solution because it requires another app.
-
That is indeed some progress. Yes having another app that runs between cmd.exe and the desired child process is not ideal, but it is good to know that can work.
I suppose if ntpinit was made like winpty then could get around this, but that introduces much more complexity.
-
FYI: I redid my previous winpty experiment, however instead of running my java program under winpty, I ran "winpty cmd" and got an cmd prompt (all in the process buffer). Then at the new cmd prompt I ran my java program that does a sleep. Then outside of SE with my "win-kill" java program, I send the Ctrl-C to the console of my Java program and it did indeed interrupt the java code. So it seems that running cmd under winpty any programs run under cmd are interruptible.
-
Are you sure the winkill program is sending a Ctrl+C event?
-
Yes, I'm pretty sure it is sending a Ctrl-C event.
You can try it for yourself, I have uploaded it to support. Look for ctrlc.7z. You can try running winpty inside the process buffer as I have.
See the included README file in ctrlc.7z for how to run winkill and the other java program that sleeps as well as where to look for the source code.
-
Never ran your test. Although I was starting to set it up.
I'm seeing some success doing some weird stuff but it looks promising. I have to investigate further as to why it works. Keep your fingers crossed.
-
OK. I found something in my winkill that is specific to my environment and I would need to change it for you to run it, otherwise it won't work for you and you'll get an error. If you are interested in doing that later then let me know and I can modify it and you can run. At least you have my source now you can refer to it.
-
I think I've got this working perfectly. Didn't even need an extra process. This console code is REALLY picky and what you can and can't do is hard to figure out.
I made ntpinit.exe the new console group. Then I made sure I didn't create a new console group when starting CMD.EXE. These changes combined with tweaking the console mode and break handling were the trick. I'm still testing but it's looking really good.
Done. No new problems as far as I can tell.
-
That is very very great news Clark! Thanks so much for your persistence and interest in this!
-
Will these changes go into the currently planned SE point release (24.0.1, or whatever it will be called)? Sounds like something useful generally, not just for running Java programs...
-
This change will be in 24.0.1 which should be available before the end of the year.