Several people frequently reported GPFs when debugging Smalltalk code.
After detecting certain patterns that lead to GPFs the VSE community figured out a set of bugs that may cause a GPF when working in the debugger.
The bugs are related to the complex mechanics when code is being debugged.
- when a process is being debugged, each time the you press "hop", "skip" the
- Smalltalk stack must be swapped between the Debugger process and the Debuggee process
- to allow hops and skips, the VM must provide a way to execute a single byte code
- and transfer control back to the Debugger. This is done, by replacing a standard compiled method with a "debuggable compiled method", which triggers debugger interrupts after each byte code. A consequence of this is a different machine code (JIT code). Besides the JIT cache thunk areas are involved. The dynamic insertion of debuggable compiled methods requires tweaking of return addresses on the machine stack. There is bug in tweaking the return address pointing into the thunk areas. Another bug is related tweaking return addresses for methods containing blocks from debuggable compiled methods.
- during stack swapping the VM must check, if the JIT cache still contains valid
- JIT code for all open message sends and must eventually rejit methods, that have been purged from the JIT cache.
Some of related bugs, can be fixed in Smalltalk. See: Debugger crashes (GPF)
The thunk area return address bug must be fixed inside the VM (no patch available so far).