I am going to preface this by saying that the cause of the error itself might not actually be related to PowerShell specifically, it may lie deeper in .NET but that is where I encountered it.
PowerShell version: 7.1.0
I defined the following type in order to resolve resource requests as used in the registry for example (the signature is exactly as seen on PInvoke.net):
Add-Type -TypeDefinition @"
using System;
using System.Text;
using System.Runtime.InteropServices;
public static class ShellHelper {
[DllImport("shlwapi.dll", BestFitMapping = false, CharSet = CharSet.Unicode, ExactSpelling = true, SetLastError = false, ThrowOnUnmappableChar = true)]
public static extern int SHLoadIndirectString(string pszSource, StringBuilder pszOutBuf, int cchOutBuf, IntPtr ppvReserved);
}
"@
Now for a test case, consider the following:
$result = [System.Text.StringBuilder]::New();
$tmp = [System.IntPtr]::New(0);
[ShellHelper]::SHLoadIndirectString("@firewallapi.dll,-50323", $result, 256, $tmp) > $null;
$result.ToString()
The output SNMP Trap is produced and everything is looking good. Some other values work just fine too. However, if I replace the number -50323 with -50326 for instance then I still get the correct value of Block any other traffic to and from SNMPTRAP service as output, but then upon almost any action I take (i.e. executing a command, or even just pressing Tab) an unending series of Fatal error. Internal CLR error. (0x80131506) exceptions is thrown until I get a Stack overflow. message and PowerShell crashes to desktop. (See https://imgur.com/a/MYZwhYa.)
The error is dependent on the string being processed and not the number of method invocations so far: if the last command is called first, in a new session, the same error happens right away.
I have been unable to determine or even make a reasonable guess about exactly what causes this behaviour though, aside from possibly certain properties of the output string (its length, the presence of certain characters etc.). Playing around with certain parameters (changing the 3rd argument, using out int and [ref] for the 4th one etc.) does not seem to affect the outcome.
I am mainly mistified by the bizarre timing of the error. It would seem like some internal process maybe causes other data in the memory to be corrupted, leading to errors later, but this is only my wild guess. I might also simply be missing something supremely obvious.
What might the cause of this behaviour be? If it is a genuine bug in .NET how could I maybe work around it, or even just go about nailing down the exact criteria for it happening?
Thanks! It somehow did not occur to me that the StringBuilder would not be able to handle being fed a larger string since I just took it for granted that it will internally increase its capacity as needed (I suppose that only works for the high level API of the class then, and we're writing directly into its contents here I guess). But now I am certainly facepalming hard. :)
This might be interesting. The
StringBuilderinstance is not used as such (e.g. by callingAppend(), which would increase the capacity) by the native code (or the marshaller), but just as a "buffer" to receive data. The size of this buffer must thus be known.