Custom Point Shader in Autodesk Forge Viewer behaves weirdly with orthograpic Camera

I am using a PointCloud for fast rendering of sprites after this blog post. Everything works fine with the perspective camera. However, if I switch to the orthographic camera via viewer.navigation.toOrthographic() the points' sizes are not calculated correctly. Does anyone know what the issue is or where I might find some clue?

My vertex shader

#if NUM_CUTPLANES > 0
varying highp vec3 vWorldPosition;
#endif
attribute float mVisible;
attribute float mSize;
varying float vMVisible;
uniform float scale;
void main() {
    vMVisible = mVisible;
    
    #if NUM_CUTPLANES > 0
    vec4 _worldPosition = modelMatrix * vec4( position, 1.0 );
    vWorldPosition = _worldPosition.xyz;
    #endif

    vec4 mvPosition = modelViewMatrix * vec4(position, 1.0);
    gl_Position = projectionMatrix * mvPosition;
    gl_PointSize = mSize * (scale / (-mvPosition.z) );
}

Zoomed out

Zoomed in

Zoomed in a little bit more