The software renderer doesn't actually clip the edges. It doesn't generate new edges, and it doesn't generate new vertices, so there's no way to use it to reshape the brushes.
The clipping is done at the spans level. It's kinda like a run-lenght encoded Z-buffer check: each span is segmented by the nearer span segments of other planes.
To port this to the GL renderer, a large portion of the software renderer would have to be used, all the way to the spans generation. And then, a fragment shader could be used to make the GPU only draw the pixels that matches the corresponding spans from the software renderer.
Ph'nglui mglw'nafh mankrip Hell's end wgah'nagl fhtagn.