just blindly tracing to all of the verts 'feels wrong' its the sort of thing that is intuitively unreasonable with large data sets and is quite obviously going to be constrained by available horsepower at some size.
precomputed vis data is the obvious optimisation and removes the O(n) behaviour. not only do you not have to test everypoint but your raycast itself need not test every edge. i wrote something ages ago about generating PVS in 2d as an example of how to approach the problem in 3D...
(http://jheriko-rtw.blogspot.co.uk/2011/05/visibility-determi...)
the other thing that bugs me here is the jitter. it is the classic 'lazy' way to soften shadows... a better effect can be gotten for less - if you offset the light position perpendicular to the direction toward the test vert, in both directions by the same amount then you can generate two ray casts. triangulating the resulting mesh might be a little painful (unless you cut the world into convex chunks - then it is nearly trivial) but you will get the outline for both edges of the shadow and can then do something like vertex interpolated colour to give you the same visual result as infinite raycasts for the price of two...