FUSD is a new implementation, but certainly not a new idea--the theory of its operation is the same as any microkernel operating system. A microkernel (roughly speaking) is one that implements only very basic resource protection and message passing in the kernel. Implementation of device drivers, file systems, network stacks, and so forth are relegated to userspace. Patrick Bridges maintains a list of such microkernel operating systems.
Also related is the idea of a user-space filesystem, which has been implemented in a number of contexts. Some examples include Klaus Schauser's UFO Project for Solaris, and Jeremy Fitzhardinge's (no longer maintained) UserFS for Linux 1.x. The UFO paper is also notable because it has a good survey of similar projects that integrate user-space code with system calls.