Single string playing on the 5 string banjo is a style that was originally popularized by players such as Don Reno and Eddie Adcock. This style of banjo playing, while still utilizing 3 fingers (thumb and 2 fingers) with fingerpicks on them emulates what you play when playing guitar with a flatpick. Players such as Bela Fleck, Noam Pikelny, and Ryan Cavanaugh have taken the single string technique to new levels.
When playing guitar with a flatpick, the most common and straight forward picking technique is to use a down/up motion so that the downbeats are played with a down stroke and the upbeats are played with an upstroke. See the example below:
To begin single string playing on the banjo with fingerpicks you can emulate this down/up motion of a flatpick by using your thumb for the down stroke and your index for the up stroke. I would recommend practicing this on one string at first without fretting it. Let's practice on the 4th string open.
Now try to play a G scale using this same picking motion. See the tab below:
Now try playing some melodies that you know by ear using this picking method. The more you do it, the more it will feel comfortable. If you are a guitar player, this is going to feel very comfortable for you.
There are other techniques to single string playing, but for now I would focus on getting the thumb and index finger going very smoothly. You want to make the notes feel as smooth and even as possible. You don't want the note played with your thumb to have a different sound as those played with your index finger. Practice this with a metronome as well so that your timing stays precise.
Here's a great video from Bela Fleck on the basics of this technique.