Nylon stringed guitars are often referred to as classical, Spanish or flamenco guitars. To begin with, these kinds of instruments use nylon strings versus steel '" hence the tonal difference. While the exact details of what makes classical, Spanish and flamenco guitars different from each other can be a bit fuzzy, it's generally based upon their size, wood, string spacing and bracings.
For example, guitars used in flamenco tend to have a lighter top than classical guitars. The bodies are often made of rosewood or cypress, with a spruce top. Since flamenco guitarists use the top of their guitars to tap rhythms on (golpes), many instruments have pick guards there to both protect the instrument and project additional sound.
The general approach to recording nylon guitars is quite similar to steel stringed acoustics. The majority of sound comes from the projection of the soundhole, and the body itself. With that in mind, you can place a good, high quality mic about 8-12 inches from where the fretboard meets the sound hole.
When it comes to the specific types of mics (Audio-Technica AT 4041/AKG Perception 170/Rode NT5/Shure PG81, its best to use something like a small diaphragm cardioid condenser '" and stay away from the old Shure SM57 type dynamics. While they sound good on amps, they have a peaky midrange which does not compliment acoustic instruments. The objective is to capture the natural tone of the guitar. You can also use large diaphragm mics (Neumann U87/Rode K2/Audio Technica 4033/Se Electronics X1), but they (for the most part) don't capture as much detail on acoustics as small diaphragm mics do.
Also, take into consideration your environment. Acoustic guitars, both steel and nylon, benefit from a neutral but live space '" preferably with no distinct reflections. So try to stay out of the corner (unless it sounds amazing). One trick I use is to buy strips of inexpensive bamboo flooring, and lay them down on the carpet in my room. I position the guitarist either on top of the wood or just behind it (depending on how much you have). With that, I can get some wooden '˜bounce' of the guitars off the floor.
When recording guitarists like Al Di Meola for example, I would turn to a pair of Schoeps CMC5 (now CMC6) with a cardioid capsule. I place them about 8 inches or so from the guitar, cross them in an XY pattern, and pan them wide to open the stereo field. He plays a Conde Hermanos nylon guitar, which has a beautiful sound to it '" which of course is a great place to start (not to mention his skill).
You can of course use a single mic to record nylon guitars, and if one is all you have, it's probably best to start with the soundhole positioning. If you do have a second mic, try placing it on the body of the instrument. Use your ears as to what sounds best and place it about a foot back.
Recording flamenco guitar is a bit different. Guitarists like Romero use the meat of their fingers and nails to generate different tones from the instrument. Sometimes he also plays quite hard, attacking the guitar aggressively. This then generates a lot more volume, which of course affects the recoding '" something you have to be careful about when setting levels. I tend to record him at a lower level than say, Al Di Meola, because his dynamic style dictates that. So you also have to take the guitarist into consideration when dialing in your preamp. '¨'¨With Romero, I tend to use Earthworks omni mics, such as the QTC1 or QTC40. They have a wide frequency response and natural sound, which is great for capturing the overall character of the instrument. The omni (omnidirectional) polar pattern records equally in a 360-degree pattern, and also helps limit proximity effect (the buildup of bass information when placing a mic close to its source). I will place one about a foot from where the fretboard meets the soundhole, and another on the body of the instrument to capture additional low end.
One thing I've noticed in my experience is that nylon guitars often benefit from liberal doses of hi frequency EQ when mixing. So often, I will add a hi shelf around 8 kHz and boost the entire upper range of the instrument. This helps cut the guitar through a mix (especially dense ones). While this of course varies with every song, I've found it to work more times than not. But with nylon guitars, take the old rule of EQing into consideration '" if it sounds right, it is right.