Well most of that behavior is completely normal. The alternator output can (will) drop if the RPM's get too low. And if the alternator output drops far enough, the warning lights are supposed to come on. There are two things that concern me a little though, and that's why I was suggesting keeping an eye on things.
1) The alternator output might be dropping off at an RPM that's higher than normal which may be a sign that the alternator may be having some sort of internal problem. I don't know what the specs are, but on my car with an internally regulated alternator, my alternator output seems to be able to keep up easily at 700 RPM idle, even with the headlights and blower motor on. In other words, even with the headlights and blower motor on, I can let my RPMS drop to idle (700 RPM) and my warning lights don't come on. Might be a warning sign that yours do
2) The other thing that concerns me is the "latching" of either the VOLTS or BRAKE lights. Even if the alternator output does drop off at low RPM's, both those lights should go back out when you rev the engine. Every time. So if one or the other of those lights ever "latch" on, it might be another sign of impending doom for your current alternator.
Makes me wonder if your brushes are worn to a nub and are making intermittent contact. Or your internal regulator may be failing.
In my experience, things like this are warning signs that something is brewing. Might not be critical. Yet. But something's going on...