As much as some think it's not necessary - getting to the place early and doing the sound check is a necessity.
If possible, make sure you can cover each type of live sound configuration during the sound check - piano, plugged in acoustic, drums, miked acoustic, miked banjo, plugging in banjo, vocals, backing vocals, electric guitar. If you have enough channels on the board try to dedicate one per type of instrument/mic. Preventing/limiting reconfigurations during a performance helps a lot in keeping the surprises down to a minimum. Open mics can be simple. It's when a full band is involved when things get interesting. Vocals, electric guitars, bass, drums, keys, etc. Most times you can get away without having to mic the drums.
Dedicating channels to specific instruments should help. The numbers of channels you mixer has will determine how problematic things can get. An 8- or 12-channel mixer will support only 6- or 8-mono inputs. Always keep in mind upper channels are stereo. So if you have a bass plugged into the mono input on channel 5 your losing channel 6. Same with channel 7/8. For some this can be misleading because the thought is "16-channels = 16 things I can plug in." In reality your only going to get to plug into channels 1 to 8 and 9, 11, 13 and 15 monuarally. If you have keyboards or a stereo guitar setup then have a blast and use the stereo channels for those instruments. But when you think about it - its still only a maximum of 12 instruments for a 16 channel mixer. So far I've only encountered a max of 4 stereo inputs on live mixers. A 24-channel mixer has a total of 20 mono channels and 32-ch = 28 monos.
I sympathize. I've done live sound on and off for about 40 years and it can be a nightmare.
Without specific details, it's hard to give specific advice. John and Rick have made some good points. From your post, its sounds like a) you're trying for too much volume in the mains and/or the monitors, b) the room is acoustically poor, c) the speaker placement is poor, you're dealing with musicians who don't have good mic and playing technique (always a problem in open mics) and/or e) you're mixing a lot of electric instruments and bands, not just one or two acoustic players. Let me add a couple things...
1. In mixing, volume is far less important than clarity. This is true of acoustic performers and electric bands as well. Volume levels should be set - if possible - so that the front rows aren't being blasted and the rear seats don't have to strain to hear. If the room is poor acoustically and/or if the speakers are placed badly, this can be a real problem. But, as a general rule, I try to keep the master volume as low as possible and set the individual channels to where the peak lights only flash occasionally. Let the performers 'work' the mics if they will. The effect of this is maximum volume with minimal electronic noise (essential in recording) and maximization of headroom in the amp. Remember that every 3dB increase in volume (the minimum change detectable by the human ear) requires doubling the power output of the amp. Leave yourself room to maneuver
2. Start, as Rick said, with totally flat EQ on all channels and the master. Give each channel only what it needs to enrich the sound produced by the mics/instruments/vocals. Set the master only during the sound check (?). For most rooms, you will want to roll off a bit of the high and low ends. Always remember that, with equalization, it's better to subtract than add.
3. As to feedback, apart from lowering the volume and repositioning the speakers, try this trick. Get some phase reversing adapters (Shure, Sescom) and use them in line with the microphones. Reverse the phase and feedback cancels itself out.
4. Mixing electric bands is problematic. In smaller rooms, the drums dominate and everybody else plays louder to try and balance. Guitarists and bassists often have vastly overpowered amps, and if they're using effects, this muddies things even more. The trick is to mute the drums if possible. An isolation booth is best. You can at least put a pillow in the kick. Once you do that, it's easier to get the guitars, bass and piano to turn it down, allowing the vocals to come through.
If I think of anything else, I'll post again.
Benji - how's it going? Has any of this been helpful to you?
It occurs to me that one thing no one has mentioned is the system setup itself. Are you using house systems or bringing your own or rented gear?
With a house system, it's pretty much WYSIWYG, especially if the speakers are installed in fixed locations. Poor placement can be a major factor in the feedback department and, apart from using phase reversers (sic) on the mics, there's not much you can do. You're also stuck with the house mixer, which may or may not be adequate to the task. If the house amplification is separate from the mix station, the amps may be set too high
If you ARE using house systems, you may want to invest in a good 1/3 octave (31 band), dual channel graphic EQ that you can patch into the system. The dual channels allow you to EQ each side of the Front of House speakers. A second unit could be patched into the monitors. A couple of Sabine FBX unit might also be helpful in reducing feedback.
Of course, if you're using your own gear, or renting equipment based on the requirements of the venue, you have a bit more flexibility and can, up to the limits of your budget, design your rig to fit your needs.
Note to John G -
Maybe we should establish a separate thread for live sound issues and comments. It would be good to hear about others' experiences with various venues and different rigs and pieces of gear.