I got a couple warnings in the process of experimenting with compression settings. One was VBR <######> Level Limit ######, the first number being about 5,000 above the second (eg: 25000 vs 20000). At one point I knew I was trying a bitrate too high because I got a warning saying try reducing qp_min, current is 10. I didn't bother changing these parameters because qp_min not being seen anywhere but in the command line appeared to require editing it, and of course the bit rate leveled out as I got better at setting it.
The warnings can be ignore if you're just going to put the videos online. You'll notice when you encode with x264 that it will have a target level, 4.0, 5.0, etc. There are exact specifications that you need to comply with for these levels if you want to be compatible with the specification. For example, if you want to encode a video to be played back on the PS3, it must conform to level 4.0. Exceeding the bitrate for that level will spit out a warning letting you know. If you are just going to play these files back on a PC and nowhere else, then you can ignore the warning.
On AAC, last time I tried it, via sampling iTunes a while ago (which I despise), it didn't sound as good to me as wma, and not even any better than LAME. Since I've been using Xvid though, I've been using mostly LAME. If AAC can be used with x264/MKV though, I'll give it another go. Upon installing AutoMKV, I noticed there was an option regarding Nero, and I thought it said something about AAC too. I opted out of the Nero thing because I didn't think it would apply to my uses, even though I burn with Nero.
LAME is the highest quality MP3 encoder out, and yet AAC can still have better quality bitrate for bitrate. Just like with LAME, though, depending on the encoder there are plenty of options to tweak the sound. Comparing with iTunes may not be the best. neroaacenc is pretty high ranked in terms of output quality. I think for the best comparison you should compare LAME to neroaacenc.
As for the audio format with MKV, you can pretty much put anything you want in that container. MP3, FLAC, AAC, WAV, etc., etc. It's really a great container. MP4 has more popularity and support over MKV, but MKV is superior in features, audio/sync, etc. They're easily interchangable, though. You can easily extract tracks out of both MP4 and MKV containers so you can switch between the two very easily.