Okay, wait...if I understand the instructions correctly...the D3DWindower program will allow you to run Starcraft (and Diablo II) at custom resolutions...in a window. But is there a way to use this program/technique to run Starcraft with custom resolution, in fullscreen?
Depending on what you mean by 'custom resolution'.
If you speak about the normal 4:3 scale resolutions, just run StarCraft as it is. Though it will just stretch the image because the engine doesn't support 3D and is fixed at 640 pixels per width. So running it at 1280x960 will make all the pixels twice that wide and long.
If you mean widescreen resolutions, no you can't, because of the fixed res again. The only way to get around is using the VGA's Control Panel options to fix scaling (so you'll get two black bars and no stretching) or the D3DWindower to prevent distortion.
Btw you can turn window mode off in D3DWindower but that won't make it prettier, see the first point.