You can do this with 4-way stretch if you do a few things to it first. Given the nature of the outfit, I am assuming this is a stretch fabric that has very good recovery, like a mid-weight spandex knit. If the fabric doesn’t have good recovery (the ability to snap back into shape easily after it has been stretched), then sagging will be more of an issue.
The first thing I would do is add fusible interfacing to the back of each fabric piece. This will help hold the fabric in place without stretching so that when you sew it, you won’t have to worry as much about sagging. Don’t get an interfacing that is too stiff, but just enough to give a bit of body to the fabric and to prevent stretch. Test a few out to see how the adhesion is to a stretch fabric (you may end up using a stretch interfacing with the less stretchy direction going vertically to help with adhesion and to prevent it from stretching under its own weight).
The next thing I would do is flat line all of the pieces in a non-stretch fabric. This is a process where you sew an exact duplicate of your pieces to the back, keeping your stitching inside the seam allowance and being sure to sew all the way around. It is commonly used to add a bit of body or structure to lightweight fabrics, and here, it will prevent your fabric from stretching.
Once you sew the jacket, I would recommend lining it in a non-stretch fabric as well. This will make it so that when you put the jacket on and take it off, the stress of that will largely be going towards that non-stretch fabric, and the outer fabric will stay intact.
Be careful when washing as to not stretch the garment too much, and lay it flat to dry. If you feel the need to reinforce any seams, you can add twill tape or ribbon to them while sewing, which will prevent the seams from stretching out.
I hope that helps! Good luck :]