I am doing machine applique with a satin stitch and using a stablizer on the back of the project. However, my square is still getting distorted and pulled as I stitch.
I am using rayon thread with a top stitch needle, size 90/14.
There are several factors that could be the cause your problem.
Before anything else, the first thing I recommend is to switch to a new needle if you haven't already. Unless the needle is broken or bent, we usually can't see the damage to the point with the naked eye.
If that doesn't make a difference, then try the following suggestions. Try only one thing at a time and then test. If you make several changes to your sewing machine setup and then test, you'll not know which change in setting was responsible for the improvement (or degradation) in your stitches.
Needle Size and Type I know many thread manufacturers recommend a 90/14 topstitch needle for use with decorative threads.
Personally, I like to match the needle size to the thread size. The finer the thread weight—the smaller the needle. If you're using a fine thread, like a 50 weight cotton, switch to a smaller 70/10 needle. For a 40 wt rayon thread, like Sulky's small spool, I'd start with an 80/12.
If you're using a Universal needle, switch to one with a sharp point. My first choice would be a Microtex Sharp with an acute sharp point for piercing through the layers, but an Embroidery, Quilting or Topstitching needle are all good choices. If you're using a metallic thread, then be sure to use a metallic needle.
The larger needle makes a larger hole which in turn creates less friction as the thread is drawn back and forth through the fabric until it forms a stitch. (Did you know that the needle thread goes back and forth about 30 times before it actually makes a stitch...that's a lot of wear and tear on the thread.) However, a bigger needle also makes a bigger hole and unless the thread is a heavier weight it won't fill the hole completely. I don't like the way that looks.
If your thread is shredding or breaking, the needle size is too small. The right size needle will neither shred or break your thread nor leave behind large holes in the fabric.
Just make sure it it heavy enough to stand up to dense satin stitching. If the product you are using right now is light weight, try two thicknesses and see if that makes a difference.
An alternative to a separate layer of stabilizer is to heavily starch your background fabric. But this extra step takes time and the starch must be washed out after the project is complete. With a stabilizer, you avoid this extra work.
Reduce Stitch Density
If the fabric is distorting as you stitch, try reducing the density of the satin stitch a bit. There are two ways to do this depending on your sewing machine.
Some machines will let you fine tune a stitch length in 'tenths' of a stitch. Bernina does this on at least some of their machines. Other may too. If your machine lets you do this, increase the stitch length by .1 and test. Repeating until there is no more distortion.
My older viking D1 only let's me increase in .5 increments. If this didn't solve my satin stitch problem, then to reduce stitch density on this machine, I would to switch to a finer thread, say from a 30 wt rayon thread to a 40 weight. That way even with the same number of stitches in an inch, there would be less thread laid down and less distortion of the fabric
Adjust Sewing Machine Tension
Many sewing machines will automatically set the tension when you input the weave and weight of the fabric and stitch. This is a good place to start, but I find that I 'tweak' my tension settings all the time.
Good tension on satin stitching will have no bobbin thread showing on the top. On the back side of your stitching the needle thread will show either on one or both sides of the bobbin thread.
If your needle tension is set too tight, it will pull your bobbin thread to the top. Reduce the tension by reducing the number on the tension dial or by turning it to the left (remember 'lefty-loosey'). Reduce and test. Repeating as needed until you are happy with the resulting stitches.
If it is too loose, there will be little no no bobbin thread showing on the back. Increase your tension by increasing the number of your tension dial or by turning it to the right (remember 'righty-tighy'). Increase and test, repeating until you are satisfied with your stitch quality.