Changing an ingredient in your favorite pasta sauce is pretty straightforward. It might not taste as good as you'd hoped, but nothing very bad will happen. It's a little more problematic in baking, where ingredients are important for their chemistry as well as their flavor. For example, baking soda won't help your cakes rise unless you also include an acidic ingredient. Vinegar and lemon juice are obvious examples, but there are lots of others.
The Chemical Reaction
Baking soda, or sodium bicarbonate, is an alkaline or "base" ingredient. When it's combined with an acidic ingredient, it reacts to form bubbles of carbon dioxide, like the science-class "volcano" you made as a child. Those bubbles provide the lift your cakes and muffins need to have a light and fluffy texture. To get a strong reaction, and good leavening, you need to add acidic ingredients to react with the soda.
Cream of Tartar
One of the most versatile acids you can add to your baking is cream of tartar. Known to chemists as tartaric acid, it's a byproduct of wine making. Crystals of tartaric acid form in the grape juice as it ferments and are removed when the wine is filtered. Those crystals are purified and ground to a fine white powder for baking purposes. You can add cream of tartar along with your other dry ingredients, using one-third as much as the amount of lemon juice or vinegar the recipe calls for. Commercial baking powder is made by combining baking soda with cream of tartar.
Cultured Milk Products
Regular milk contains lactic acid, so it combines with baking soda to provide some leavening power. If you find your baked goods don't rise enough, or if there's a noticeable taste of soda, try substituting cultured milk products such as buttermilk, plain yogurt or kefir. They're more acidic, so they'll provide a more powerful reaction with the soda and generate better leavening. If you have milk in your fridge that's gone past its prime, it also has more acidity than fresh milk and works well in your baking.
Your choice of sweeteners provides another opportunity to add some acidity to baked goods. Plain old granulated sugar is pretty neutral, and it doesn't bring much to the table. However brown sugar is more acidic, due to its molasses content, and helps to raise your baking. So can molasses itself, though its strongly distinctive flavor limits its versatility. Another option is honey, which has been widely used in baking for centuries. Like molasses, it adds a distinctive flavor, but that flavor is less overpowering.
Many other ingredients can add a welcome degree of acidity to your batters and doughs if their flavors and textures are appropriate. For example, coffee is acidic and a few tablespoons of strongly concentrated coffee can add an earthy note to many recipes. Natural cocoa is acidic, though Dutch-processed cocoa is not. Fruit juices also are good in many cakes, though it's best to use frozen concentrates to avoid making your batter too wet. Fruit purees, such as applesauce, can add both acidity and moisture. They're an excellent option if you also want to reduce the amount of fat in your baking.