How to Split Test Across Email and SMS Simultaneously

Why Cross-Channel Testing Matters

Most marketing teams test email and SMS separately, optimizing each channel in isolation. This misses the bigger question: for any given campaign, which channel produces better results, and does using both channels together outperform using either one alone? These are questions that only cross-channel testing can answer.

A promotional campaign might get a 25% open rate and 3% click rate via email but a 95% read rate and 12% click rate via SMS. The SMS numbers look dramatically better, but SMS also has higher opt-out rates and tighter content constraints. Cross-channel testing helps you understand the full picture, including how each channel affects subscriber retention and long-term engagement, not just immediate click rates.

Three Ways to Structure Cross-Channel Tests

Channel Comparison Test

The simplest cross-channel test sends the same message to three randomly selected groups: group A gets the email only, group B gets the SMS only, and group C gets both. Measure conversion rate and revenue per contact for each group. This tells you whether one channel dominates the other and whether the combination provides incremental lift.

Important: make sure contacts in each group are excluded from the other groups for the duration of the test. If someone in the "email only" group also receives the SMS, your test is contaminated and the results are uninterpretable.

Sequence Test

Test different channel sequences for the same campaign. Group A gets email first, then SMS follow-up 24 hours later. Group B gets SMS first, then email follow-up 24 hours later. Group C gets only email. Group D gets only SMS. This test reveals whether the order of channel touches matters and whether the follow-up provides enough incremental value to justify the additional send.

Channel-Specific Content Test

Some messages work better in email format (detailed content, multiple links, visual layouts) while others work better in SMS format (urgent, time-sensitive, single action). Test whether adapting your message format to each channel's strengths outperforms sending the same core message across both channels.

Managing the Attribution Challenge

The hardest part of cross-channel testing is attribution. When a contact receives both an email and an SMS and then makes a purchase, which channel gets the credit? Most marketing platforms attribute the conversion to the last channel the person interacted with before converting, but this can be misleading because the email might have created awareness that the SMS then converted.

For testing purposes, the cleanest approach is to measure total conversion rate per group rather than trying to attribute each conversion to a specific channel. If the "both channels" group converts at 8% and the "email only" group converts at 5%, the combination is producing incremental lift regardless of which specific touch drove each individual conversion.

Timing Considerations

Email and SMS have very different engagement windows. Email engagement peaks within 2 to 4 hours of delivery but continues for 24 to 48 hours. SMS engagement peaks within 30 minutes and is essentially done within 4 hours. If you send both channels simultaneously, the SMS will likely capture the conversion before the email has had time to work. Consider staggering your sends by 4 to 8 hours to give each channel time to perform independently.

Also be mindful of channel fatigue. Sending an email and an SMS about the same promotion within the same hour can feel aggressive and increase opt-out rates for both channels. Space your touches appropriately, and for the "both channels" test group, monitor opt-out rates alongside conversion rates to make sure the incremental revenue is not coming at the cost of long-term subscriber loss.

What to Measure

• Conversion rate per group: the primary metric for comparing channel effectiveness
• Revenue per contact: accounts for differences in order value between channels
• Opt-out rate per group: ensures that aggressive cross-channel campaigns are not destroying your list
• Incremental lift: the difference between the "both channels" group and the better-performing single channel
• Cost per conversion: SMS costs more per message than email, so factor in send costs