In Praise of A/B Split Testing

Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test.

For an online ad to build my online subscriber list, we tested two headlines:

A–“Earn $2,000 per week ‘working’ an hour a day”
B–“Earn $100,000 a year selling simple e-books online”

One outpulled the other by 25%.

Which do you think was the winner — and why?


40 thoughts on “In Praise of A/B Split Testing

  • It’s usually stupid to guess which is going to do better… but I’ll do it anyway.

    I would guess Headline A outpulled B.

    $2,000 sounds more realistic even though it’s slightly more than $100K a year.

    Also, the trivializing of ‘work’ might push results toward A.

  • I would say that B worked better because:
    – it is specific about how the money can be made (selling e-books);
    – it sounds more believable than A, — the one week time frame in A is too short and a lead up period would be needed to make $2000/wk;
    – the words in B flow better.

  • I’d be shocked if B hadn’t performed better.

    There’s just something about “$100,000” and “simple” being used in the headline that sounds better than “working” and “$2,000”. Plus, I agree with the others about it sounding more believable.

    Having said that, I think B could perform EVEN BETTER if you replaced “Earn” with “How to Make”.

    Coach Donnelly

  • I think A pulled better. $ 2000 sounds more believable than $ 100000, even though one is in a week and the other in a year. Also, it is not specific. One has to look into it more to find out how to make $ 2000/week.

  • Bob,
    I like those “challenges”. I made a mistake but I already have read the previous comments. Now I am biased. So I asked my wife (she’s an artist). She believes A is the winner.

    What could be the conversations in the prospects mind?

    * $2,000 is just what I could use right now.
    * ‘work’ sounds easy, someone tells me what to do, and I do it.
    * might be a scam but I check it out anyway. maybe it’s for real.
    * “selling”? No, I won’t do that.

    * $100,000 a year selling something … sounds like a business plan.

    Of course, they don’t see the two ads next to each other. Someone with a job might go with A (if hasn’t been ‘burned’ before). Someone with a business mindset might go for B.

    A lot depends on actual source/quality of traffic.

    Can’t wait for the solution.


  • My vote is for B. First of all, six figures is the holy grail for a lot of folks – so $100,000 carries a lot of emotion.

    So does the lack of the word ‘working’ in the headline. People say they’re willing to work for something. But, emotion rules. They want big money with the promise of little effort.

    Finally, “simple” ebooks adds to what I just mentioned above.

  • I’m surprised at the variety of responses. It has to be A. People want to know more about how to earn well without working very hard. Selling e-books to make 100K it is not clear how much work will have to be done to get there, but it’s not difficult for anyone to imagine that result with the right amount of effort.

    However 2K per week for one hour per day says all that needs to be said to those surfing for opportunity. slr

  • I’m shocked that A performed better and would be interested to know what kind of audience it was being advertised to.

    Personally, I see so many “make money on the Net” ads that B stands out much more as something more compelling, something more specific.

    I guess at the same time, being specific and telling people they’ll actually have to create an ebook may sound like too much work for the average Joe who sees a random ad.

    Coach Donnelly

  • I’d say A was the winner because:
    1) As mentioned by a couple of commenters, it’s more immediate. Weeks and days, not years.
    2) It’s more attainable. $2000 is easier to achieve than $100k.
    3) It appeals even to people who think they can’t write or sell or work online.
    4) It’s shorter.
    5) As Henry said, it trivializes the ‘work’ involved.
    6) Also agree with Steve’s comment above: option A discusses the commitment required.


  • I’m voting A. The quotation marks framing “working” and the need only to work “an hour a day” create a cantilever of force and reinforcement hard to pass by.

    Also, e-books don’t seem simple to many who grew up with books “classic.” There’s all that technology to master, and the non-tangiblity of the ebook, tends to further distance a good number of potential prospects from the task, and therefore from the offer.

    “Working” only “an hour a day,” there’s language that’s universally appealing.

  • B was the loser because it tells too much. More to the point, it tells people it’s promoting info on how to do something they probably don’t want to learn about or do — sell e-books — regardless of how much money they might make doing it.

    A, on the other hand, promises info on how to make a lot of money for very little work. Yes, it smacks of get rich quick scamminess as some have said. But those who are curious will at least respond to the ad to learn what it’s about.

    While A outpulled B, it wouldn’t surprise me if B had a higher conversion rate, since the people who responded were predisposed to learning about selling e-books. But I’ve been wrong in these kinds of guesses before.

  • We Factory wholesale Cheap Nike Shoes,Authentic Jordan sneakers, Nike Dunk,Air Force one, Jordan Fusions, Nike Air Yeezy, Nike ATO shoes, Supra Sneakers, Mauri shoes, Gucci shoes, Prada shoes, D&G, Coach, Puma Trainers, UGG, Timberland boots, Adidas, Chanel shoes, Nike Womens shoes, Nike Kids shoes, and carry some pop Hoodies, LRG, A&F, Affliction, Ed Hardy Jackets, BAPE BBC, Evisu, COOGI, True Religion Jeans, Christian Audigier, polo, Lacoste,
    Shirts,New ERA Hats, LV, COACH, CHANEL Handbag…etc, feel free to contact me at any time.
    Free shipping: (EMS/TNT/FEDEX/UPS/DHL).


  • I go for “B” 100k yearly by simply selling ebooks online. It’s far better than the need of working an hour daily in order to earn 2k per week. Though, the choice “A” will result to much earning, it’s still tolerable cause you’ll only missed less than 5%.

  • “A” is probably the winner because I would tend to go for “bite size chunks” than something I would have to build for a year.

  • We just couldn’t abandon your world wide web site ahead of proclaiming that we really loved your beneficial facts you provide within your guests… Will most likely be back often to check on by way of to new posts

  • Great line accessible. We hardly ever at any time permit any normal shut withouth receiving not too long ago released information making use of this reference web site.

  • Like a web website owner I believe the data the following is reallyfantastic. I be thankful for the efforts. You ought to continue the very good work as soon as and for all! Good luck.

Leave a Reply

Your email address will not be published. Required fields are marked *