In Praise of A/B Split Testing

December 9th, 2009 by Bob Bly

Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test.

For an online ad to build my online subscriber list, we tested two headlines:

A–”Earn $2,000 per week ‘working’ an hour a day”
B–”Earn $100,000 a year selling simple e-books online”

One outpulled the other by 25%.

Which do you think was the winner — and why?

Share

This entry was posted on Wednesday, December 9th, 2009 at 9:50 am and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

43 responses about “In Praise of A/B Split Testing”

  1. Twitter Trackbacks for In Praise of A/B Split Testing - bly.com blog - bly.com direct marketing blog [bly.com] on Topsy.com said:

    [...] In Praise of A/B Split Testing – bly.com blog – bly.com direct marketing blog bly.com/blog/general/in-praise-of-ab-split-testing – view page – cached Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split… Read moreDespite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test. View page [...]

  2. Henry said:

    It’s usually stupid to guess which is going to do better… but I’ll do it anyway.

    I would guess Headline A outpulled B.

    $2,000 sounds more realistic even though it’s slightly more than $100K a year.

    Also, the trivializing of ‘work’ might push results toward A.

  3. Debra Dalgleish said:

    I’d guess A because the time period is shorter. We want results now, not a year from now.

  4. JB said:

    I would say that B worked better because:
    - it is specific about how the money can be made (selling e-books);
    - it sounds more believable than A, — the one week time frame in A is too short and a lead up period would be needed to make $2000/wk;
    - the words in B flow better.

  5. Bill Perry said:

    I would say A, like Henry and Debra, because it’s a smaller chunk of time and it’s also closer to the “now”.

  6. Eric S. Mueller said:

    I’d say B. The first one sounds like a scam.

  7. Gerrid Smith said:

    I agree. B is more believable and A sounds like a scam.

  8. Coach Donnelly said:

    I’d be shocked if B hadn’t performed better.

    There’s just something about “$100,000″ and “simple” being used in the headline that sounds better than “working” and “$2,000″. Plus, I agree with the others about it sounding more believable.

    Having said that, I think B could perform EVEN BETTER if you replaced “Earn” with “How to Make”.

    Coach Donnelly

  9. D Michael Miller said:

    I think A pulled better. $ 2000 sounds more believable than $ 100000, even though one is in a week and the other in a year. Also, it is not specific. One has to look into it more to find out how to make $ 2000/week.

  10. Maine Copywriter said:

    I’d guess B because people are greedy. It’s more specific. And nobody likes to “work.”

  11. John W. Furst said:

    Bob,
    I like those “challenges”. I made a mistake but I already have read the previous comments. Now I am biased. So I asked my wife (she’s an artist). She believes A is the winner.

    What could be the conversations in the prospects mind?

    * $2,000 is just what I could use right now.
    * ‘work’ sounds easy, someone tells me what to do, and I do it.
    * might be a scam but I check it out anyway. maybe it’s for real.
    * “selling”? No, I won’t do that.

    * $100,000 a year selling something … sounds like a business plan.

    Of course, they don’t see the two ads next to each other. Someone with a job might go with A (if hasn’t been ‘burned’ before). Someone with a business mindset might go for B.

    A lot depends on actual source/quality of traffic.

    Can’t wait for the solution.

    Yours
    John

  12. Curtis said:

    My vote is for B. First of all, six figures is the holy grail for a lot of folks – so $100,000 carries a lot of emotion.

    So does the lack of the word ‘working’ in the headline. People say they’re willing to work for something. But, emotion rules. They want big money with the promise of little effort.

    Finally, “simple” ebooks adds to what I just mentioned above.

  13. Steve said:

    I’m surprised at the variety of responses. It has to be A. People want to know more about how to earn well without working very hard. Selling e-books to make 100K it is not clear how much work will have to be done to get there, but it’s not difficult for anyone to imagine that result with the right amount of effort.

    However 2K per week for one hour per day says all that needs to be said to those surfing for opportunity. slr

  14. Bob Bly said:

    A is the winner, and I think the reasons Steve cites above may be the correct ones.

  15. Coach Donnelly said:

    I’m shocked that A performed better and would be interested to know what kind of audience it was being advertised to.

    Personally, I see so many “make money on the Net” ads that B stands out much more as something more compelling, something more specific.

    I guess at the same time, being specific and telling people they’ll actually have to create an ebook may sound like too much work for the average Joe who sees a random ad.

    Coach Donnelly

  16. Joe said:

    I’m betting A because most people want instant gratification and $2,000 a week sounds like a quicker turn-around.

  17. Glenn Murray said:

    I’d say A was the winner because:
    1) As mentioned by a couple of commenters, it’s more immediate. Weeks and days, not years.
    2) It’s more attainable. $2000 is easier to achieve than $100k.
    3) It appeals even to people who think they can’t write or sell or work online.
    4) It’s shorter.
    5) As Henry said, it trivializes the ‘work’ involved.
    6) Also agree with Steve’s comment above: option A discusses the commitment required.

    Cheers.
    Glenn

  18. Jay Hepner said:

    I’m voting A. The quotation marks framing “working” and the need only to work “an hour a day” create a cantilever of force and reinforcement hard to pass by.

    Also, e-books don’t seem simple to many who grew up with books “classic.” There’s all that technology to master, and the non-tangiblity of the ebook, tends to further distance a good number of potential prospects from the task, and therefore from the offer.

    “Working” only “an hour a day,” there’s language that’s universally appealing.

  19. Ken Norkin - Freelance Copywriter said:

    B was the loser because it tells too much. More to the point, it tells people it’s promoting info on how to do something they probably don’t want to learn about or do — sell e-books — regardless of how much money they might make doing it.

    A, on the other hand, promises info on how to make a lot of money for very little work. Yes, it smacks of get rich quick scamminess as some have said. But those who are curious will at least respond to the ad to learn what it’s about.

    While A outpulled B, it wouldn’t surprise me if B had a higher conversion rate, since the people who responded were predisposed to learning about selling e-books. But I’ve been wrong in these kinds of guesses before.

  20. www.outlet-apparel.com said:

    We Factory wholesale Cheap Nike Shoes,Authentic Jordan sneakers, Nike Dunk,Air Force one, Jordan Fusions, Nike Air Yeezy, Nike ATO shoes, Supra Sneakers, Mauri shoes, Gucci shoes, Prada shoes, D&G, Coach, Puma Trainers, UGG, Timberland boots, Adidas, Chanel shoes, Nike Womens shoes, Nike Kids shoes, and carry some pop Hoodies, LRG, A&F, Affliction, Ed Hardy Jackets, BAPE BBC, Evisu, COOGI, True Religion Jeans, Christian Audigier, polo, Lacoste,
    Shirts,New ERA Hats, LV, COACH, CHANEL Handbag…etc, feel free to contact me at any time.
    Free shipping: (EMS/TNT/FEDEX/UPS/DHL).

    website: http://www.outlet-apparel.com

  21. Lose The Back Pain System – $46.45 Avg Payout. at Free Anger Management said:

    [...] In Praise of A/B Split Testing – bly.com blog – bly.com direct … [...]

  22. Èn ændring = 196% bedre konvertering said:

    [...] In Praise of A/B Split Testing – bly.com blog – bly.com direct … [...]

  23. Annie Daniel said:

    I go for “B” 100k yearly by simply selling ebooks online. It’s far better than the need of working an hour daily in order to earn 2k per week. Though, the choice “A” will result to much earning, it’s still tolerable cause you’ll only missed less than 5%.

  24. Ryan said:

    “A” is probably the winner because I would tend to go for “bite size chunks” than something I would have to build for a year.

  25. D Bnonn Tennant said:

    So are you gonna put us out of our misery and tell us, Bob…or leave us hanging? (:

  26. Nan Wojtkowski said:

    Superior material included in content material may make upwards to find the top e-book

  27. Latarsha Galer said:

    Hello there business owner! I located your weblog employing Google so when google search optimizer

  28. Jeanna Ratering said:

    We just couldn’t abandon your world wide web site ahead of proclaiming that we really loved your beneficial facts you provide within your guests… Will most likely be back often to check on by way of to new posts

  29. Brice Biegler said:

    Your blog is fairly wonderful, I like your articles, Its really help me to.

  30. Yong Maassen said:

    We appreciate you the straightforward, informative and simple to grasp write-up Savings

  31. Otis Garfield said:

    Great line accessible. We hardly ever at any time permit any normal shut withouth receiving not too long ago released information making use of this reference web site.

  32. Tangela Villenas said:

    Outstanding weblog below! In addition your web site tons upward very quickly! Just what number do you assume you happen to be utilizing?

  33. Kimber Zenisek said:

    I really need, understanding!

  34. Margret Baratto said:

    Like a web website owner I believe the data the following is reallyfantastic. I be thankful for the efforts. You ought to continue the very good work as soon as and for all! Good luck.

  35. gestational diabetes said:

    I believe that you know producing males and women hear whatever you have to state, specifically with an issue i am talking about so important

  36. custom cowboy boots said:

    Numerous thanks expressing your notions about this site.

  37. d3dcompiler_43.dll is missing said:

    Spot on with this write-up, I actually think this amazing site needs a great deal more attention. I’ll probably be returning to read more, thanks for the information!

  38. download d3dcompiler_43.dll is missing said:

    Excellent beat ! I would like to apprentice while you
    amend your site, how could i subscribe for a blog site?

    The account helped me a acceptable deal. I had been tiny bit acquainted of this your
    broadcast offered bright clear concept

  39. mstsc.exe fix said:

    Greetings! This is my first visit to your blog! We are a collection of volunteers and starting a new project in a community in the same
    niche. Your blog provided us valuable information to work on. You have done a outstanding job!

  40. AtBroker.exe xp fix said:

    You can certainly see your skills in the article you write.
    The sector hopes for more passionate writers
    such as you who are not afraid to say how they believe.
    Always go after your heart.

  41. normaliz.dll file download said:

    Helpful info. Lucky me I found your web site by chance, and I’m stunned why
    this coincidence didn’t took place in advance! I bookmarked it.

  42. mfc90u.dll said:

    I appreciate, cause I found just what I was taking a look
    for. You have ended my four day lengthy
    hunt! God Bless you man. Have a great day. Bye

  43. ngen.exe said:

    Since the admin of this site is working, no doubt very soon it
    will be famous, due to its feature contents.

Leave a Reply