In Praise of A/B Split Testing
December 9th, 2009 by Bob Bly
Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test.
For an online ad to build my online subscriber list, we tested two headlines:
A–”Earn $2,000 per week ‘working’ an hour a day”
B–”Earn $100,000 a year selling simple e-books online”
One outpulled the other by 25%.
Which do you think was the winner — and why?
This entry was posted on Wednesday, December 9th, 2009 at 9:50 am and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
December 9th, 2009 at 9:53 am
[...] In Praise of A/B Split Testing – bly.com blog – bly.com direct marketing blog bly.com/blog/general/in-praise-of-ab-split-testing – view page – cached Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split… Read moreDespite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test. View page [...]
December 9th, 2009 at 10:16 am
It’s usually stupid to guess which is going to do better… but I’ll do it anyway.
I would guess Headline A outpulled B.
$2,000 sounds more realistic even though it’s slightly more than $100K a year.
Also, the trivializing of ‘work’ might push results toward A.
December 9th, 2009 at 10:37 am
I’d guess A because the time period is shorter. We want results now, not a year from now.
December 9th, 2009 at 11:12 am
I would say that B worked better because:
- it is specific about how the money can be made (selling e-books);
- it sounds more believable than A, — the one week time frame in A is too short and a lead up period would be needed to make $2000/wk;
- the words in B flow better.
December 9th, 2009 at 11:46 am
I would say A, like Henry and Debra, because it’s a smaller chunk of time and it’s also closer to the “now”.
December 9th, 2009 at 12:36 pm
I’d say B. The first one sounds like a scam.
December 9th, 2009 at 3:51 pm
I agree. B is more believable and A sounds like a scam.
December 9th, 2009 at 6:14 pm
I’d be shocked if B hadn’t performed better.
There’s just something about “$100,000″ and “simple” being used in the headline that sounds better than “working” and “$2,000″. Plus, I agree with the others about it sounding more believable.
Having said that, I think B could perform EVEN BETTER if you replaced “Earn” with “How to Make”.
Coach Donnelly
December 9th, 2009 at 6:29 pm
I think A pulled better. $ 2000 sounds more believable than $ 100000, even though one is in a week and the other in a year. Also, it is not specific. One has to look into it more to find out how to make $ 2000/week.
December 9th, 2009 at 9:15 pm
I’d guess B because people are greedy. It’s more specific. And nobody likes to “work.”
December 10th, 2009 at 6:38 am
Bob,
I like those “challenges”. I made a mistake but I already have read the previous comments. Now I am biased. So I asked my wife (she’s an artist). She believes A is the winner.
What could be the conversations in the prospects mind?
* $2,000 is just what I could use right now.
* ‘work’ sounds easy, someone tells me what to do, and I do it.
* might be a scam but I check it out anyway. maybe it’s for real.
* “selling”? No, I won’t do that.
* $100,000 a year selling something … sounds like a business plan.
Of course, they don’t see the two ads next to each other. Someone with a job might go with A (if hasn’t been ‘burned’ before). Someone with a business mindset might go for B.
A lot depends on actual source/quality of traffic.
Can’t wait for the solution.
Yours
John
December 10th, 2009 at 12:21 pm
My vote is for B. First of all, six figures is the holy grail for a lot of folks – so $100,000 carries a lot of emotion.
So does the lack of the word ‘working’ in the headline. People say they’re willing to work for something. But, emotion rules. They want big money with the promise of little effort.
Finally, “simple” ebooks adds to what I just mentioned above.
December 10th, 2009 at 12:58 pm
I’m surprised at the variety of responses. It has to be A. People want to know more about how to earn well without working very hard. Selling e-books to make 100K it is not clear how much work will have to be done to get there, but it’s not difficult for anyone to imagine that result with the right amount of effort.
However 2K per week for one hour per day says all that needs to be said to those surfing for opportunity. slr
December 10th, 2009 at 1:59 pm
A is the winner, and I think the reasons Steve cites above may be the correct ones.
December 11th, 2009 at 3:15 pm
I’m shocked that A performed better and would be interested to know what kind of audience it was being advertised to.
Personally, I see so many “make money on the Net” ads that B stands out much more as something more compelling, something more specific.
I guess at the same time, being specific and telling people they’ll actually have to create an ebook may sound like too much work for the average Joe who sees a random ad.
Coach Donnelly
December 11th, 2009 at 11:31 pm
I’m betting A because most people want instant gratification and $2,000 a week sounds like a quicker turn-around.
December 13th, 2009 at 6:22 pm
I’d say A was the winner because:
1) As mentioned by a couple of commenters, it’s more immediate. Weeks and days, not years.
2) It’s more attainable. $2000 is easier to achieve than $100k.
3) It appeals even to people who think they can’t write or sell or work online.
4) It’s shorter.
5) As Henry said, it trivializes the ‘work’ involved.
6) Also agree with Steve’s comment above: option A discusses the commitment required.
Cheers.
Glenn
December 16th, 2009 at 6:34 am
I’m voting A. The quotation marks framing “working” and the need only to work “an hour a day” create a cantilever of force and reinforcement hard to pass by.
Also, e-books don’t seem simple to many who grew up with books “classic.” There’s all that technology to master, and the non-tangiblity of the ebook, tends to further distance a good number of potential prospects from the task, and therefore from the offer.
“Working” only “an hour a day,” there’s language that’s universally appealing.
December 18th, 2009 at 12:12 am
B was the loser because it tells too much. More to the point, it tells people it’s promoting info on how to do something they probably don’t want to learn about or do — sell e-books — regardless of how much money they might make doing it.
A, on the other hand, promises info on how to make a lot of money for very little work. Yes, it smacks of get rich quick scamminess as some have said. But those who are curious will at least respond to the ad to learn what it’s about.
While A outpulled B, it wouldn’t surprise me if B had a higher conversion rate, since the people who responded were predisposed to learning about selling e-books. But I’ve been wrong in these kinds of guesses before.
December 28th, 2009 at 3:40 am
We Factory wholesale Cheap Nike Shoes,Authentic Jordan sneakers, Nike Dunk,Air Force one, Jordan Fusions, Nike Air Yeezy, Nike ATO shoes, Supra Sneakers, Mauri shoes, Gucci shoes, Prada shoes, D&G, Coach, Puma Trainers, UGG, Timberland boots, Adidas, Chanel shoes, Nike Womens shoes, Nike Kids shoes, and carry some pop Hoodies, LRG, A&F, Affliction, Ed Hardy Jackets, BAPE BBC, Evisu, COOGI, True Religion Jeans, Christian Audigier, polo, Lacoste,
Shirts,New ERA Hats, LV, COACH, CHANEL Handbag…etc, feel free to contact me at any time.
Free shipping: (EMS/TNT/FEDEX/UPS/DHL).
website: http://www.outlet-apparel.com
January 6th, 2010 at 1:57 pm
[...] In Praise of A/B Split Testing – bly.com blog – bly.com direct … [...]
January 15th, 2010 at 9:22 pm
[...] In Praise of A/B Split Testing – bly.com blog – bly.com direct … [...]
February 6th, 2010 at 11:09 pm
I go for “B” 100k yearly by simply selling ebooks online. It’s far better than the need of working an hour daily in order to earn 2k per week. Though, the choice “A” will result to much earning, it’s still tolerable cause you’ll only missed less than 5%.
April 4th, 2010 at 3:20 pm
“A” is probably the winner because I would tend to go for “bite size chunks” than something I would have to build for a year.
May 17th, 2010 at 6:27 pm
So are you gonna put us out of our misery and tell us, Bob…or leave us hanging? (: