In the February 1963 issue of Bullwinkle Comics (from Gold Key), that machine that Peabody and Sherman use to travel back in time is spelled "WAY-BAC" (capitalized as in the original comic, although everything Peabody says in the original comic is capitalized). This is as close to a contemporary reference to the actual spelling of the thing I've been able to find:
Bullwinkle and Rocky Number 4 (May 1988, Marvel) spells it WABAC, in bold capitals:
"The Moose That Roared" by Keith Scott from 2001 also uses WABAC. (Thank you Amazon.com's "Search inside this book".)
"The Rocky and Bullwinkle Book" from 1996 also mentions it, but spells it "Wayback" (if my memory serves me correctly).
Until I have some sort of actual, in-show reference to the spelling, I'm going with WAY-BAC, simply because it's the older reference.
Edited to add: even older reference to WAY-BAC, from Bullwinkle 1 (Dell, July-Sept 1962):