The Stags began life at Elm Grove Recreation Ground and in the Surrey Intermediate (Central) League, finishing runners-up in the Premier Division on three occasions prior to moving to the Western Section in 1965, where they spent three years before joining the newly-formed Surrey Senior League in 1968. This coincided with a move to their current Franklyn Road ground, although two years later, when Casuals became founder members if the Suburban League, they spent nine years away form "home", spending a season with neighbours Walton & Hersham and then eight years later at Addlestone's Liberty Lane.

In 1982/83, walton achieved their first major honour - the Suburban League (Southern) Championship - clinched with a 23 match unbeaten run. The Stags just failed to retain the title the following season, finishing runners-up to Sutton United.

Despite reaching the Surrey Premier Cup final in 1987, a slow decline set in, arrested only by a decision to switch across the non-league pyramid to the Surrey Senior League in 1992. Back at Franklyn Road, Casuals appointed current Kingstonian manager Kim Harris as boss and his SSL experience with Hersham RBL saw Walton win the Surrey Challenge Cup in 1994, defeating Holmesdale in a replay.

The 1994/95 season saw promotion to the Combined Counties League following a successful SSL campaign, enabling the Stags to taste senior football for the first time. Floodlights were installed in 1998. 1999-2000 saw their first honour as a senior side, when they lifted the CCL Challenge Cup beating Viking Greenford 4-2.

Five years 2004-05 later the club romped the Combined Counties league by a record margain of 26 points and promotion to the Ryman League South (Level 4) followed, we finished 15th in our first season and 17th last season.