The Museum opened on October 3, 1981, originally called The Valley Museum of Northwest Art. The vision of photographer Art Hupy, the Museum lived, for its first 14 years, on the second floor of the historic Gaches Mansion in La Conner. In 1982, the Museum board began to look for a larger space. Plans for relocation and expansion simmered over the next decade while support for the Museum grew. During this time the Museum mounted many exhibits, featuring artists including Mark Tobey, Clayton James, Philip McCracken, Mary Randlett, William Ivey, George Tsutakawa, Flora Mace and Joey Kirkpatrick, Paul Havas, and others.

In 1991, the Museum board and a legion of volunteers began a major fundraising drive to acquire a new home. About that time, a commercial building in downtown La Conner, with 12,000 square feet, came on the market. After two years of negotiation, the deal closed and renovations began. Given a fresh face inside and out by the Henry Klein Partnership, architects, the renovated building on South First Street gave new life to its founders' visions.