Bar Leader Home | Archived Bar Leader Issues | Bar Cat Search
The Survey Says: Bars seek member feedback to help them find their way
With her resources limited, Rae Jean McCall, executive director of the Kansas City (Mo.) Metropolitan Bar Association, turned to the electronic survey firm SurveyMonkey to solicit opinions from members as part of a strategic planning push.
“I was really amazed at the analytical tools available through SurveyMonkey,” she says. “The survey helped us turn a corner that we needed to turn.”
At the 80,000-member District of Columbia Bar, Director of Communications Cynthia Kuhn wanted to gauge the effectiveness of the new “e-briefs” being e-mailed to members—about 75 percent of whom have e-mail. An online survey was developed.
“About 70 people responded. It was interesting, but it wasn’t scientific by any stretch,” she says. “It’s hard to write a good survey. We’re bar people. We’re legal writers. We’re not technical people.”
Whether it’s a short critique of CLE or an in-depth review of bar services, developing, using—and paying for—surveys can be a challenge for any bar association. But with the arrival of e-mail and Internet-based survey vehicles such as SurveyMonkey and Zoomerang in the last few years, it has become a bit easier to gather and analyze data—often for just a few hundred dollars a year.
Just how effective are those surveys, and how simple is it to create them and analyze their results? Is there still a place for paper and phone surveys, selective sampling, focus groups, and outside consultants? Those are some of the questions that some bar associations are looking at as they “survey” the changing landscape.
But no matter which form the surveys take, many bar executives and survey professionals agree that the best way to get started is to carefully consider what you want to know, whom you want to know it from, and what you’re going to do with the results. That investment alone, they say, could be the best savings of time, money, and resources.
Online surveys gain steam
“All bar executives like to know what their members think,” says Helen Desmond McDonald, executive director of the Rhode Island Bar Association. “The challenge is trying to get people to respond. They’re lawyers. They’re busy people. If you make it difficult for them, they’ve got other things to do.”
Making it easy for people to respond is the objective of companies such as Zoomerang, SurveyMonkey, QuestionPro, SuperSurvey, and the dozens of other companies that offer online survey tools—tools that are becoming more popular.
“We’re growing like crazy. We have a very high customer satisfaction rating,” says Paula Rivers, general manager and vice president of Zoomerang. “It’s just becoming one more of the tools in the toolkit of organizations. Our goal is to help anyone out there to get fast feedback cheaply and quickly.”
Many online survey providers offer some sort of free version of their software, along with varying prices depending on the services provided. Some providers, such as SurveyMonkey, charge a monthly fee for service, and others, such as Zoomerang, offer discounts to nonprofit organizations.
McCall turned to SurveyMonkey, tempted in part by the company’s $19.95 monthly subscription that can be cancelled at any time. The bar hadn’t done a complete member survey in a decade, and with 86 percent of the bar’s 5,000 members using e-mail regularly, it seemed like a good opportunity to try an online survey, she says.
Using other bars’ surveys as guides, the Kansas City bar crafted an online survey that took about 10 minutes to complete. A mail-in survey was developed for those without e-mail. The combined response rate, McCall says, was 17.8 percent.
Analytical tools built into SurveyMonkey gave bar leaders a good overview of what members were saying, McCall says. Open-ended questions gave the bar another snapshot of what members were looking for.
“It was a real wake-up call to the board of directors. They thought that, ‘Maybe we don’t know what our members want after all,’ ” she says. “We discovered that we’ve got to quit collecting information and do something with the information we have.”
One of the first results, she says, was a redesign of the bar’s annual bench-bar conference, a working retreat for members. Based on survey input from disaffected corporate bar members, the conference title was changed this year to the “Bench-Bar and Boardroom Conference.”
One of the more time-consuming tasks for the Kansas City bar, McCall says, was analyzing other bars’ surveys and adapting them to fit Kansas City’s particular situation. It is similar to the frustration Kuhn and her staff felt as they wrote and developed online surveys.
“As bar execs, to think that we’re survey people is really presumptuous,” Kuhn says. “We really want to listen to our members, not just ask them questions.”
A hybrid approach
A user of more traditional surveys, the Rhode Island bar’s McDonald had heard the talk from other bar executives about online surveys, and had been skeptical. But faced with a demand for a thorough yet cost-conscious membership survey about a year ago, McDonald went with a bit of a hybrid approach. Survey firm 6 Degrees was hired to develop and analyze a survey sent online to a sample of members. A small percentage of bar members without e-mail was provided with a mail-in survey.
Spurred on, in part, by a daily raffle for small prizes, more than 1,700 of the 4,000 people who were e-mailed surveys responded—a return rate of more than 40 percent, which is considered to be a strong sampling. As many as 200 surveys a day were being e-mailed to the bar from members.
“I was expecting everyone to say, ‘Help me make money,’” McDonald says. “What they said was, ‘Your role is to look at the big picture and to do things for the greater good.’ I was skeptical that a survey could give me so much information from my members, but we got good information.”
The bar and 6 Degrees have since developed a series of surveys for specific segments of the bar community. The total cost is expected to run at about $10,000, according to McDonald.
“I think e-mailing is a lot simpler. Lawyers are just pressed for time,” says Glenn Andersen, research director for 6 Degrees. “We package it so it’s extremely simple to use.” The company has done surveying and research for several bars and related associations, such as the New Hampshire and Montana bar associations and the ABA Center for Professional Responsibility.
Andersen says there are three key aspects to electronic surveys that are important to bar leadership: They create a wealth of information for the bar to use, they let members “get their two cents in,” and they help distribute the bar’s message to its members.
The value of in-depth research
The technological gains made in survey software have been so dramatic just over the last five years, Zoomerang’s Rivers figures that “the tools we launched in ’99 probably wouldn’t be acceptable today.” Color graphics and survey interpretations can be made soon after results come rolling in. But even Andersen, a firm believer in technology, says it only provides a part of the information.
“I think traditional research has been effective for many years. The promise of technology is great—but for the results to be effective, you need to get into analysis of the results,” he says. “You shouldn’t throw the baby out with the bathwater.”
Association Management and Marketing Resources is a full-service research and marketing consulting firm that uses SurveyMonkey itself to obtain what company President Stephen Carey calls “Level I” rudimentary survey information for its clients. The company then expands on that information with further research. But he cautions that the survey information alone provides only a limited snapshot that is likely of limited use (see “Know your segments: Targeted marketing for associations,” January-February 2005, page 19).
“As soon as you want to start subdividing your membership and you start asking values questions, then you’re beyond the capacity of your local bar association,” he says. “The new genre of surveys has been misused mightily by associations. Too many executives are using Level I research to make decisions.”
The Pennsylvania Bar Association recently completed a member research project that used a series of focus groups, rather than online surveying, says N. Bruce Pinto, the bar’s communications and marketing manager.
“Online surveys are very good, but there are certain limitations. Not everyone has an interest in using the Internet,” he says. “It’s very useful in giving you a general direction. But further quantitative measure is probably needed to follow up.”
At the Hillsborough County (Fla.) Bar Association, Executive Director Connie Pruitt credits intensive use of focus groups, mail surveys, and roundtable meetings for gathering important information from members and potential members that helped grow membership by 10 percent over the last two years, while improving services to members.
“I found a lot of people who were very excited to do it. The majority of people wanted to be participants,” she says. “We only did online stuff sporadically.”
A change in direction?
While many see the value of outside consultants, the thoroughness of mail surveys, and the face-to-face input provided via focus groups, the lure of online surveys remains strong—thanks in large part to its economics and the ability to reach large numbers of members quickly.
In 2001, the Allegheny County (Pa.) Bar Association spent more than $25,000 in consulting and mailing fees to conduct its first extensive membership survey in more than a decade, according to Executive Director David Blaner. He doesn’t regret the decision.
“There’s a logic to developing surveys. There’s a method to it. There’s a process. And unless you’re trained in that and use it every day, it’s difficult to do,” Blaner says. “It was great to have a professional marketing person working with us.”
But Blaner said one of the findings of the bar’s survey might change how the bar conducts future research. It was found that the bar’s e-mail newsletter is very popular, and that e-mail is now the most favored way for members to communicate with the bar.
“In the last two or three years, survey tools have gotten much better,” he says. “I think we should very strongly consider an e-survey.”
While the D.C. bar was disappointed with some of its forays into the world of electronic surveys, Kuhn isn’t ruling out another try as the bar prepares to embark on member services research.
“We think it’s ultimately easier to do. The question is, ‘How do we make it secure and easy to access?’ ” she says. “We’ll probably look into it to get broad member feedback.”
The key, Kuhn and others say, is to go into any survey project fully prepared and to make sure that the right questions are being asked and that useful information is being gathered to help the bar learn more about its members and improve its performance.
Says McDonald, “You need to have an idea of where you’re going before you ask any questions.”
Getting the most from your online survey
Want to develop an online survey that sings? Zoomerang offers these 10 tips:
1) Clearly define the purpose of the survey. Good surveys have focused objectives that are easily understood. For a survey to be successful, you need to spend time up front to identify, in writing, the goal of this survey and how the data will be used.
2) Keep the survey short and focused. It is generally better to focus on a single objective than to try to create a master survey that covers multiple objectives. Shorter surveys generally have higher response rates.
3) Keep the questions simple. Get to the point, and avoid using jargon.
4) Use closed-ended questions whenever possible, such as yes/no, multiple choice, or rating scale. But take care in writing them, to make sure they don’t force survey takers into picking a “less bad” answer.
5) Keep rating scale questions consistent. Rating scales are a great way to measure and compare sets of variables.
6) Make sure your survey flows in a logical order. Begin with a brief introduction and don’t reveal the survey objective. Start from broader-based questions, moving to those narrower in scope.
7) Make sure you pretest your survey with a few members of your target audience to find glitches and unexpected question interpretations.
8) If you’re sending your survey by e-mail, avoid Friday, Saturday, or Sunday. Mondays are also good days to avoid because many people have loaded in-boxes to wade through on Monday morning.
9) While not appropriate for all surveys, sending out reminders to those who haven’t responded can often provide a significant boost to response rates.
10) Consider offering an incentive. Depending upon the type of survey and survey audience, offering an incentive is usually very effective in improving response rates.
The importance of asking: Some additional tips on surveys
When it comes to surveys, tightly focused questions are great, said Helen Desmond McDonald, executive director of the Rhode Island Bar Association, but you also have to “have a place for people to just sort of vent.”
Won’t that just encourage members to gripe unproductively? McDonald said her bar was surprised, when looking at results from a recent survey, that “a lot of the venting was positive.” And for those who do have a bit of an ax to grind, offering an open-ended place to do that can prevent any bad feeling from coloring the responses to your more closed questions, she said. But the venting wasn’t a simple feel-good exercise for members, McDonald noted—the president of the bar read all those responses, which provided valuable information beyond what could be tabulated and analyzed.
Also key, said McDonald, speaking at the Midyear Meeting of the National Association of Bar Executives, is to avoid simply reading the survey results, sharing them internally, and putting the report on the shelf. “You have to give it back to members of the bar,” she said, explaining that communicating the survey results—and what you intend to do with them—to members shows that you really do care what they had to say.
To get good results requires good participation, and McDonald had a suggestion there, too. The first communication members receive about the survey, she said, should come from the bar president and should explain, simply and clearly, the purpose of the survey and why it’s so important to participate.
McDonald knows many bar leaders are tempted to develop and analyze their own surveys, perhaps through an online service, but she thinks “it’s much better to outsource.” Working with outside consultant 6 Degrees was worth every penny of the $10,000 the bar is paying for a series of four surveys, she said. One of the bar’s aims in this survey project is to draw members out, via questions that probe a bit deeper than standard survey questions, regarding the challenges they face and how the bar can help. The seriousness and scope of the project make it something the bar “couldn’t possibly do in-house,” McDonald said. Also, she said, an outside consultant has the know-how to pull out unusual correlations—such as different responses according to gender and other factors—that the bar might not have considered.
As a plus, a number of other New England bars are conducting or have completed surveys recently, meaning the bars have been able to compare notes.
A survey can be a great tool in improving members’ perception of the bar—which can come in quite handy when there’s a dues increase in the works, said Nancy Diehl, president of the State Bar of Michigan. Diehl, who agreed with McDonald regarding the value of an outside consultant, said the firm the bar hired recommended conducting a series of focus groups throughout the state. After reviewing the data collected in the focus groups, the bar was able to approach members with the message, “Here’s what you said you wanted, and here’s what money we need”—which Diehl said is far more effective than simply asking for more money. The bar did not end up with the increase it sought, but not for lack of member support. Diehl finds great satisfaction in the fact that, although it was ultimately not approved by the state supreme court, the dues increase “sailed through” the bar’s representative assembly—largely because members had been asked what they needed.
A survey isn’t the only tool for gathering member feedback, said Ronald Ward, president of the Washington State Bar Association. Because survey response tends to be low, he said, the Washington bar also sends short queries via blast e-mail, appoints board of governors liaisons who meet with local bar leaders, and conducts “listening sessions” with members of local, minority, and specialty bars. For the past couple of years, Ward said, the bar has conducted half- to three-quarter-day listening sessions on the subject of diversity, also attended by state supreme court judges and judges from other courts.
Whatever the topic, Ward said, the listening sessions have produced real results. The state bar now has a fully funded staff position in support of diversity and has worked in other ways to strengthen ties with minority and specialty bars. Other concrete results from the listening sessions have included establishing a leadership insitute for lawyers who have three to 10 years of experience in practice, and joining the Casemaker online research consortium.
Whether through traditional surveys, informal listening sessions, or some other means, Ward said, it’s critical that bar associations “use whatever works” to get them in touch with members. What’s the ultimate benefit to these outreach efforts? “Ten years ago, the bar was moribund,” Ward said. “We now have more volunteers than we know what to do with.”
Priceless information ... on the cheap
It might have lost favor in today’s cash society after nearly 10,000 years, but that doesn’t mean that bartering is totally out of style—at least, not at the Kansas City Metropolitan Bar Association. In this case, it just has a new name: corporate sponsorship.
“We needed to do some serious strategic planning and we needed to do it on a shoestring,” says Rae Jean McCall, the bar’s executive director. “And we did it for less than $800.”
That meant some creative financing was in order to pay not only for an online survey ($64), but also to hold focus group sessions and interpret data. That’s where bartering became part of the bar’s strategy.
An insurance company paid more than $2,000 to provide meals at 10 focus group meetings in exchange for publicly co-sponsoring the event, known as “Focus on the Future,” McCall says. A court reporting firm provided reporters to produce transcripts of the meetings in exchange for being the other co-sponsor.
A public relations firm was then offered free advertising in various bar publications and venues in exchange for compiling and analyzing the transcripts and for providing a report to the bar’s board of directors. In a related cost-saving move, the bar hired its former communications director—a stay-at-home mother—to moderate the focus group discussions.
The end result? A solid plan for the bar’s future at a fraction of the cost of a full-blown survey.