|
Innovative Research Strategies:
The Population Council’s Horizons Program has tested and utilized handhelds for data collection in a variety of survey settings around the developing world since late 2004. In this interview, Louis Apicella, Senior Data Manager for Horizons, shares the program’s experience using handheld computers. Q: How does data collection using personal digital assistants (PDAs) work? A: The survey questions are individually programmed along with their categorized answer choices into drop-down menus, multiple choice check boxes, or other available configurations. Any questions that may have the need for open-ended text entry such as an “other” option are entered by using the touch screen keyboard or through other forms of written character recognition. If needed, inexpensive portable keyboards can be attached to the handheld device to facilitate text entry. Q: What types of surveys would most benefit from the use of PDAs for data collection? A: All of them! What investigator wouldn’t want their data quicker and cleaner? PDAs combined with the right software packages can be used for many purposes, such as: 1) cross-sectional quantitative surveys where baseline and follow-up data are collected from a population (e.g., those who visit a clinic for a certain service), 2) longitudinal studies where subjects are interviewed multiple times, and 3) true transactional databases, equivalent to the environment found in a doctor’s office where a patient’s records can be viewed and updated in real-time. At Horizons we’ve had great success collecting cross-sectional data in a variety of environments from clinics where hundreds are interviewed to households as part of large multi-country surveys with thousands of records. Q: What are the technological advantages of using PDAs for data collection? A: The most obvious advantage and the one most often mentioned by the data collectors/interviewers is the ability for the software to automatically follow complex skip patterns. Individual interviewers do not need to interpret (and potentially misinterpret) manual skip patterns that would be part of a paper questionnaire. This capability can even allow for the combination of multiple surveys into one using complex skip pattern programming. This significantly shortens the overall interview time and burden on the respondent. Our field experience has shown that it takes 20 percent less time to administer a questionnaire. Another advantage of using handhelds is the ability to program a requirement that the interviewer select a response to the current question before going onto the next question. This ensures that there is no missing data because of an accidently skipped question. PDAs can allow for more robust programmed algorithms for checking data quality which can alert the interviewers of inconsistencies or errors in the data while the interview is taking place, giving them a real-time opportunity to correct the problem. Also, entering data electronically removes the possibility of errant marks or poor handwriting in open-ended or hand-written responses because all information is entered electronically. Finally, one of the most important aspects of this evolution in data collection is that it bypasses manual data entry. There is no longer a need to hire/train data entry personnel, find office space, find large blocks of time, or purchase extra personal computers (PCs) to set up a data station. The errors made during data entry do not exist, and this portion of data management no longer takes place. Data goes from the field directly to cleaning and creation of analytic variables. This is a great advantage for any organization who wants to keep pace with the demands for quick turnaround time from project inception to published findings. Other small but crucial benefits are: no need to find large secure areas for storing surveys for long periods (imagine storing 10,000 surveys x 25 pages per survey, which can result in a stack of surveys 52 feet high!) and all parties can have an exact copy of the raw field data at their fingertips. Q: Isn’t handheld technology expensive? A: The initial costs of equipment and software purchases, which can be substantial, are offset by the costs of paper surveys that are no longer incurred. The biggest cost savings come with the removal of data entry costs (equipment, creating screens, training personnel, actual data entry), in addition to paper and printing costs. Furthermore, since handheld computers and the related software may be used on any project, and possibly utilized by project partners for related activities, cost savings multiply with every round of data collection. Many other tasks that collect information on paper such as registration at mobile clinics, keeping track of items distributed like mosquito nets and school bags, can be moved from paper to PDA. This builds the capacity of organizations and helps them to achieve greater cost efficiency during evaluation activities. Q: What are issues to consider when selecting software? A: It’s important to keep in mind what types of questions are to be asked. For Horizon’s purposes, we decided that at the minimum the software must contain:
In late 2004, after some time testing and review of a variety of software packages we chose to use Perseus’s SurveySolutions. Other software at the time demanded a very high level of programming expertise, lacked functionality, or was prohibitively expensive. Q: How do software packages link the PDAs to a host computer? A: All software consists of two components: one part is installed on your desktop PC and the other part is installed on the PDA. The desktop component is where the actual survey is created and programmed and where the data is processed and exported. It also manages the synchronization with the PDA component of the software, to download field surveys from the PC to the PDA as well as uploading the completed surveys from PDAs back to the database on the PC. The PDA software component is installed on each handheld unit, and it can be programmed to keep track of the number of surveys taken, start and end time of the survey, as well as other information available since the last upload. Q: What are the most important considerations when choosing hardware? A: Field experience has taught us that PDAs should have replaceable batteries. Devices that have a battery hardwired into it make it very difficult to replace, thus tying the life of your device to the life of the battery, which can last as long as the average cell phone battery, i.e., 1.5 to 2 years. Processors should run at 400 MHz or faster, have at least 64 MBs of RAM, and at least one expansion slot for an external data card (necessary for backing up) or external device (such as a keyboard). Q: How do you adapt questionnaires for use with PDAs? A: Traditional survey/questionnaire development consists of designing questions and arranging them by theme to enhance flow. This results in the creation of a clean, highly formatted paper version that would be pre-tested and revised until it is ready for use in the field. With the use of PDAs, there are two development phases for the actual questionnaire. The first is the development of the paper instrument as mentioned above, and the second is adapting the paper version for the PDA interface. The paper version serves as a backup for the interview in case of device failure in the field. Q: What are the advantages and disadvantages of this questionnaire development process? A: Obviously, the extra step in development requires there be more lead-time from when the researchers finish their written instruments to having the survey ready for the field. Converting the survey from a word document to run on a PDA can be time consuming. In fact, the usual time spent to create a user-friendly paper instrument is nearly doubled as a significant portion of time is spent programming the electronic version. However, one of the benefits of longer survey preparation is that many more hours are spent with eyes on the survey. This ultimately contributes to improved flow and survey organization. Q: What are the essential components of training interviewers to use PDAs? A: A regular training schedule, plus two extra days for the interviewers, is recommended. Depending on the local data managers’ skill level and experience, they need to attend the interviewers training sessions in addition to setting aside a couple of days to learn the data management routine. Horizons has developed procedures for training interviewers and data managers in the use of handheld computers for data collection:
Time needs to be spent to ensure that data managers are thoroughly trained in proper power management of the handheld computers, uploading data, downloading new surveys, installing (or reinstalling) software, safely backing up the database to an on- and off-site archive. Supervisors must also check over each interviewer’s work, especially early on as we want to catch any problems at the beginning. By building this capacity, at the end of the collaboration the local partner should be able to conduct future studies using the PDA technology.
Q: What has been the reaction of interviewers to the use of PDAs? A: They love it! I ask for interviewer feedback during and after every survey and it’s always extremely positive. All commented on how easy and quicker it was to gather data, how much more convenient it was to carry in the field as opposed to a large bundle of paper instruments, and not having to worry about correctly following skip patterns. Q: How does the use of PDAs impact obtaining the consent of interviewees? A: Consent can still be obtained using paper and a question can be added to the instrument in the PDA that asks the interviewer if the consent form was completed. If the study decides not to have paper consent forms, questions can be inserted that prompt the interviewer to obtain verbal consent. Q: Is security an issue with the handheld devices? A: We have always required that the interviewers use the PDAs optional password that, just like a regular PC, prevents unauthorized access. Therefore, survey data is never found “laying around” like paper instruments might be, ensuring that confidentiality is never comprised. Q: How do you safeguard the integrity of the raw data? A: Data uploaded directly into the PDA is considered the field data equivalent to the completed paper instruments. Once that day’s data are loaded and appended to the database, a copy of the file is compressed, password protected, and emailed offsite to a Horizons’ data manager. The field data are archived once transferred to a read-only CD. Q: What is the possibility of data loss in the field? A: From our experience to date, there has been one device that failed in the field with the loss of that morning's data. In all other cases, once pre-testing was complete there were no problems. And any data loss that occurred during pre-testing was always traced back to user error. Q: How does preparing a data set obtained through PDAs compare with paper surveys? A: Some data cleaning still exists, but not nearly as much if the data were collected on paper and entered into a database. Performing any sort of double data entry is not necessary as data entry type errors cannot be introduced, and this step of data processing is completely removed. Data coming from many different handhelds to these appended databases takes some concentration and skill so as to not overwrite old data with new data. These procedures must be worked out with each on-site data manager. We prefer appending data as any duplicate records can be easily removed during the data-cleaning phase. Q: Can PDAs be used for languages that do not use the Roman alphabet? A: At the moment, only languages that use the Roman alphabet such as English, French, Swahili, Zulu, etc. are supported by the software. No capacity to create questions using some other type of alphabet exists. Q: Can you provide some specific examples of the kinds of studies and the contexts in which Horizon has used handhelds? A: During the past few years, the Horizons Program and the Population Council has utilized handheld computers with great success, conducting 1,200 baseline and follow-up interviews with HIV-positive prenatal and postnatal women in clinics, and in secondary and tertiary hospitals to assess prevention of mother-to-child transmisssion services in KwaZuluNatal South Africa; a study of 2,500 street, brothel, and home-based sex-workers and their knowledge, access, and use of social services and micro-credit programs offered by nongovernmental organizations and the Indian government and in the states of Karnataka, Maharashtra, Andhra Pradesh and Tamil Nadu, India; a multi-country impact assessment census of 7,500 households and 2,500 adolescents quantifying comprehensive care and support services offered to orphans and vulnerable children affected by family members living with HIV in urban, peri-urban and rural settings in Tanzania, Uganda, and Mozambique to name a few. In all cases data were quickly analyzed and reports produced. For example the largest data collection, the one that took place in Tanzania, Mozambique and Uganda, the 10,000 records were collected in five weeks. This data were cleaned, analyzed and a report was produced five weeks later. Waiting for data entry would have added months to this timeline. People still suggest that additional studies are needed to better estimate the differences in response rates, data loss, time saved, and costs, etc, when compared to using paper surveys but I disagree. Obviously there is a threshold where a sample size or small budget would warrant paper surveys over electronic data capture using PDAs but this threshold lowers continually with the advent of cheaper and more user friendly software and hardware and growing knowledge and expertise needed to conduct surveys using PDAs. I think we’re approaching the time when really the only question you need to ask yourself when pondering whether or not to use handhelds to collect data is would you rather use a typewriter or computer to write a report?
| ||||||
