On-demand feedback

Well-designed, on-demand feedback programs give employees the opportunity to influence their work experience, their growth, and the business when leaders respond to the feedback in a timely and focused manner. The potential risk of on-demand feedback is that the volume becomes overwhelming. There are at least two common ways to deliver comprehensive and strategic insights (Macey & Schneider, Reference Macey, Schneider and Kraut2006; Schiemann & Morgan, Reference Schiemann, Morgan and Kraut2006) without overwhelming leaders. One way is to reduce the number of questions asked (e.g., strategic engagement surveys) while increasing the use of open-ended comments, which provide more context not just more volume. Another is to use event-based feedback (e.g., manager effectiveness surveys deployed when a manager reaches 6 months in role, new hire surveys at 30 days), or passive data (e.g., participation in a talent program, time spent engaging with learning modules) to capture specific information in the moment that can be linked together with other sources at a later time. We have seen both of those tactics work well in practice.

Shortened measures can be just as effective as longer ones. This is especially true in employee surveys like those that are used to measure employee engagement, where many sources of common method variance exist (Podsakoff et al., Reference Podsakoff, MacKenzie, Lee and Podsakoff2003) and for which the construct of interest is complex and susceptible to that variance becoming bias (Spector, Reference Spector2006). Prevailing best practice on mitigating bias is to use an a priori approach in which the potential noise is eliminated before the measure is deployed rather than doing so after the results come back (Conway & Lance, Reference Conway and Lance2010). In our own studies, we have found one- and two-item measures of employee engagement to capture up to 90% and 95%, respectively, of the variance in a typical five-item engagement measure. Furthermore, these one- and two-item measures predicted business outcomes with large and statistically significant effect sizes (Glint Inc., 2017). Although these Glint studies investigated the feasibility of short measures for employee engagement, the methods should be investigated for other attitudinal constructs as well. Interestingly, the validity of shortened measures for attitude measurement has also been validated outside of the psychology discipline (e.g., Ang & Eisend, Reference Ang and Eisend2018).

Advances in natural language processing (NLP) make it possible to analyze volumes of open-ended comments at a level of accuracy similar to that of human analysts (cf., Speer, Reference Speer2018). Leaders can instantly see what the team meant when they said they were underrecognized, and how they felt about it (positive, mixed, neutral, or negative), and isolate prescriptive comments to streamline the solution-generation process. This capability—at this level of accuracy and utility—is only recently being introduced in the organizational survey market after decades of research and applications in fields like computational methods and linguistics.

The integration of multiple data sources into a holistic view is not new, but the ease with which this can be done has increased substantially over the past few years. This has moved well beyond multisourced dashboards into cross-source analysis. For example, it is possible to see in a matter of seconds which aspects of the onboarding experience might be correlated with engagement levels at Year 1 for a particular cohort (e.g., female engineers hired for remote positions). The back-end ability to combine, aggregate, and filter data on the fly means that feedback can be sought on the front end for specific reasons and at the time it is most relevant.

On-demand reporting

Well-designed on-demand reporting makes timely information available in personalized ways that all stakeholders and even employees should see them. Timeliness is solved by making results available instantaneously. Although the benefit of rapid analysis is not felt as strongly at the overall level of reporting (i.e., one-second delivery versus one-day delivery is not likely to make or break an action-taking strategy), managers who are empowered to dynamically cut and reanalyze their data across various team views express the greatest influence on rapid analysis and delivery. Doing so eliminates one of the common excuses that leaders give for not taking action on employee engagement data, that the results are “stale” or “too generic.” This type of on-demand reporting is widely available today for organizational surveys.

Personalization is where “reporting” begins to be complemented and ultimately replaced by “augmented intelligence.” Google Maps does not serve up a list of traffic reports, it just tells you which way to go. Some basic versions of this are available now based on formulas that rank survey items based on a combination of descriptive (e.g., difference to benchmark) and correlational (e.g., effect on the employee engagement outcome score) effect sizes, but these solutions are still generic relative to what is possible with augmented intelligence.

On-demand analytics and augmented intelligence

Business intelligence tools and some advanced employee engagement platforms already make it possible to do basic on-demand analytics with people data. More advanced analytics still require significant manual work, particularly when the relationship of interest requires many contextual factors with yet-to-be-measured effects. Thankfully the field is evolving, evidenced by the availability of real-time analytics highlighting specific issues to address or groups^{Footnote 1} of people to support. Although Weiner & McMahon, Reference Weiner, McMahon, Macey and Fink2020) covered the use of AI in survey action taking, our focus here is the foundations of machine-assisted insights. Our strong belief is that insights from augmented intelligence are differentiated from past reporting by relevance, timeliness, and scalability.

Insights need to be relevant and timely to be useful. In an age that is characterized by myriad drains on attention, having the right nuggets personalized and adapted to the user is immensely valuable. The kinds of insights that are valuable and actionable by a Human Resources Business Partner (HRBP) are different from those of the CEO. When insights are relevant and timely, the chance of a leader dismissing results as “stale” or “unrelated to my goals” is reduced. But relevance also mitigates the risk of analysis paralysis by surfacing insights in a way that motivates action instead of spurring endless exploration. Consider automatically surfacing to a leader that, given the team’s operational goals this quarter and their ratings and comments about “resources,” they should focus on establishing foundational systems/tools and set fewer goals because they have a tendency to overcommit. These benefits of fast and tailored access are increasingly felt as organizations move to strategies that involve rapid dissemination of results. For example, after deciding to release results to managers right after survey close, LinkedIn saw a 36-point year-over-year increase in the percentage of managers who accessed their results on the first day that they were available. Among senior leaders, they saw a 29-point increase year over year in rapid access. These findings tell a compelling story about the power of easy-to-use technology and instantly available insights.

Augmented insights have a predictive quality that is part machine and part human expert. The utility of a model is determined by how well it can predict the future. Predictive quality, measured by how accurately and quickly the insight can be generated, is achieved by automating a model that is both tuned and then maintained as needed by human experts. It is far more helpful to inform leaders in advance than in retrospect about an attrition issue that stems from a perceived lack of growth opportunities among high-performing women who work remotely.

Although much of the above can be done manually, it is hard to do this at scale. The two types of scale that matter the most to augmented intelligence are the amount of data that are used and the number of people who are using it. Scale increases as the amount of information we collect grows. Survey practitioners will be increasingly working with datasets that contain tens or hundreds of thousands of comments, tied to hundreds of demographic and organizational attributes, across hundreds of point-in-time measurements from surveys, performance feedback, and other passive listening entry points. At this scale, generating relevant and predictive insights on demand to a few administrators is possible with minimal computing power. However, doing so across a large user base so that all stakeholders can have relevant and timely insights personalized to their needs requires smart automation (e.g., selective processing) and human-taught machine intelligence that adapts to different user types (executives, managers, individual contributors, etc.).

Computational methods and data models

Advances in computational methods and statistical modeling have allowed for very efficient methods for doing complex analyses in real time (Vicknair et al., Reference Vicknair, Macias, Zhao, Nan, Chen and Wilkins2010) and also for doing large-scale automated machine learning. One key element of these advances is an approach called “parallelization,” whereby a large task is virtually delegated across multiple processors. Imagine multiple people working to calculate one mean of millions of data points: Each person takes a chunk of data and performs the same step on their chunk before bringing them back together to get the final average. Rather than multiple people, multiple processors (sometimes in the same computer, sometimes in different ones) can be used to automatically operate “in parallel” to support processing large datasets extremely quickly.

In a people-centric model, each unit is a person who is associated with demographics, timelines of events, feedback, responses, or even other individuals (generating a graph structure). For example, one dataset might have 50,000 employees of a manufacturing organization that includes ethnicity, department name, tenure, a behavioral outcome like safety incident frequency, and survey data across multiple programs (onboarding, engagement, and safety culture). Along with each employee, there may be HRIS transactions spanning multiple years (e.g., promotions, training, and changes in reporting relationships). Finally, one might have both numerical responses and comment data (e.g., topics, sentiment, prescriptive phrasing, and favorability all preextracted from the text). The end result is a massive dataset that is built around the individual, and parallelization provides a fast framework for using highly sophisticated algorithms with these data on shorter timelines. Better yet, with the advent of cloud-computing services, many of these complex computing processes can be outsourced to platforms that handle parallelization without human intervention. Finally, survey vendors can build products directly on top of these cloud technologies and remove the need for client organizations to build such “tech stacks” in house.

Regarding algorithms, most modern machine-learning approaches fall into two categories: supervised and unsupervised methods. Supervised methods are typically predictive in nature, meaning they are used to produce a final value based on preexisting data. This final value might be a label (e.g., should the employee termination reason “Unhappy with my pay” be labeled as related to “Compensation” or “Team dissatisfaction”?) or a precise numeric value (e.g., the probability of an employee leaving is 65%). In a process similar to that prescribed in our field for validating predictors (cf. Cohen et al., Reference Cohen, Cohen, West and Aiken2003), these supervised methods produce a validated model that is built using input data (say, two historical years of safety data) and is tested for predictive accuracy on an “unseen” pool of data (e.g., a third year of safety data; Gholamy et al., Reference Gholamy, Kreinovich and Kosheleva2018). On the other hand, unsupervised methods are often used to impose structure on data rather than produce a predicted value or label. One common unsupervised method is “clustering,” which is an approach for surfacing natural groups that emerge in the data (e.g., Are there unique, unseen patterns of employee responses to an engagement survey that might warrant action planning differently for these “groups”?).

The combination of these statistical methodologies with advances in technology like parallelization means that massive data sets can be scanned for statistical anomalies or relationships and those relationships can be further tested against held-out data. This process is like having a team of data analysts working to find deeply interesting insights and only surfacing the most relevant and accurate insights, automatically. Interestingly, these algorithms can be further optimized through a process called “tuning” to boost insights that are linked to specified outcomes of interest (e.g., turnover, safety), generating highly personalized insights. Furthermore, multiple insights can be statistically linked together in ways that drive clearly to specific action. For example, a linked insight that could conceivably be generated automatically is “Did you know that factory workers in the following jobs have 60% fewer injuries when they report being satisfied with their training and that training satisfaction is highly correlated with the completion of course X for workers with tenure < 3 years, but course Y for workers with tenure ≥ 3 years?”

Data protection and privacy

In order to be compliant and ethical across the world, we must deliver augmented intelligence and its personalized insights in a way that protects the confidentiality of people giving feedback, and a people-centric architecture enables that. We believe the field can go beyond compliance, not only to protect users but also to proactively enhance their experience through language- and culture-based personalization that makes people feel valued. Because the data are centered around the individual and metadata can be used to manage confidentiality and access permissions, a well-designed data architecture democratizes access to insights while preserving commitments to respondents’ data and privacy. With a people-centric architecture, it could even be possible for companies to facilitate the portability of peoples’ own data so that they can bring their data with them to new organizations without violating company or individual data protections. Some large technology companies like Facebook have already begun to offer such functionality in the wake of increased public scrutiny regarding personal data.