I've really enjoyed this string of emails on "why and when to run ESM studies."
I'm a fan of ESM studies mostly because the day is such a rich and well-defined
unit of time. You can really get at the event or behavior level and study how
people respond both at work and at home. I think that's really neat and the
information that is gleaned is not as readily accessible with other/longer
frame designs. In the future, though, I’d love to study whether events at the
day level influence people differently over the longer-term. Helping, for
example, can be good and bad at the day level, but it may be just good over a 6
month period because people gain status, knowledge, etc. This, unfortunately,
would require daily studies that last several weeks, and a whole ton of money
to run too :(.
Klodiana
---
Klodiana Lanaj
Assistant Professor
University of Florida
352-392-9760 | Faculty Site
-----Original Message-----
From: dailyworknet-bounce@xxxxxxxxxxxxx
[mailto:dailyworknet-bounce@xxxxxxxxxxxxx] On Behalf Of John Trougakos
Sent: Thursday, September 22, 2016 11:07 AM
To: dailyworknet@xxxxxxxxxxxxx
Subject: [DWK-NET] Re: varied musings on variance
Hi all,
I agree with all the points raised. Also, here are some of my random thoughts:
I might venture to say that when many of the theories that have been used in
our field were developed, they probably largely stemmed from observing and
conceptualizing people in their daily lives (often through qualitative research
or even in lab experiments, but also less formally through personal experiences
and casual observations) and then generalized more broadly to larger
populations. Thus, I would suggest that the origins of many of our theories
start at the within person level and have been applied to between persons due
to the need /desire to generalize or because methodal and statistical methods
in our field hadn't advanced to be able to more fully study within person
phenoma until recently. Additionally, the focus of organizational research for
some time largely focused on the perspective of the organizations. The
individuals making up organizations were usially of interest only to the extent
of examining them as they fit into the contexts of maximizing the effectiveness
of the organization. Ramblimg over now. Have a great day all!
JT
John P. Trougakos, Ph.D.
Associate Professor
University of Toronto
Rotman School of Management
UTSC - Management
Telephone: (416)287-7181
Fax: (416)287-7392
________________________________________
From: dailyworknet-bounce@xxxxxxxxxxxxx [dailyworknet-bounce@xxxxxxxxxxxxx] on
behalf of Koopman, Joel (koopmajo) [koopmajo@xxxxxxxxxxxxx]
Sent: Wednesday, September 21, 2016 7:33 PM
To: dailyworknet@xxxxxxxxxxxxx
Subject: [DWK-NET] Re: varied musings on variance
Hi all:
My view, going at Marcus’ comment a bit differently, is that I think it is
incumbent on reviewers to ensure that the theory matches the design. So, an ESM
isn’t warranted simply because 32% of the variance was within-individual..it
needs to be warranted because there is theoretical reason to expect that the
phenomenon varies on a daily basis. I agree with Dan & Chris that I think much
of our theory does support a within-individual conceptualization, but the
burden should be on the authors to explain this. Pointing to variance is an
empirical answer to a theoretical question, and we shouldn’t let authors off
the hook that easily…
J
From: dailyworknet-bounce@xxxxxxxxxxxxx
[mailto:dailyworknet-bounce@xxxxxxxxxxxxx] On Behalf Of Daniel Beal
Sent: Wednesday, September 21, 2016 4:16 PM
To: dailyworknet@xxxxxxxxxxxxx
Subject: [DWK-NET] Re: varied musings on variance
I agree with Chris. A great deal of theories tested typically at the
between-persons level are based on logic and theorizing at the within-persons
level. Here's a telling example: our department offers an introductory PhD
course that uses Smith and Hitts' book, "Great Minds in Management: The Process
of Theory Development." The first part of that book deals (presumably though
it's debatable) with more micro levels of management, and even refers to the
section as Individuals and Their Environment. The theories covered include,
social cognitive theory, image theory, theories on organizational fairness,
goal setting theory, job characteristics theory, theories related to
organizational commitment, psychological contract theory, escalation of
commitment, and expectancy theory. An odd assortment yes, but reasonably wide
ranging and influential. If you take those that I listed, most of them have
been examined over the years with numerous between-subjects designs and models
that describe properties of individuals. Yet most or even all of them discuss
processes that involve how people generally react to changes in various aspects
of the workplace (e.g., shocks to decision images, unfair procedures, the
setting of difficult and specific goals, reactions to work conditions,
violations of expectations, etc.), which sounds a lot like Chris' description
that the intervention works for everyone. Not all of them play out over the
span of a week or two, but many of them do (or at least have components that
do).
Clearly, this puts me in the "ESM could be used to test most theories" camp,
but that's hardly surprising. I agree with Marcus though that just because it
can be used, doesn't mean it should be or has been used to greatest effect. And
yes, I would balk at feeling confident over having 32% of the variance at the
within-person level, particularly since that likely lumps varying degrees of
error variance in with it. There's a lot going on at both the within- and
between-person levels for many constructs, and I would let the nature of the
theory being examined dictate the approach used. My main point here, however,
is that you can easily frame your theoretical arguments at the within-person
level for most theories that matter in our field. Whether you've done so
convincingly is a totally different question, particularly if you are choosing
your design based on the ease of implementation.
-Dan
On Wed, Sep 21, 2016 at 9:37 AM, Christopher Barnes
<chris24b@xxxxxx<mailto:chris24b@xxxxxx>> wrote:
I'm not sure if I agree that our theories are by default at the between persons
level. I guess anything with a clearly stable trait would be. But aside from
that, our theories are often either ambiguous or may even be implicitly tilted
toward within person effects. Many of them have an implicit assumption that the
intervention works for everyone (on average), which can sounds like a
within-persons theory.
Sent from my iPad
On Sep 21, 2016, at 6:20 AM, Butts, Marcus M
<mbutts@xxxxxxx<mailto:mbutts@xxxxxxx>> wrote:
Hi ESMers,
I had a thought the other day that is tangentially related to Dan’s last
comment about within-person variance. I’ve seen more and more ESM studies
recently where the theorizing doesn’t match or warrant an ESM design, but it
felt like the authors did an ESM because the sampling is done faster and with a
smaller total sample size. As a Reviewer, I think my default is that our
theorizing usually takes a between person type of angle and the sampling should
match that angle. So, I guess my question to the group is when do you decide
an ESM framing and design is warranted (from a researcher or reviewer
perspective)? I know, personally, I find more and more people that say, “my
ESM worked out great, I had 32% within person variance in variable X.” But to
me, that means 68% of the variance is still residing at the between person
level…and empirical evidence of within-person variance has little to do with
theorizing something as a within-person versus between-person phenomenon.
Thoughts?
Thanks all!
-Marcus
<B636040F-F046-4B54-B846-82400A2A3AFB[20].png>
Marcus Butts
Associate Professor
Department of Management
Box 19467
701 S. West Street, Suite 212
Arlington, Texas 76019
T 817.272.3855<tel:817.272.3855>
F 817.272.3122<tel:817.272.3122>
mbutts@xxxxxxx<mailto:mbutts@xxxxxxx>
From:
<dailyworknet-bounce@xxxxxxxxxxxxx<mailto:dailyworknet-bounce@xxxxxxxxxxxxx>>
on behalf of Dan Beal <dbeal@xxxxxx<mailto:dbeal@xxxxxx>>
Reply-To: "dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>"
<dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>>
Date: Monday, August 29, 2016 at 4:42 PM
To: "dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>"
<dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>>
Subject: [DWK-NET] Re: Welcome to the DWK listserv!
Hi all,
In response to Joel's point about selecting out subsets of items, if there's
reasonable evidence of unidimensionality, then extracting out smaller subsets
of items should theoretically not matter. On the other hand, if a scale is
simply high in internal consistency but there's no good information about
factorial structure, then you're right, extracting a few items from a larger
scale could easily change the nature of the resulting scores.
Concerning within-person variance, it's interesting to me that it's sometimes
hard to predict which items will vary more or less. The broadly framed ones
seem to be based on more stable perceptions and so they might be more resistant
to daily fluctuations. For example, if responses to the item are affected by a
lot of different behaviors (like "I can trust my supervisor") then it might
require changes on a number of relevant behaviors to generate a change in
response to the broad item. In contrast though, narrow-domain items (e.g.,
"asked what is the right thing to do") just might not happen too often,
creating the same level of low daily variability. I suppose one strategy is to
shoot for items that maximize specificity (in terms of behavior) while
maintaining a desirable base rate of variability. You could come at it from
either direction... If the item is too broad, break it down into more specific
behaviors that still seem likely to fluctuate; if it's too narrow, broaden it
without over-diversifying the set of relevant behaviors. Sounds easy, eh? In
any event, both Joel's point about changing the nature of the construct and
Chris' observation of low daily variance suggest the need to incorporate some
measurement validation work within our ESM papers (or perhaps that's separate
avenue for publication... Development and Validation of Daily Judgments of
Burnout: The DJB Scale).
-DJB (!)
----------------------
Daniel J. Beal
Associate Professor
Department of Management
Pamplin College of Business
Virginia Tech
On Aug 29, 2016, at 4:46 PM, Christopher Barnes
<chris24b@xxxxxx<mailto:chris24b@xxxxxx>> wrote:
I think the use of those 3 items is reasonable. But I bet there would be a lot
of different opinions about that.
Sent from my iPad
On Aug 29, 2016, at 1:41 PM, Koopman, Joel (koopmajo)
<koopmajo@xxxxxxxxxxxxx<mailto:koopmajo@xxxxxxxxxxxxx>> wrote:
Hi all:
I hope I am not sending this email twice… I sent this over the weekend, but I
think it might not have gone through? I asked a couple folks and they said they
did not get it.
Anyway, apologies if it did and I’m sending twice, but I wanted to weigh in on
the question with daily ethical leadership. Please see below:
--------------------
Hey folks:
So, I just realized that I have some data that can speak to this also. Chris,
did that study measure all 10 items for Ethical Leadership? We did this in an
ESM study, but only chose 3 items: "my supervisor set an example of how to do
things the right way in terms of ethics" | "my supervisor asked 'what is the
right thing to do' when making decisions" | "my supervisor disciplined
employees who violated ethical standards." This was just based on us reading
each of the items and coming to consensus on which we thought might be likely
to have the most daily variance.
For us, these seemed to work pretty well. Each item had 49% or more of the
variance at the within-level (high of 61% for the third item). And, a scale
created of the three items had 48% of the variance at the within-level. And so
Chris, my question about whether all 10 items were measured, is I wonder if
you're getting variance across items in terms of how much within-individual
variance there is. Perhaps some items ("conducts his/her personal life in an
ethical manner") have very little within variance and so are dragging the
measure down? Alternatively, I think your other explanation (acquiescence)
could also easily explain that.
The broader point, I think, that choosing the items carefully is really
important to get variance. But, are we creating different problems? As we know,
it is common in ESM studies to use only a few items. But I always wonder when
doing this how much of the construct we're actually capturing, and even
further, is it even ethical "leadership" anymore? At what point is it a
different construct altogether?
Dan, I've attached a recent Mayer paper with the 10 items in a nice Appendix.
If it helps, I have some highlighted and some underlined...there is a
distinction sometimes made between "moral manager" and "moral person" and so I
have that distinguished for myself (highlighted are moral manager, and from
those we chose our items). And so, this might also have implications for item
choice as well, in that we cover one subdimension pretty well, but completely
omit the other...Maybe Chris' data would show a distinction between these in
terms of variance if they measured all 10?
J
From:dailyworknet-bounce@xxxxxxxxxxxxx<mailto:dailyworknet-bounce@xxxxxxxxxxxxx>
[mailto:dailyworknet-bounce@xxxxxxxxxxxxx] On Behalf Of Christopher Barnes
Sent: Monday, August 29, 2016 12:52 PM
To: dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>
Subject: [DWK-NET] Re: Welcome to the DWK listserv!
You may be right. At least some of these items are asking participants to make
an inference rather than report a behavior.
I think the more concrete the behavior, the more opportunity for daily
variance. There’s also probably a floor effect on some of these items (like
observing a leader punishing someone) that makes it harder to detect any
variance at all at this level of granularity.
From:dailyworknet-bounce@xxxxxxxxxxxxx<mailto:dailyworknet-bounce@xxxxxxxxxxxxx>
[mailto:dailyworknet-bounce@xxxxxxxxxxxxx] On Behalf Of Gabriel Rossetti, ;
Allison S - (asgabriel)
Sent: Monday, August 29, 2016 9:34 AM
To: dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>
Subject: [DWK-NET] Re: Welcome to the DWK listserv!
Hey Chris—I am wondering if these leadership behaviors are as easily observable
as other leader-based phenomenon? For instance, abusive supervision seems to be
more easily observed by employees throughout the organization (e.g., I heard my
boss yelling at someone down the hall; I saw my boss today and he looked angry,
etc.), but for these, it seems as though you really have to be having close,
daily conversations with that leader to know if he/she is being ethical or not.
This item seems as though you might get more variance since it is more of a
generic evaluation:
1.
Today, my supervisor can be trusted
1
2
3
4
5
Just my two cents this sunny Tucson morning. ☺
Allie
____________________
Allison S. Gabriel, Ph.D.
Assistant Professor of Management and Organizations Eller College of Management
The University of Arizona
1130 East Helen Street, McClelland Hall 405FF Tucson, Arizona 85721-0108
Email: asgabriel@xxxxxxxxxxxxxxxxx<mailto:asgabriel@xxxxxxxxxxxxxxxxx>
Phone: (520) 621-1390<tel:%28520%29%C2%A0621-1390>
From:dailyworknet-bounce@xxxxxxxxxxxxx<mailto:dailyworknet-bounce@xxxxxxxxxxxxx>
[mailto:dailyworknet-bounce@xxxxxxxxxxxxx] On Behalf Of Christopher Barnes
Sent: Monday, August 29, 2016 9:29 AM
To: dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>
Subject: [DWK-NET] Re: Welcome to the DWK listserv!
Here is the scale:
Please indicate the frequency, with which your supervisor engaged in each of
the following behaviors today at work, Ranging from 1 = Never to 5 = Five or
more times.
1.
Disciplined an employee who violated ethical standards.
1
2
3
4
5
1.
Discussed business ethics or values with employees.
1
2
3
4
5
1.
Set an example of how to do things the right way in terms of ethics.
1
2
3
4
5
1.
Defined success not just by results but also the way they are obtained.
1
2
3
4
5
1.
Made fair and balanced decisions.
1
2
3
4
5
1.
Had the best interests of employees in mind.
1
2
3
4
5
1.
Listened to what department employees had to say.
1
2
3
4
5
1.
Asked, “what is the right thing to do?” when making decisions.
1
2
3
4
5
1.
Today, my supervisor can be trusted
1
2
3
4
5
1.
Conducted his or her personal life in an ethical manner
1
2
3
4
5
From:dailyworknet-bounce@xxxxxxxxxxxxx<mailto:dailyworknet-bounce@xxxxxxxxxxxxx>
[mailto:dailyworknet-bounce@xxxxxxxxxxxxx] On Behalf Of Daniel Beal
Sent: Saturday, August 27, 2016 3:41 AM
To: dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>
Subject: [DWK-NET] Re: Welcome to the DWK listserv!
Hey Chris,
Could you give us an example item? I find it much easier to think about daily
or monetary variability by kinda walking through answering the items in my head.
Thanks,
-Dan
----------------------
Daniel J. Beal
Associate Professor
Department of Management
Pamplin College of Business
Virginia Tech
On Aug 27, 2016, at 12:08 AM, Christopher Barnes
<chris24b@xxxxxx<mailto:chris24b@xxxxxx>> wrote:
I'll add a little detail to Marcus's recap of my part of the conversation. Some
colleagues and I just collected some data to examine daily sleep as a predictor
of daily ethical leadership. I was invited into the project at a relatively
late stage, with data collected by someone in China. We adapted the wording of
the scale to indicate "today".
Only 11.5% of the variance in ethical leadership was at the day level. At this
point, I can't be sure of the reason why, but these are possibilities:
1. This construct really doesn't vary at the day level.
2. The participation rate was nearly 100%. So maybe people felt pressured to
participate, and gave junk data.
3. Maybe better adaption of the scale is needed.
4. Something else.
Anyway, just thought I'd share a heads up. Tread with caution if you want to
examine daily variance in ethical leadership.
Chris
Sent from my iPad
On Aug 26, 2016, at 8:59 PM, Butts, Marcus M
<mbutts@xxxxxxx<mailto:mbutts@xxxxxxx>> wrote:
Hi DWK netters:
I’ve finally gotten around to creating a listserv for our Daily Work Knowledge
Collaborative. Feel free to use this listserv as a opportunity for any
thoughtful questions/comments about ESM concepts or methods you might have.
For those that were able to get together at the AOM social, we had some lively
discussions about ESM research. For example, Chris talked about example
leadership constructs that he felt don’t vary much from day-to-day, Klodiana
mused about better integrating daily ESM findings with more static findings
that are in opposition (i.e., variable X is harmful over time, but is helpful
day-to-day), and Joel and I discussed multi-source missing data in the context
of ESM. Those types of discussions would be great to have occur over this
listserv. So, feel free to post away! Oh, but one caveat. I’d like to ask
that we steer clear of posts such as “I just got an R&R where Reviewer 2 wants
us to do…How should I address that?” Because it is such a small ESM community,
there is a good chance one of us is a Reviewer on your manuscript. If you are
going to ask “advice” questions, please pose them in a way that best maintains
the blind review process.
Thank you all again for being so enthusiastic about this endeavor! I hope it
benefits us as fellow scholars, and helps ESM research and accepted
guidelines/reporting progress further. Below are the members of this closed
listserv, and some obligatory listserv directions are also provided.
Best,
-Marcus
DWK-NET list members:
Chris Barnes
Dan Beal
Marcus Butts
Mike Christian
Charlotte Fritz
Allison Gabriel
Russ Johnson
Joel Koopman
Klodiana Lanaj
Brent Scott
Kristen Shockley
Lauren Simon
John Trougakos
DWK-NET info:
- To post on your mailing list, simply send email to
dailyworknet@xxxxxxxxxxxxx<mailto:dailyworknet@xxxxxxxxxxxxx>.
- Subscribers can join your list by sending email to
dailyworknet-request@xxxxxxxxxxxxx<mailto:dailyworknet-request@xxxxxxxxxxxxx>
with 'subscribe' in the Subject field OR by visiting your list page at
//www.freelists.org/list/dailyworknet.
Requests are then reviewed by the moderator.
- Subscribers can leave your list by sending email to
dailyworknet-request@xxxxxxxxxxxxx<mailto:dailyworknet-request@xxxxxxxxxxxxx>
with ‘unsubscribe' in the Subject field OR by visiting your list page at
//www.freelists.org/list/dailyworknet.
- Online, searchable archives of your list are available at
//www.freelists.org/archive/dailyworknet Each list's archive is
automatically updated as new messages come in.
- There's a FAQ at //www.freelists.org/wiki/the_faq that you might want to ;
let your subscribers in on.
If you have any questions or comments, at any point, please feel free to
contact us by visiting //www.freelists.org/contact.html or by emailing ;
staff@xxxxxxxxxxxxx<mailto:staff@xxxxxxxxxxxxx>.
-- The FreeLists Team
Marcus Butts
Associate Professor
Department of Management
Box 19467
701 S. West Street, Suite 212
Arlington, Texas 76019
T 817.272.3855<tel:817.272.3855>
F 817.272.3122<tel:817.272.3122>
mbutts@xxxxxxx<mailto:mbutts@xxxxxxx>
<Mayer, Aquino & Greenbaum (2012, AMJ).pdf>
