10 Ways To Become A [Bad] Learning Scientist

So far, in our work towards the Learning Science book, Sae, Geoff, and I have written some thoughtful articles about complex ideas like Learning Ecosystems, Social Metacognition, and even the Nature of Knowledge itself. We’ve tried to provide a thoughtful, practical, and research-grounded narrative. 

But… what if there were an easier way… a secret, accelerated path to success that skipped past the lengthy analysis and plodding methodologies?

In today’s post, we’re offering a handy guide that will help you to add sparkle to any idea, and provide the tips you need to wow clients and partners with the thinnest veneer of empiricism and credibility – without all of that boring work.

After all, on the 1st April, why strive and struggle, when there is a shortcut?

Read on for out top 10, all powerful tips that can turn your lacklustre L&D report into a powerful, scientific research paper, to impress your friends, and wow your boss:

Part 1: Finding Published Research

Our first ideas involve creative ways to use published research (the best sort, right?) to build on an academic foundation and prior evidence. These tips will help you create [the appearance of] rigour and quality, and let everyone know you’ve done your ‘due diligence’. Pretty much whatever you are looking at, you can find some relevant articles to support it… Here’s how: 

1. Use evidence from individual laboratory studies (because the real world is just like a lab). Nearly every concept in L&D has been researched… somewhere. Frequently, these studies are conducted in deliberately constructed environments with the object in question (such as a particular instructional method) carefully isolated for evaluation in a helpful (unrealistic) vacuum. Some of these studies include just a handful of participants, and about half of the time, their positive results are just a statistical fluke [1] — so you’re certain to find a publication with some shiny statistics to support whatever you might claim! When you have found it, just paste the citation into every document.

2. Be creative in how you generalise research (because ‘context’ is just a detail really). Closely related to the recommendation above, this next piece of advice is to apply research findings broadly and into new spaces. Don’t worry about the populations involved, whether the learning conditions were realistic, or if the results are replicable. Just search online, find an article with good numbers, et voilà! If you need a good example of this, just look at the work on Growth Mindset: Empirically examined in one domain and context, and then widely generalised as if it were a universal ‘thing’.
   
   
3. Emphasise the ‘statistical significance’ (because like probably 99% of the time you’ll be right). A lot of people aren’t well-versed in parametric statistics, but most people in the L&D community have probably heard of ‘alpha’ or ‘p-value.’ Your best approach is to showcase that statistic prominently, and when you cite foundational research, make sure to emphasise its p-value (for example, “p < 0.05”). Consider adding exclamation marks! We advise liberally using the phrase ‘statistically significant’ or even just ‘significant’ when referring to research with a p-value of less than 0.05. Using the word ‘significant’ lends credibility to the research findings. Consider this the research equivalent of ‘artisanal’ (when describing cheese) or ‘craft’ (when discussing beer).

Part 2: Creating Original Research

Next, you probably need to do some ‘validation’ research on your own, so you can ‘prove’ that your specific L&D offering is effective. Here are some tips for getting the best results: 

4. Use a pretest/posttest design (because ‘better’ is ‘better’ in every case). Let’s say you’ve made a new training program and want to show how awesome it is. (Think: performance review coming up soon.) You need to collect some training outcomes. Here’s how to make sure they look good. First, give participants a pretest, then do your training, and afterwards give them a posttest. Don’t worry too much about what happens in between the tests. You’ll probably get a medium effect size improvement [2] just from the retest effect [3] alone – which is like free progress really. And magically, this works best if the two tests are the same, but that’s not actually even a requirement. Adding some test-prep and coaching into your training, you’ll get even bigger results!
    
5. Use Placebo and Hawthorne effects liberally (because if you can measure it, it counts). The medical community has studied [4] placebos extensively and found them to have massive impacts. Although the percentage varies depending on a study’s purpose and participants, it’s often around 20–30% – but can be upwards of 72% [5]. A related organisational phenomenon is the Hawthorne effect [6], which basically shows that when workers are given special attention and observed, their performance increases. So, you can easily find impressive results simply by creating an intervention that piques learners’ Placebo/Hawthorne responses. Just fuel their expectations, give them some attention, and make sure they know that you’re watching. This technique is particularly useful as it liberates you from actually creating effective learning.
   
   
6. Design your experiment for success (because, why take the chance?). Once you have your pre- and posttests and Placebo/Hawthorne triggers ready to go, it’s time to create the study’s protocol. Some experimental designs work better than others. Specifically, you’ll have bigger effect sizes [7] if you use (a) correlational or quasi-experimental designs (in other words, avoid participant randomization and blind/double-blind assignments!), (b) proximal testing (evaluations that closely mirror the intervention and are completed close to it, like a written posttest completed shortly after training), and (c) a small population (stick to fewer than 500 people). Remember: we use veneers because they allow us to use a valuable material (the beautiful veneer) very cost effectively. Think of your time as the veneer: the more you save, the more of The Mandolorian you can catch up on later.
   
   
7. Count everything (because MEASUREMENT FOR THE WIN). We’ve already talked about collecting pre and posttest outcomes, but you’ll need more than that! Collect data on everything, so that you have a lot to play with after the experiment. Start by asking for detailed demographic data, because you might find your experiment works best for left-handed, bilingual women ages 25 to 50 – so you’ll need all of those variables in-hand to find that needle in the haystack. Next, collect data on anything that’s countable, for example, number of hours spent in training or number of words read. You can also selectively count parts of self-response surveys, such as the number of items rated above ‘satisfactory’. 
   
   
8. Use statistical tricks (because it’s not cheating if it’s just maths). If you’ve followed our prior recommendations, then you already have some impressive results, but if you’re still struggling (or want to boost the results further), you can massage the data. There’s a large toolkit of data-dredging hacks [8] that (bad) scientists have perfected over the years, such as p-hacking [9] (manipulating the statistics to get a suitable p-value), fishing (playing with the statistics until some superficially nice-looking result appears, whatever it might be), or simply continuing to run the experiment [10] until you get enough data to support some desired result. This is all good: after all, what’s the point of putting in the effort unless you can show success? Nobody ever learnt from getting anything wrong. And that’s a [statistically significant] fact!

Part 3: Communicating About Your Amazing, Evidence-Based Results

We’ve made great progress so far. Using the foundations of western scientific methodology, we’ve been able to add real value at minimal effort. But there’s one more step. After you’ve assembled a supportive literature review and conducted your own empirical testing, it’s time to share your results. There are plenty of good guides to writing (bad) research articles, with excellent advice such as, “never explain the objectives of the paper in a single sentence…in particular never at the beginning” [11] and make sure to “use different terms for the same thing” [12]. In addition to that great guidance, we’ll add two more suggestions:

9. Build on personal experience (because feeling IS believing). People love personal stories, and we’ve all experienced education and training before – so, we’re all mini-experts on the subject of learning. Work with that. Use your own experiences or, even better, reference common human experiences as naturalistic evidence. After all, we’re all humans, and we all think and learn in the same ways. So, these common experiences will help people relate to your new L&D idea. Draw readers or customers in with anecdotes about personal experiences, and then generalise from those experiences to help explain and support your concept.
   
   
10. Use snazzy terminology (because with a growth mindset, we can be neuro-informed): Like a well-tailored suit on a businessperson, certain words add polish that can make or break your L&D idea. At a minimum, make sure to use both ‘Machine Learning’ (ML) and ‘Artificial Intelligence’ [13] (AI). (Don’t worry if you don’t actually use AI because a lot of so-called AI startups don’t either! [14]) Next, pick a few L&D terms that describe your idea or offering. Finally, include a few classic innovation words, like ‘emerging’ or ‘cutting-edge’, so that people know this is a new concept. Don’t worry if this seems like hard work: Sae has put together a table to help. Start with the following prompt, and then select a word from each column to fill it in:

Our concept uses AI/ML and [column 1], [column 2] [column 3] to optimise [column 4].

Column 1	Column 2	Column 3	Column 4
innovative	personalised	algorithms	employee engagement
emerging	cloud-enabled	serious games	bench strength
virtualized	data-driven	analytics	growth mindset
agile	mobile-first	master classes	cross skilling
bleeding edge	big data	expert seminars	design thinking
synchronous	extended reality	gamification	double-loop learning
net-centric	neuro-informed	blended systems	core competencies
real-time	adaptive	microlearning	learning fidelity
context-aware	evidence-based	virtual classrooms	team workflow
right-sized	hybrid learning	experiential learning	your business ecosystem
digital	neurolinguistic	visualisations	upskilling
higher-order	self-paced	instructional methods	learner empowerment

Conclusion

To summarise: if you follow these 10 steps you should be well positioned to become a published learning scientist with a spate of innovative, evidence-based AI/ML concepts (among other more questionable descriptors) tied to your name. 

BONUS tip!

Make beautiful data visualisations. Any data represented in an infographic is automatically more valid than a table. Ideally you should embellish your presentations with animations. To avoid confusion, eliminate distractions such as standard deviation notions or error bars, which just get in the way of a good story. Instead, opt for basic graphs wherever possible, like bar charts with just one or two items. You can, for example, make a dashboard of vanity metrics such as number of hours spent learning, smile-sheet scores, change in pre- to posttest results. Basically, anything that is countable can be included (so long as the numbers look right, of course). And use orange, because it’s a warm colour, and everyone loves a winner.

[1] https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/

[2] https://www.illuminateed.com/blog/2017/06/effect-size-educational-research-use/

[3] https://onlinelibrary.wiley.com/doi/full/10.1002/ets2.12300

[6] https://psycnet.apa.org/record/2000-13580-004

[7] https://evidenceforlearning.org.au/news/effect-sizes-in-education-bigger-is-better-right

[8] https://catalogofbias.org/biases/data-dredging-bias/

[9] https://files.de-1.osf.io/v1/resources/xy2dk/providers/osfstorage/623224d733d8540487f8ad21?action=download&direct&version=2

[10] https://theness.com/neurologicablog/index.php/p-hacking-and-other-statistical-sins/

[11] https://pubs.acs.org/doi/10.1021/ac2000169

[12] https://www.elsevier.com/connect/authors-update/10-tips-for-writing-a-truly-terrible-journal-article

[13] https://www.verdict.co.uk/ai-in-education-buzzwords-hyperbole/

[14] https://www.theverge.com/2019/3/5/18251326/ai-startups-europe-fake-40-percent-mmc-report

#WorkingOutLoud on Blogging

Just sharing a quick reflection today, as i write about blogging for the doctorate. 

The Blog is written in the context of the Social Age, and explores that context, and one aspect i discuss at length is the evolving nature of knowledge itself.

The nature of my work is not to discover ‘a’ truth, for there is no single truth, but rather a landscape of ‘meaning’ – individually constructed, socially co-created, dynamic as opposed to fixed, and contextual. My work is as a cartographer of this landscape.

This last paragraph illustrates the core contention of my work: that we inhabit a space of belief more so than knowledge alone, and that belief is what fills in the gaps. We construct our view of the world and then inhabit it as if it were real. My work walks through this landscape, collecting fragmentary ideas and the Blog (alongside my head, or as an extension of it) is where this knowledge is stored.

And this second paragraph illustrates the nature of my work as both philosophical enquiry or, possibly, cod psychology. My work draws maps that may mislead, or lead to treasure, if we choose to believe in the landscape that it illustrates.

The Blog carries out this function through various models of enquiry, some of which use a broad lens, others of which are more focussed:

Research Based Enquiry: more traditional global research projects like ‘The Landscape of Trust’, ‘The Landscape of Communities’, ‘Quiet Leadership’, ‘What it means to Belong’, and so on. These projects, run through the blog and into the engaged blog community typically have a research statement, gather primary data (albeit typically narrative and qualitative), and seek to use these data to derive understanding.

Reflective Enquiry: such as the work on humility, leading at the intersections, the nature of connection, complexity. In this mode, the Blog is more a sequential narrative space, 

Dialogue Based Enquiry: both in dialogue with my Community, but also with myself, i often refer to, rewrite, or critically reappraise previous work. It is normal for me to rework pieces, or to revisit underlying topics, sometimes as many as a dozen times.

Sequential Enquiry: in some instances i return to topics with regularity, such as the sequence of ‘Maps of the Social Age’ or ‘Learning Technology Maps’, which develop year by year. These ‘maps’ are not simply sequential additions (as more territory is visited), but rather may form a revised ‘way of knowing’.

For example: my latest work on Quiet Leadership explores fairness, through a dialogue based enquiry framework, but it is not my first attempt to tackle this subject. I have previously shared a ‘Framework for Fairness’, which attempted a more traditional diagnostic approach and representation, but which ultimately proved unsatisfactory. Similarly, the work ultimately published in ‘The Humble Leader’ on ‘humility’ originated in blog posts about Social Justice in San Francisco almost a decade ago.

Fragmented Enquiry: both the most useful and most challenging aspect of the Blog is it’s fragmented nature, but this fragmentation is really it’s greatest strength. The Blog is not ‘an’ enquiry, and nor is it one focussed narrative. It is a jumble of ideas, many of which represent either data or knowledge without context. This is almost a unique feature of the blog: it is not a space of synthesis (or rather, it can be, but is not obligated to be). It holds knowledge or it’s derivatives and precursors in both space and time, to be dredged up or buried, synthesised or rewoven at a later date.

So: the blog is a memory bank, but in this function it differs from, the one i hold between my ears.

Whilst my brain remembers, and holds knowledge, these memories and that knowledge is constantly re-contextualised by my current worldview and experience, and even my environmental context. The Blog, by contrast, remains constantly fixed and decontextualised.

In this sense, it is easier to explore the origin and evolution of ideas in the blog, through the progression of static artefacts, than it is in my head, which would require me to construct a narrative of understanding through my lens of myself ‘today’.

One could potentially argue that the Blog therefore represents a different way of knowing, allowing me both the fortune of my contemporary perspective and knowledge, whilst also cross linking to my legacy structures of knowing, in quite a detailed sense.

Maps

My work is filled with landscape metaphors, which leads me into thinking about maps.

Maps that show:

1. A landscape, representing a truth, with accuracy, and to a uniform scale
2. A landscape, representing what i care about, with bias, and scaled to interest or the contemporary
3. A landscape, representing things i do not know, populated by whispers and hearsay
4. What i’m thinking of, represented by words and their proximity to each other
5. What i know, represented by boundaries and borders
6. What lies beneath, represented by layers and strata
7. Terrain, represented by contours or moulding, showing peaks and troughs and uncrossable chasms
8. Connection, represented by roads, tramways, rail lines or commonality, shared beliefs, or proximity
9. Resources, represented by surplus and deficit, with overlays of power and control
10. The boundaries of the public and private, the owned and the free
11. Belief, and how it clusters, overlaid by the norms of culture and behaviour
12. Places that i have been to, but will not return to. Ideas that i will not have again
13. Shared spaces and communally owned ideas. 
14. How things flow, and how flow is controlled by dams, sluice gates, ideas, beliefs, or force.
15. How things have changed, and from what and into what
16. Things that never change. A map that is carved in stone.
17. The ephemeral, a map like a Polaroid, showing the impermanent in flow
18. A map of hidden meaning, which is only apparent from above
19. A map to help you get lost, which can never be used
20. A map of the self, which can never be read

#WorkingOutLoud on Blogging

I’ve been reflecting on the process of Blogging as part of the doctoral programme work. The Blog is really a monologue more so than a conversation. Considered as my external brain, it’s a series of 2,193 separate statements so far, each of which reflects what was passing through my head on the day it was written. The shortest posts will be a simple paragraph. The longest are full length essays of several thousand words.

I spend anywhere between 15 minutes, and four hours a day writing and illustrating this work. And then i let it loose to a subscribed global audience, from where it is typically shared onwards to an open global audience.

Over time, the Blog has evolved, in my understanding, from primarily a content sharing channel, to a way of being. Being a thinker, and a writer.

It may sound odd to hold an identity as ‘thinker’, but in this i am referencing a more deliberate act of consideration. Almost a meditative act: of connection, sense making and storytelling, through mixed media.

The books (17 so far) all originate from thinking in the Blog. The notion of ‘The Social Age’ was born there. My consulting work, and my Programmes, my teaching and my thinking, all are grounded here.

I say ‘here’ but latterly The Blog has taken flight from it’s ‘home’ on WordPress to now being cross published on LinkedIn, where engagement and growth has accelerated. So The Blog has transitioned from being a space, to being a conversation, or possibly a community.

I came to blogging in 2010, although the generally credited first blog was published in 1994. This digital writing space follows in a longer tradition of writing and, sits somewhere between individual journaling and diary writing, and the publication of diaries, pamphlets and letters, in that it is a personal space (although not private), and it is published (although not perfect).

There is nothing specifically unique about blogging in terms of creative expression, but rather it changes the context of that expression, and the moderation and validation of that expression, and by doing so it changes what that expression can do.

The advent of blogs was a personal affair, but the field has not been static: Organisations and brands have sought to appropriate the format in service of fostering culture and knowledge, or brand perception, and dedicated technologies have emerged and evolved which have, in turn, shaped the practice.

The rise of the Corporate Blog has been both as prolific as it is uninspiring, probably because the change in context, from authentic personal, to broadcast formal, erodes the validity of the mode itself.

I would argue that the blog, with it’s internal validation, external visibility, social accountability, and almost stream of consciousness flow, is a very different thing than a proofed, edited, controlled, and formally mandated publication, and to move away from these principles and features invalidates the format.

For me, the essence of a blog, and hence the key mechanism of engagement, is the authentic self. In this, it is like a diary: the ‘self’ is what gives validity, even though that validity is held as imperfection.

Where historically i would have used a notebook to keep notes, latterly i use the blog.

Central to this are certain key ideas:

• That writing can be incomplete or fragmentary
• That writing can be imperfect or wrong
• That writing can be selfish, vulnerable or confrontational
• That writing here exists in fluid context
• That writing does not have to make sense, and in some contexts should stretch the boundaries of comprehension
• That rules are not universal

That latter is important: here i do not describe the rules of blogging, or the formal definition, for neither can exist, any more than there can be ‘rules for writing’ or ‘rules for art’ that hold any validity beyond familiarity.

It is not hyperbole for me to say that blogging has transcended writing into part of my way of being: somewhere akin to worldview or belief, most certainly creative process, and in part an obligation and burden.

‘Webs of Limitations’ – #WorkingOutLoud on Learning Science

Today i am sharing part of a new section of the Learning Science book that i’m working on with Sae Schatz and Geoff Stead: this piece considers the ‘webs of limitations’, or why it’s so hard to change a Learning Organisation. Over the last several weeks, we’ve discussed core Learning Science as well as various enabling technologies, which form part of the Learning Ecosystem.

There are quite a few compelling reasons for Organisations to embrace this work: for example, the growing demand for lifelong learning, and to keep workers up to date on changing technologies and contexts of work.

There are also positive relationships [1] between Organisational Learning (including training programs [2]) and Organisational performance.

A true Learning Organisation [3] is exciting. It has a Culture that evolves fluidly to cope with unexpected industry change, and both Learning and Development sit at its heart. The knowledge and capabilities exist for every organisation to be an amazing Learning Organisation, without unnecessarily burdensome investments or unbearable risks.

It’s not necessarily about ‘doing more’, or spending more money, but rather about changing what we do, and how we think about it, as well as stopping doing certain things.

Despite this clear truth, Learning and Development (L&D) professionals are often stymied by Organisational inertia.

Sometimes it feels like there’s a web of limitations that holds innovation back. But L&D can be the hero of this story: all you need are a few tools to cut through the  pesky web.

Web #1: Intra-Organisational Boundaries

Organisational change is challenging, and Organisational Culture may unintentionally draw us back into repeating, familiar patterns. So if, in the past, L&D was only seen as a deliverer of training courses, it can take significant work to rewrite expectations with your peers, to be seen as the goto team to shape new Learning Culture (beyond traditional courses). It’s almost as though we need to claim a new space, against the expectations of others.

But improving human performance requires more than this: it may require changes to equipment, infrastructure, processes, and incentives (as we explored in the post on a Learning Engineering approach). It may challenge legacy structures of power, ownership and control.

It’s not simply about new knowledge or skills.

The transformation required is systemic, whilst projects, authority, and budgets will tend to be domain based.

Or to put it another way: we are trying to solve a systemic challenge within vertically segmented structures. We are within the problem we are trying to solve.

Web #2: Old-School Procurement Systems

The ‘old’ tends to be ‘sticky’.

Our systems are intentionally designed for stability: essentially they are structured to be resistant to change, in that we do not wish our Organisational structure to ‘accidentally’ change.

We want that change to be both directed and purposeful.

This ‘bias for stability’ manifests as established processes, tribal expectations, and limitations on deviations from the norm.

In practice these constraints may begin exerting influence at the earliest inklings of Organisational Learning – at the pre-concept phase.

Firstly, consider most organisational procurement systems: These systems are designed to purchase tangible ‘things’ – ideally in neatly defined packages (e.g., a piece of software, set of licences, or number of days in a dedicated seminar).

Organisations typically buy what they know how to buy.

And commercial vendors typically sell what they know people will buy.

That means, in many cases, the ‘organisational development’ offered by a packaged program or learning activity is driven by what the market is offering, instead of what is actually needed by the Organisation, or where the Organisation’s most impactful opportunity lies.

Note that this is not an aberration of the system: this is the system.

But it begs the question of what capability you wish to hold internally, and what you will ‘buy in’ – what markets offer is not necessarily calibrated to success, beyond being a measure of what markets will buy. The system may be self fulfilling, but again, not out of malice or ignorance – rather through self reinforcement and lack of holistic measurement or calibration.

Secondly, procurement of Learning Technologies tends to be a long term venture, and hence people perform, deliver a project, and move on. Successful implementation within the established Organisational models of employment will tend to reward the associated leader and team with new opportunities, promotion, or a change of context.

A new leader subsequently replaces the original proponent which can lead to a loss of Organisational knowledge or context’ as a procurement program advances.

In this framework, there is little incentive to change your mind – and yet as technology changes, that is exactly what we should do.

Anyone who completes a learning technology implementation, who then turns around the next week and says ‘we should drop this and try something new’ would be brave indeed. And yet this is what the evidence may tell us we need to do.

Yesterday’s decisions do not automatically carry legacy value.

The stickiness of systems is partly held in their monolithic nature, largely a result of market forces and an outdated notion of what ‘risk’ may mean.

We often associate IT risk with stability, and data security, and desire uniformity, but risk may equally be being left behind or out competed. It’s not that we want unstable or insecure systems,  but a safe system in a failed Organisation is of little use.

Julian has written previously about the need for Diverse Ecosystems of Learning Technologies: core infrastructure systems, alongside a landscape of much lighter systems which may be divergent, small scale, local, even crowd sourced.

And crucially, are rapidly disposable.

These are test beds, and we need to build appropriate sandbox environments to test these new components, as well as appropriate crowd sourcing approaches to find them in the first place. An IT team may not be the only source of insight into new technologies, and an opportunity may lie in a parallel or divergent space, but which can be repurposed into an L&D context.

The challenge of learning technology is a classic one: benefits may be given by evolved technology, but it is often the underlying conception of need, and understanding of the Learning Science, which can drive a true ‘redefinition’.

Or to put it another way: technology may facilitate learning, but does not fundamentally change the cognitive foundations of learning.

Technologists can ‘build’ technology. Learning Scientists can build knowledge of how we learn. But it’s a dance that is complicated by cultural and commercial factors.

And then on top of that, people become politically and powerfully invested in the systems of procurement and ownership. So the systems we are trying to modify or redefine are not neutral, but are owned, and change may erode power and control.

So, much as technology can remain anchored in older models (as we discussed in our recent SAMR post), merely substituting and adapting new technology into old systems, so too can Organisational structures. That is, organisations may attempt change and innovation but find themselves stymied by old existing paradigms, power structures, and procurement methods. Hence we see new and emergent concepts and technologies adapted and constrained to do what we already know how to do, as opposed to upending the model and re-conceiving what’s possible, from a true learning perspective.

Web #3: Hard Maths (Difficulty Quantifying Outcomes Directly)

Organisations may struggle to understand and explain the relationship between L&D investments and meaningful organisational outcomes.

They may resort to counting easy – but relatively meaningless –  things, like how satisfied employees were with a training program (so called ‘smile or happy sheets’) or how many hours employees have spent learning. (Per LinkedIn’s annual survey for 2023, the top five metrics Organisations use to evaluate their L&D are still these sort of ‘vanity metrics’.)

These sorts of metrics provide minimal insights, and the outcomes collected are usually divorced from the things business leaders care about.

We should note that this failure is well intentioned but constrained: not only are these the things that have ‘always been’ measured, they are often the things that are easiest to measure – and they have a tempting appearance of ‘usefulness’ or ‘truth’.

Surely if people stay longer/watch more/click next, then they have learned?

When a real data-driven assessment is conducted, Organisations tend to use a ‘one-and-done’ approach. For instance, an organisation might measure the debut iteration of a new L&D program. (And then simply use smile-sheets for ‘data’ thereafter.)

Learning Engineering (and other general best practices) would tell us to measure each iteration over time, and to continuously use those data to improve the program long term (but per our earlier point, this often challenges resource allocation and budget paradigms). Similarly, another best practice is to use distal measures, for instance, examining performance impacts six months following an intervention.

The L&D programs who invest in psychometricians and data scientists typically produce much better analysis, but understanding what has been said and seen is itself a complex thing/specific skill.

We cannot assume that ‘data literacy’ is universal: clearly there are emergent skills within our evolving Organisations that we may need to train people in.

And we can over-simplify: for instance, we’ve seen complex sets for very large-scale Organisations reduced down to dial-indicators on a summary dashboard – stripping away any information about data reliability and validity or the assumptions made in the analysis. Similarly, we’ve seen the converse: Where (nerdy) data scientists explain each analysis used, along with a whole alphabet of Greek letters outlining the mathematical details to anyone who can read them.

There is a way to do this well!

There are viable ways to measure the impact of an L&D program or specific learning activities on relevant outcomes. (Sae insists we link to her favourite business book, How to Measure Anything, here.)

The problem isn’t that L&D is ephemeral or unmeasurable. Similarly, as Edward Tufte popularised, there are excellent ways to accurately portray complex information.

It’s just hard to do.

And so we often don’t see the explicit connections between L&D and Organisational outcomes monitored objectively and effectively, and as a result, Organisational leaders have difficulty placing a monetary value on these programs – making them easy to ignore (or cut completely).

This speaks to the emergent roles of L&D: the Institutional Storyteller may be a valid role. Or the Data Interpreter!

---

Notes

[1] This is a meta-analysis: The purpose of this meta-analysis study is to examine the correlations between the Dimensions of Learning Organisation Questionnaire (DLOQ) and frequently examined outcomes including organisational performance and employee attitudes.

Positive relationships were found between the DLOQ and organisational performance (e.g., financial, knowledge, and innovative performance) and employee attitudes (e.g., organisational commitment and job satisfaction) and the sub-dimensions (e.g., affective, continuance, and normative commitment), with a notable exception of a negative relationship between the DLOQ and turnover.

The constituent questions of the DLOQ scale make up seven dimensions that measure the positive impact and the cultural features of a supportive Learning Organisation. These dimensions are the following: i) continuous learning; i) inquiry and dialogue; iii) team learning; iv) embedded systems; v) empowerment; vi) system connection; and vi) strategic leadership.

[2] This is a meta-analysis: Specifically, a one standard deviation increase in training was associated with 0.25 standard deviation increase in Organisational performance.

[3] Ju et al. (2022) (that first meta-analysis above) has a research-based definition of a Learning Organisation: the learning Organisation can be defined as an Organisation in which (a) people continuously learn, (b) learning creates collective meaning and values, and (c) members’ behaviour reflects new knowledge and insights.

Setting Context

I’m working on the context of my work as part of the doctorate this week. I was reminded of my time as an archaeologist, where i was taught that everything is about context.

There is more to archaeology than simply digging things up: in fact, often the surrounding soil or sand is of more interest than the artefact itself, because as objects cannot tell their own story, we have to construct it from context.

This context may be relational – the ‘thing’ on top is more recent than the ‘thing’ underneath (on the assumption that ‘things’ are dropped ‘on top’ of each other, not ‘under’ the other) – but also based on similarity or difference (this ‘thing’ looks kinda like that ‘thing’, and hence we may assume they are related. Context may even be assumed by absence, where this ‘thing’ looks nothing like that ‘thing’ and hence we can assume their context is entirely different or parallel in nature.

The context of an archaeological object is hence fixed both in three dimensional space, and also within a broader and linear progression of time.

It is essentially a description in terms of ‘where’, with or next to ‘what’, and hence ‘when’.

Context is important in my work: the context of Social Leadership, the context of #WorkingOutLoud, the context of the Social Age.

In that context, of the Social Age, some things are being layered, becoming complicated, or simply buried. Some things are recognisable, but different, and some things are simply abstracted, no longer relevant or of value. Knowing which is important.

Context is everything.

#WorkingOutLoud on the Social Age

In parallel with my work on the doctorate this year i am building on the manuscript for a core book on the ‘Context of the Social Age’. It will be based around a dozen key shifts that we see (such as the ‘rebalancing of power’ or ‘radical connectivity’, with a broader exploration of what these shifts mean for our Organisations and society more generally).

Today i am sharing some writing that i’ve been doing in the context of the doctorate, but describing the context of the Social Age! This is early stage work and pretty rough and unpolished, but shared as part of #WorkingOutLoud.

My work explores the context of the Social Age, and by a strange twist is, itself, held within the context that it describes.

At risk of being recursive, to explore it is to create it, and to create it is to be subject to it, which will itself require description.

I typically introduce the Social Age as follows: you can look at the world around you and see that almost everything is a little different. The technology, our cultural paradigms, social norms, models of retail, consumption, production, and effectiveness, our organisations and institutions, our mechanisms of government, and our social structures themselves.

And you can look at that collective difference and come to one of two conclusions.

The first would be to say that these are essentially aberrations from a norm. That nothing fundamental has changed, we are simply seeing new expressions of old ideas, and that everything will be ok if we hold on tight, constrain the worst of the excess, and keep one eye shut.

The other conclusion would be to say that everything has changed, but the old order has not yet realised it. That we have entered a new epoch, and that legacy ways of knowing and being may be fully subverted by newer frameworks: for example, that we belong in different ways, to different types of structure, that we are wired up and connected differently, that power has shifted, that agency is unlocked, and that technology has fundamentally changed what it means to be human.

Or to put it even more simply: you could say that almost everything is nearly the same as it used to be, or that almost everything is a little bit different that it used to be, and then decide what that means.

One interpretation is that of progressive change, and the other is that of fracture.

The present, the time that we stand in, casts a shadow. Or more precisely, it casts two: one forwards, and one backwards. The one facing back is the long tail of the present, and includes historical precedent and social norms. The one facing forwards is the one that we will operate within, and represents emergent norms and expectations. And the two do not align.

It is an open question as to whether the legacy one has predictive value as to the shape and reality of the future one. Or whether it simply blinds us to new truths.

I subscribe to the notion of fracture, partly because i believe it to be true, and partly because my cost of being wrong is lower than the other view. A cost i am more willing to bear.

If we believe we still inhabit an Industrial, Post Industrial, or a Knowledge Age, then we are essentially going to be ok. Our radical connectivity and associated shifts in structure and power are simply new manifestations of an established order.

But if you believe in a Social Age, you would argue that technology has changed the sociology, has shifted us into a place where we need new models of organisation, structure, and engagement.

In a new type of world, we will not belong in the same way.

This is a challenging hypothesis, all the more so as i am very clear that my work does not carry an answer. If anything, it seeks to make the journey to a different hilltop, where our perspective may be sufficiently different to allow us to figure out our new ‘truth’.

There is a sense whereby my work is better understood in terms of belief than of science, in that it requires a certain leap of faith to find the value.

8 Emergent Roles in Learning

As our Organisations evolve, so too will the roles that we serve in. The shift towards a more Socially Dynamic Organisation, where structure itself changes, where learning shifts to be more contextual, personalised, co-created, in the flow, applied, will see some legacy roles become redundant, and some newer ones emerge. In a rather playful way today, i want to consider what some of the new Learning roles may be.

Social Storyteller – a role to connect the local to the global, and to interconnect between disparate communities. Not an ‘add on’ to an existing role, but an itinerant one: requires 60% listening, 20% writing, and 20% interpretation and provocation. Sits alongside, not within, existing Domain structures, so relatively unconstrained. Possibly has a pass key to join different spaces, meetings, or teams.

Learning Conductor – a specific facilitation role to integrate generative AI into dialogue based sense making and communities of practice. Specific skills in prompt scripting and storytelling. Does not play an instrument, but has focus on flow of conversations and progress – helping groups move into metacognitive state. Great communicator. Probably has a background in journalism or social psychology.

The Leaker – no, not the sort that shares secrets, but rather the person with a role of poking holes in things. The Socially Dynamic Organisation will be more permeable to expertise, and more loosely structured internally. So ideas, insight, stories, challenge, resources etc may all flow more easily internally, even to a degree externally or inter-organisationally. The role of the Leaker is to cause leaks. To work with leaders and teams to figure out exactly what they need to hold within a container, and what can be shared.

On-Call Data Visualiser – an itinerant role, available by the hour or day, this is a person who can help you understand data, and tell your story with it. Most likely from a popular science or scientific communication background. Not a marketer. Not attached to one team, but rather will help find your shared styles, vocabulary, and approach. In their downtime, works on an underground magazine that illustrates data fallacies and fictions.

Process Dis-Engineer – someone with a specific remit to spray red paint over the sticky points of processes. This person experiences the Organisation at a holistic level to find the sticky spots. To find the things that ‘have always been done’ and to ask ‘why’? Will suit the socially adept troublemaker.

Related to the Dis-Engineer is the Complexity Watcher: like a bird watcher in approach (lurks behind hedges, carries binoculars), their purpose is to find the needlessly complex and to unwind it. In particular they watch for complexity from the user perspective, and in the spaces between functions, where constraint and lethargy may lie.

The Metacognition. Like a magician, but instead of producing rabbits, they help you find different perspectives. Kind of like a coach, but available at Team level, and carries a toolkit of questions and prompts. Specific capability is to spot when jargon, context, or insight is becoming too homogenised, constrained, or certain. Probably plays a lot of video games, or is a mixed media artist. Weirdly could be a philosopher or learning scientist. Is probably constantly frustrated. As an added bonus, the skills of the Metacognition are learnable, so they probably only need a fixed term contract.

The Timesharer: has a role in helping people find time. It’s like that person who comes and declutters your house, but they do it for you at work in a non judgemental way. How? Because they will be expert in the small things that confound everyone: simple workflow and time budgeting tools and skills, using collaborative spaces and shared docs effectively, working out email strategies, and in particular helping you to disengage from spaces, communities or conversations that are draining your time. Not to mention your will to live. Uses the unsubscribe button like a pro.

I’m pretty sure there are more. Feel free to add below!

Generative AI and the Potential for (Anti) Social Learning

As ChatGPT and it’s comrades take the world by storm, amazing and delighting us with their natural language and peculiar quirks, it is no surprise that both hype, and hyperbole are ripe, alongside opportunism and marketing. But make no mistake, a deployable technology with virtually no barrier to usability, a technology that is veiled behind that most human of things – conversations – will change almost everything.

Some of that change will be contextual: the Story Engines will erode legacy contexts, especially where structural knowledge, or infrastructure, or in many cases long held skills, were required, and make them things of the everyday. For starters, the creation of art, of story, arguably the construction of expertise, and even the languages of music and film. These things are not simply moving from the exclusive to the commoditised, they are moving into the flow of our imagination.

Many effects will be secondary: the thing that they directly enable will seem minor, but the change they effect at scale will be dramatic.

And much of this will revolve around learning: today i want to consider the impacts of the varied Story and Art Engines on Learning itself, and specifically on Social and Collaborative learning.

My own work around Social Learning really sits at the intersection of formal and social systems, and between the owned and the distributed ones. It takes a view that some learning is fully formal (a story told to us by the Organisation), some is fully social (stories constructed and held in our tribal communities), and some may land in the middle space – essentially tribal in origin (so grounded, authentic, lived, experiential, owned), but accessible globally (if we earn the right).

The mechanisms of this type of learning are varied, but we can pick out core aspects of methodology and operation: in Organisational contexts we would look at Scaffolded Social Learning, where we create some structure and a path, but allow learners to make their own way down the track, using a series of spaces and gateways to provide both freedom to explore, and the structural safety and assurance that Organisations require. This type of learning creates diverse capability: not everyone doing one thing one way, but potentially everyone doing the same thing their own way, but with a broad range of ideas and approaches, which gives a greater versatilely in capability. I have taken to calling this a Generalised Capability, and it ties int directly to notions of the Socially Dynamic Organisation: one that can adapt, that carries a diversified strength.

But what about the Story Engines?

Well: Social Learning takes place through a range of traits and circumstances, approaches and features, such as curiosity, challenge, storytelling, story listening, iteration, prototyping, rehearsal, action learning, looping, fracturing certainty, consensus, trust, consequence, and failure. And pretty much all of these things may be enhanced, replicated, or re-contextualised by ChatGPT and it’s ilk.

Take the most simple of features of a learning system: curiosity. Within a Social Learning scaffolding, we would seek to create space, and community, for curiosity. To facilitate the behaviours, and dialogue, of it.

Social Learning is essentially dialogue based: and it’s no surprise that a new technology, which is also dialogue based, would impact so directly.

Until this point, the specific discipline of the Social Learning designer was to create the scaffolding, which is essentially an information architecture role, and differed from the Instructional Designer per se in that it was focussed more on creating spaces and structure than assets and infrastructure. But the Story Engines do not just answer questions: they prompt and scaffold curiosity itself. They can signpost and direct. They can help you to be more curious.

This is a fundamental shift from something which had become so endemic that we allowed the miraculous to seem utilitarian.

For all of human history, knowledge has been power, and we went to great lengths to capture it, codify it, and control it. I have shared before my story of standing in Morocco before an 800 year old armoured library door, with three keyholes, and to gain access to said library you would have to debate and discuss your right with the three scholars who held the keys. But search engines demolished the boundary: knowledge became distributed and hence democratised, but in the process, knowledge itself evolved. Becoming more dynamic, socially co-created, fluid, and dare we say worth less? In parallel work i have argued that the creation of ‘meaning’ is now the key ability for an individual or group. To create ‘meaning’ out of knowledge, and to operate within it.

So the first point of fracture was twofold: the distribution of knowledge, but also the evolving nature of that distributed knowledge itself. And now we come to a second point of fracture, because the Story Engines have a bolt on module. They can be Curiosity Engines, and essentially act as Capability Engines too.

You could correctly argue that the search engines, and the spaces of co-created knowledge, like Wikipedia, or YouTube, have enabled this type of approach before, and they have. But the difference is that the experience is now essentially a human one, a conversational one. Not an act of search.

Which raises the notion of (Anti) Social Learning: collaborative, but not with others. Curious in company, but alone.

The notion that we can embed the Story Engines into the learning experience, not as repository of information, and not as searchable data, but as learning companion, and directly as ‘sense making’ entity.

Engines as Sense Makers.

Sense Making, the creation of meaning, sits at the heart of Social Learning, and the heart of learning more broadly, by whichever mode or mechanism it occurs.

Purists will continue to argue that creativity, curiosity, intelligence, art, storytelling, even true insight, are uniquely human traits. Just yesterday i took part in a seminar where we considered the spiritual dimensions of knowledge. But there is always a point where philosophical endeavour bumps up against practical reality.

Or to put it another way, if it sounds like a duck, and looks like a duck, it is, essentially, a duck.

Part of our human nature is to believe in our uniqueness. And unique we are. But not alone. And the place to focus may be on story and art.

We are made of stories: they are really the only way we share what is ‘inside’ with the ‘other’. When i tell you what i feel, think, want, these are all stories. My shopping list is a story, as is a poem i write, or a legal contract.

And now we share these stories, and not just in words. The latest generations of AI will give us almost synchronous image creation, the ability to ‘create’ music, to speak different languages (visual and otherwise) and to generate films of our innermost ideas. As we plug these systems together, we will be able to imagine sculpture, to dance poems, to write colours, and to breathe knowledge.

Within the context of Social Learning, we can create spaces for dialogue, alone. We can create scaffolding, but also switch to sharing prompts. Developing perhaps a guide for the journey that is less about structure, more about topics, questions, ideas, spaces, even images.

It speaks to an evolution, again, of the role of the ‘designer’, if that term even persists. Perhaps more the Learning Conductor.

And of course it will speak to an evolution of assessment and measurement itself. The ability for analysis and meta analysis of the ‘answers’, the ‘stories’ created within the learning experience will itself lead to further insight and understanding. The creation of new meaning.

‘Story’ has been liberated from the human. And with it we have the potential for collaboration itself to be more meaningfully held between Story Engine and Storyteller. Or Story Listener.

When people talk about bias, about limitation, and about inaccuracy or quirks, that is to miss the point. What has arrived is a new paradigm of knowledge and learning. Everything else is a kink.

These systems do not advance iteratively, an inch at a time: growth is exponentially powerful. Or to put it another way: the technology is already leaving almost all aspects of ethics, power, control, and structure, in it’s wake.

Most likely this will impact directly into Organisations: those that manage to grasp this, and to creatively utilise the Story Engines not simply as enhanced search tools, but tools of dialogue, problem definition, prototyping, and for the creation of direct meaning, will leap ahead.

Tools that can analyse the quality of documentation, can provide real time improvements in communication and human to human collaboration, which can allow for rehearsal of language and story, and which can provide contextualised challenge for leaders.

If our very human imagination does not get in the way, the possibilities are endless.

We are liberated, we are freed, we are stories.

There will be a rush to product, but we would be wise to remain agile. Whilst the roots of this opportunity are technological, the potential is social.

The Contexts of my Work

This is early stage #WorkingOutLoud as i consider the ‘Contexts of my Work’. Throughout this year i am taking a pause, to look back at my work so far, exploring the context of the Social Age, and to look forwards, as i pivot into the next stage of this journey. This illustration is a first consideration of the ‘lenses’ through which i may carry out that examination of my core work: the ‘historical’ and ‘technological’ contexts, the ‘social’ and ‘cultural’ ones, the ‘intellectual’ landscape, and my ‘personal’ worldview, and finally the ‘moral’ dimensions of work and the Organisations within which it is framed.

There’s a lot here (which is partly why i am around three months into what i anticipate will be an 18 month journey to do this). Today i just want to use this space to sketch out a few thoughts around each of those areas:

Historical: there is a historical context to my work, as it explore some of the notions around how we construct the post industrial Organisation. This perspective can potentially track back to the medieval feudal system, where power and ownership became set and directly led to patterns of ownership and social structure that powered the Industrial Age. Similarly, there are geopolitical aspects as social collaborative technology challenges notions of Nation and national governance and reach, as well as questions of national culture and identity. And finally, economics becoming more multi dimensional, and potentially abstracted from legacy geopolitic structures.

Technological: there are clear themes to explore my work that relate to the evolution, proliferation, and democratisation of communication technology, and hence into distributed and tribal collaboration (and latterly Human/AI dynamics in this), then there is the shift in manufacturing to the ‘infrastructure free’ Organisation, and brand and Organisation. Encryption technology distributes trust, and hence tribal structures of power. Finally, transport technology erodes notions of Nation and identity, as well as facilitating strong social ties that are increasingly distributed – transport and communication technologies are probably paradigmatic to the concept of the Social Age.

Intellectual: the cumulative effects of the Social Age may be leading to a redistribution of intellectual cartography, with legacy structures of Universities succumbing to emergent Guilds or Salons, alongside probably technological impacts of AI driven narrative engines, and synchronous machine translation removing or crossing legacy cultural contexts of the intellectual sphere (e.g. breaking down barriers between geographical/cultural ways of knowing – with accompanying concerns about erosion of culture and appropriation no doubt). I write widely about the nature of knowledge, and it’s evolution, alongside it’s abstraction in favour of the creation of meaning. Mechanisms of validation are also shifting, distributing, away from formal, central, and codified structures into more socially co-creative ones. I could argue that #WorkingOutLoud shifts the veil of intellectual life away from a purely performance one, into a more continuous dialogic one? And the relationship between legacy models of publishing (where ownership of the press counted), and contemporary ones. The evolution and fragmentation of publishing feels important.

Cultural: in the technological context above i think we see unity being held in different cultural structures, again moving away from legacy formal and geographic ones, as well as the strong and dominant emergence of global cultural narratives and structures (e.g. Marvel and KPop), my work also considers Dominant Narratives as a structure of understanding and these are strongly related to cultural structures i think. Also: the roles of ritual and artefacts, in that technological context, probably form a useful lens to look through, and relate also to evolving relationships between ‘stuff’ and power (e.g. buildings, cars, books etc).

Social: my work documents the ‘rise’ of community as a core theme and aspect of the Social Age, so to look through this lens at how power and protest operate, and the structures of tribes as trust bonded entities. This is essentially a view of the re-wiring, or reweaving, of social structure.

Personal: i will need to consider my personal lenses, my worldview, and individual experience, as this forms a filter on my analysis and interpretation, as it does for us all.

Moral: this last one is challenging, but i think it relates to questions around ‘why we work’, and the nature of our ‘duty’ into systems, as well as broader questions of the fairness of our systems. Potentially this is more an ethical lens.

These are really just ‘notes to self’. My aim is to use these in some rapid analysis to see where i get to.as i said, this is very early stage #WorkingOutLoud.