Make critical judgements and evaluations, identifying general principles and evaluating competing perspective.

Assignment type: Report.

Report topic: Part A: “What do you think you’re looking at? Direct vs indirect observations – what are we actually studying?” Part B: “(How) has conducting this assignment helped to develop my understanding of Psychology in Question?”

Word count: 2000 words. No more than 1700 words should be used for part A. Any words exceeding the word limit will not be marked. The word count includes everything but the title page (including title, abstract and keywords), reference list and appendices. You should include a word count at the bottom of your report.

Weighting: 50% of module grade.

Submission date: Submit via the PYU612 Moodle page. Please refer to the Assignment Brief Component 2 for further information and Marking Criteria. Tutor support for the assignment will end five working days before your submission deadline. Any questions asked before this point will be addressed via email, however, no follow-up communications will be addressed.

Format: You are required to submit two documents:

 1. Your report and reflection (as a .doc file)

2.  Your content analysis code book (as an .xls file)

File names: PYU612_Report_[insert student number] and for the code book; PYU612_CA_ [insert student number]

Targeted learning outcomes:

01: Make critical judgements and evaluations, identifying general principles and evaluating competing perspective.

02: Evidence the ability to use research techniques to explore an issue or controversy within Psychology. 03: Show the ability to construct fair, coherent, convincing, and sustained arguments, using an appropriately wide range of evidence

How to use this document

This document is split into several sections, each giving details of different aspects of the assignment.

Section 01: ‘Question to address’. This section tells you the question you need to answer for this assignment. Use this question as the assignment title.

Section 02: ‘Content analysis’. This section explains how to do a content analysis. You are strongly encouraged to read Marsh and White (2006) alongside this document. You may also want to return to week 10 of PYU520 in which content analysis was introduced.

Note that if you do not manage to successfully complete a content analytic study for this assignment, then you will not pass!

The ‘content analysis’ section is split into three parts: 01: A general outline of the approach. 02: A detailed description of the stages of a content analytic study. 03: A very simple example of content analysis.

Section 03: Decisions. There are two sections about decisions. ‘Decisions made for you’ specifies things you must do in order to pass this assignment. ‘Decisions you must make’ details things you must decide if, how, and why to do while you are completing this assignment. Other issues may arise during the course of this assignment which are not covered in this section. If so, include these in your write up (and justify them with reference to relevant academic literature).

Section 04: Presenting your report. The final section gives some pointers about how to write-up your report. You should use this alongside guidance from your research methods classes

Questions to address

For this assignment, you are required to write a short report exploring an area of controversy in psychology, as introduced by this course. You will be required to conduct and report the findings of a content analytic study.

Part A. The first part of your assignment should address the following question. You should use this question as the title of your report:

“What do you think you’re looking at? Direct vs indirect observations – what are we actually studying?”

In order to answer this question, you will need to collect journal articles and look at what phenomenon they claim to be interested in, how they operationally define (RibesIñesta, 2003) this phenomenon, and how they proceed to measure this phenomenon. . This question draws heavily on Plato’s Allegory of the Cave, from the third lecture. You may wish to re-familiarise yourself with this content now. Or you can watch this short YouTube clip: https://youtu.be/SWlUKJIMge4

Part B. The second part of your assignment (submitted as part of the same document) should reflect upon the following issue:

“(How) has conducting this assignment helped to develop my understanding of Psychology in Question?”

Remember, this is not a reflection on how well you thought the assignment went, or your engagement with it, but how it has changed your understanding of psychology.

Sources of support Do not ask the graduate assistant for help with this assignment. Plenty of contact time has been set aside during taught sessions, and the graduate assistant has no idea what I want you to do (because I have not told him anything). If you have any questions, ask me during the lectures / seminars, or drop me an email, or ask your course rep to pass on your questions. Any questions about the technical aspects of how to conduct content analysis should be posted to the ‘PYU612 Student forum.

Content Analysis

What is content analysis?

At its simplest, content analysis is a count of how often something happens. This allows you to make a judgement about how common a given phenomenon is.

More finessed definitions of content analysis describe it as a robust and flexible method of analysing both quantitative and qualitative data (Marsh & White, 2006). It is a technique to objectively and systematically quantify the nature of any communication, medium or message (Kassarjian, 1977).

Typical content analysis questions

Contents analysis usually seeks to address questions such as:

How often does X occur?

Is X different to Y ?

When is X different to Y ?

Note about your assignment: In this assignment you are exploring how we gather knowledge about a given topic. Specifically, if this is done in a direct, or indirect manner. You need to develop a specific question i.e. select what topic you are exploring the knowledge on. You might choose to explore the frequency of direct v indirect observations across two different topics, or at two different time points, or across different journals. The important thing is that you provide a rationale for this!

Data in content analysis

Content analysis can draw upon virtually anything as ‘data’. You may apply content analysis to pictures, words, magazine articles, actions, behaviours, and other things. What is important is that whatever is to serve as ‘data’ can be clearly and unambiguously defined, and then counted.

A simple example

Figure 01. Gendered imagery
Look at figure 01. Code the two images for gender:

Our data here is an image (figure 01) which contains representations of two genders. If you were able to successfully identify the figure on the left as ‘female’ and on the right as ‘male’, then you can also do content analysis!

Note about your assignment: In this assignment the content you are exploring is 50 empirical studies, specifically the method they employ to access knowledge on your chosen topic. The specific content you are exploring is referred to as your “sampling unit”. This is for you to determine. It might be 50 articles from a certain journal, or a particular database. But again, you need to justify why you selected this.

Stages of content analysis

Typically, there are seven stages of content analysis. These are:

1. Choose general topic
2. Identify a sampling unit
3. Formulate testable research questions and hypotheses (if appropriate)
4. Collect your sample
5. Identify a coding unit
6. Generate / collect data
7. Code the data
8. Answer your testable research question(s).

For this assignment, many of the decisions necessary to make at each of these stages have been made for you. For example, the general topic has been specified (indirect v direct observations), and with that, the sampling units (method sections of experimental papers) and testable research questions implied (Does x or Y utilise more direct or indirect observations to inform our knowledge?). Additional concerns and decisions will still arise during the course of this project, and you must decide how best to resolve these concerns.

Stages of content analysis: Detail

1. Choose general topic. For this assignment, the topic has been specified for you. The topic is stated in part A, where the question is given.

2. Identify a sampling unit. The sampling unit is your data source. What do you want to count? Where are you going to find it? For this assignment, your sampling unit will be peer-reviewed research articles published in the discipline of Psychology.

In order to select your data, you will need to:

Generate suitable inclusion criterion. How will you decide what is an eligible source to count? How will you know which journal articles to include, and which to exclude? Why?

Suggested inclusion criterion for this assignment include:

ü  Published within the last seven years

ü  Is an experimental study (not a polemical, reply, review, overview, editorial etc.)

ü  Published in a specific sub-discipline of psychology (e.g. social, cognitive, developmental etc.)

NB: This list is not exhaustive. You may apply other criteria if you wish.

Note about your assignment: This information will need to go in your method section under “sample”. There will be two required sections for your method section; “sample” and “procedure”.

3. Formulate testable research questions and hypotheses (if appropriate). When performing a content analytic study, you do not need to specify a hypothesis unless there is a very good reason for doing so. For example, if your research question is interested in the difference between two features of your data, you may wish to perform a χ2 (Chi-square) test on the data. If your question is only interested in how common an occurrence is, you may be satisfied with simply reporting frequencies and / or proportions.

1. 4.      Collect your sample. Now that you have your inclusion criteria, you need to collect your sample. If you have fewer than 50 cases then you should use them all. If you have more than 50 cases, you will need to determine sampling criteria. Bjork, Roos, and Lauri (2009) estimate that 1,350,000 academic journal articles (of any discipline) were published in 2006 alone! If only 1% of these were published in the field of Psychology, that would still be 13,500 journal articles. As this is more than 50 cases, we must establish a sampling strategy to reduce this to a manageable number. Content analytic studies use the same sampling strategies as do any other research projects. These include (but are not limited to):

• Simple random sampling. The ‘gold-standard’ of sampling: Every member of the target population has an equal chance of being selected.
• Principled sampling. You would select all cases which meet a certain criterion – e.g. all journal articles which are cited more than 50 times. Don’t forget to justify this criterion.
• Systematic sampling. If we know the total population, then we can (e.g.) pick every fourth item, or everything published on a specific date, or everything published in a specific (range of) journals, or every fourth article published in a range of journals. Beware however: this sampling approach is very vulnerable to bias!
• Stratified sampling. Divide your population into groups relevant to your research question, and then pick your cases from within these groups.
• Opportunity sampling. Just take whatever you can get!

5.      Identify a coding unit. The coding unit is the thing which will give you your data. It is the equivalent to your participant, in an experimental study.

1. 6.      Generate / collect data. For this study, the sampling units are all research papers which have been published in the field of Psychology (as you could sample from any of these). Once you have applied your inclusion criteria to the sampling units, you will be left with your coding units. The coding units then might be the specific research papers written by specific authors, published in whatever sub-field of psychology you are interested in. Alternatively, they might be the method sections (participants subsections) of your sampled papers.

1. 7.      Code the data. You should now give each coding unit (individual published research article) a unique identifying number (code number). This is akin to assigning a participant number in an experimental study. (e.g. A1, A2, A3, A4 – A50)

Having given each coding unit a code number, you should now apply an appropriate coding frame to describe all cases. The coding frame should be a thorough and exhaustive description of each possible category you will include in your content analysis. (i.e. direct, indirect….)

The categories included in your coding frame must be comprehensive, exhaustive, and mutually exclusive. Your coding scheme should describe all possible events which could occur in your data, and there should be no overlap between coding categories.

Your coding frame should comprise of a list of categories and their descriptions. These descriptions should render each category unambiguous. The coding frame should also include a response code number which can be assigned to each category.

If, for example, you were interested in looking at whether or not the published research article uses a student population, as were Henry (2008); Sears (1986) then you might produce a coding frame which looks something like this:

Table 01:

Coding frame for publication sample populations

Category	Response Code
None	0
Student population	1
Non-student population	2
Mixed population	3

          Unclassifiable                 9                                 

Note however, that this coding frame example lacks a description of each category. You must include a description.

Avoid the use of an ‘other’ category in your coding frame where possible. An ‘other’ category implies that your coding frame is not exhaustive and is therefore incomplete. This would undermine confidence in your reliability and validity of your conclusions. You may, however, find it useful to use the ‘other’ category when there is such small an occurrence of a code or codes that it does not make sense to create categories for each of them. For example, we may find one paper which tests animals, rather than humans, and one which tests robotic algorithms. It would not make sense to have a category for such a small number of ‘animal participants’ and another for such a small number of ‘robotic participants’, nor would it make sense to combine both the animal and the algorithm into one ‘non-human participants’ category as that would imply that animals and algorithms are somehow the same.

You may find it important to include an ‘unclassifiable’ category in your coding frame.  Such a category is useful in content analysis studies where it is not possible to assign any category to the coding unit you are working with. For example, you may have a research study so badly written up that you cannot tell who or what the sample population was, and so cannot confidently assign it the value of ‘other’. In such circumstances, you may like to assign it the value of ‘unclassifiable’.

Enter the coded data into your favourite statistical software. The left-hand column should be for your coding units, while the right-hand column should be for your response codes.

If there is more than one researcher working on the data, you will need to check your inter-rater reliability. (That’s right, if you want to work with a friend, collecting 50 articles each and combining for some sort of comparison, you can!) You may use Cohen’s Kappa or, more simply, calculate the % agreement between coders. You should aim to achieve greater than 80% agreement (some argue 95%).

NB: The more category codes you use, the harder it will be to achieve inter-rater reliability.

 

1. 8.      Answer your testable research question(s). Having coded your data, count how many instances of your phenomenon of interest you have. Now relate this back to your research questions (or hypotheses, if you specified one). You may do this using just frequency counts or proportions, if you only specified a research question.  For example, if you asked how often something happened then simple frequency counts will answer this. If you specified a hypothesis, then you may like to consider a χ2 or a Fisher’s Exact test. You may also like to consider using some statistical tests if you wanted to know if one feature of your data is more common than another. This is not necessary however, and indeed may add unneeded complexity to your research.

As with any research project, you must relate these findings to your research question, and to the broader academic literature which surrounds this issue.

Stages of content analysis, applied to our simple example

Code the following images for gender:

 Figure 02. Gendered imagery.

Note that although this is one sampling unit, it contains two coding units.

Accordingly, each coding unit has been assigned a unique identifier – A1 and A2 respectively.

 

1. 1.      Choose general topic. The general topic should be specified according to your interests. Here, we are interested in representations of gender in door signs.
2. 2.      Identify a sampling unit. A suitable data source for addressing this issue is ‘typical bathroom door signs’. Inclusion criteria. For this example, images  should be freely available on Wikimedia Commons.
3. 3.      Formulate testable research questions and hypotheses (if appropriate). We will have a simple question: Will different genders be represented on typical door signs?
4. 4.      Collect sample, well in this sample we have one figure, let’s say I used opportunity sampling.
5. 5.      Identify a coding unit (see below)
6. 6.      Generate / collect data. The sampling unit for this small example is the above image (reproduced in figure 02). This image has two coding units – a figure on the left, and one on the right. These coding units have been assigned code numbers – A1 and A2.
7. 7.      Code the data. In order to code the data, we need a coding frame. The coding frame will prescribe a label for each category, describe each category so we know what criterion must be met in order to assign that label, and then will provide the numerical code to be used in our data spreadsheet.

 

Coding frame.

Table 02:

  Codes for each gender                                     

  Category       Description      Numerical code

Male    bipedal humanoid,                  0

narrow body shape

Female            bipedal humanoid,                  1

            fluted body shape                                    

NB: The descriptions given in the above coding sample are specific to this particular image. How would you code other types of figures for male and female genders?

 

1. 8.      Answer your testable research question(s). In order to answer your research question, you will have to (at the least) create a summary table of your data. As usual, you will first need to create a spreadsheet of your raw data. From this spreadsheet, you will be able to create the table of results for your data. Note that in this simple example, we have only looked at one sampling unit, containing two coding units, featuring one of each gender. As such, our table of results may seem a little sparse!

Spreadsheet of raw data for analysis

Figure 03. Spreadsheet of results, ready for analysis

Note that column A contains a row for each coding unit - the figure we have coded from the image. Column B contains the value we have assigned according to our coding frame. If we had further figures from other gendered imagery, this would also feature in the  spreadsheet.

Table of results. In accordance with current APA guidelines, when presenting your results you should avoid the unnecessary use of tables and figures. With that in mind, if you feel your results are best represented using a table, chart or graph, you may do so. The table below summarises the data from our simple example. From this table, we can see that our simple example depicted two figures, one of each gender.

 

Table 03

 Results of coding for gender

Male                1

Female                        1                        

Discussion. After presenting the results, and as is normal for a research report, you should discuss your findings, relate them back to the broader academic literature and to consider the relative strengths, weaknesses, implications and future directions of the work. For this simple example, we might simply conclude that two genders were present in the door sign. Thus, it is concluded that door signs do indeed represent different genders.

The codebook

When conducting a content analytic study, you will have to produce a codebook. The code book should include the coding frame, and a complete set of instructions for completing your research project. You should be able to give your codebook to any other individual, and have them perfectly recreate your study down to the final detail.

For this project, your codebook should also include your spreadsheet of raw data.

Failure to include your coding book will result in your submission being capped at a pass. So don’t forget

Decisions. . .

Decisions made for you

In order to complete this assignment successfully, you must satisfy certain criteria. If your project does not do at least the following, you will struggle to pass!

ü  You must address the question (of course)!

ü  You must use the assignment question as the title of your report

ü  You must follow the steps and procedures for completing a content analysis, as outlined above. Failure to do so may mean your report does not qualify as a content analytic study, and as such you will not be able to pass this assignment.

ü  You will write up your work as a research report, in accordance with guidance you have been given in your research methods classes (if applicable), and the notes given below.

ü  Your data for this report must be research articles – not polemics, replies or literature reviews.

ü  You must upload a copy of your codebook.

Decisions you must make

While the basics of this assignment are specified for you, there are some important decisions which you must make for yourself. These are outlined below. You must be able to justify each decision that you make with reference to wider academic literature. You should include this justification in your write-up. Failure to justify the decision you make will cost you marks!

 

• You might want to focus on a specific sub-discipline of Psychology (e.g. bio-psychology, cognitive, developmental, social, etc). Justify your choice.
• You might want to focus on a specific topic area in psychology, such as prejudice, decision-making, or attachment. Justify your choice.
• Decide which journals you will look at and why. Will you look at anything you can grab, or the top ranked journal by impact, or the BJSP, or some other selection / source? Justify your choice.
• Choose a type of research: Will you consider the use of participants in only quantitative research, only qualitative research, or in mixed-methods research? Justify your choice.
• Chose a sampling criterion. How will you include or exclude certain publications from your research question? Justify your choice.
• What statistics will you include in your report? While frequency counts are enough, you might also want to include proportions, and / or percentages. Are there any relevant statistical tests you want to perform (e.g. χ2 or a Fisher’s Exact test)? Remember, statistical testing is not always necessary in content analytic studies. If your data coding scheme is nominal or ordinal and your research question is only interested in how often something happens, you may simply use frequency counts for analysis. If you have ratio level data you should use means rather than frequencies.
• Are there any factors which may undermine the credibility of your findings? Consider any questions which may arise concerning the reliability of your data coding and/or the standardisation of your procedure as well as any questions concerning the relevance of your data source, and/or the adequacy of your sample.
• Are there any ‘deviant cases’ which do not match any other examples from your sample. What is different about these cases, and why (for example, are there any meta-analysis)? Considering this may help enhance your report.

Working as a group. You may collect and analyse data together with another student (or with a small group of students) if you so desire. Indeed, this might help strengthen the quality of your assignment, as you will be able to consider more journal articles if you are working in a group (50 per person). You will also be able to calculate your inter-rater reliability (the percent agreement), adding credibility to your findings.

However, you must work alone when writing up. If your writing too closely resembles another students, you will both fail the assignment.

If you decide to work in a small group, you must include a list of the student numbers of everyone whom you have worked with in the appendix. Failure to do so may result in everyone who worked together failing this assignment! You should not include the names of anyone whom you worked with, as this might compromise anonymous marking. If you worked in a group, you should also divide up your list of all coding units used according to who looked at what.

Presenting your work

Your write-up should flow as you have been taught in your research methods classes. If for some reason you do not have notes from a research methods class, you can also follow the guidance provided by the Owl at Purdue University (see Driscoll & Kasztalska, 2017a, 2017b). When writing your report, bear the following in mind:

ü  Write for an audience (who doesn’t know what you did)

ü  Maintain a clear line of argument

ü  Write with precision (choose your words carefully)

ü  Do not plagiarise (of course)!

ü  Include the following sections:

 

Title                 Use a descriptive title which lets the reader know exactly what you have done. You should use the ‘question to address’ from this document as your title.

Abstract          Your abstract should range between 150– 250 words. The abstract should be given its own page.

Keywords       You may also include up to six keywords beneath the abstract, which relate to the con- tent of the report. The keywords should take the form of a comma separated list. Keywords should appear in alphabetical order. Note also that strictly speaking, the keywords would be more accurately thought of as keyphrases, as they may include more than one word.

Introduction The introduction should introduce relevant psychological literature on this topic. It should serve to justifying what you have done for this assignment. The introduction should finish with a clear statement of your research question and hypotheses (if applicable). Try and keep the introduction short.

Method          The method should explain what you did and how you did it. A well written method sec- tion could be given to anyone (your cousin, the postman, anyone) and they would be able to follow its instructions to recreate your study perfectly. You may reference this document in the method if you wish to. Keep this section very short, since you’re mostly just doing what I tell you in this guidance document.

Results            Detailing what you found. This section should include the results of any statistical tests you conducted. You should never, ever cut-and- paste output from SPSS into your work. Always create any tables or graphs using appropriate, specialist software. Cutting and pasting output from SPSS will cost you marks. If the results section is more than half a page, I’d be surprised.

Discussion       What your results mean. You should relate your findings back to your research question and the broader academic literature which you discussed in your introduction. This should be your longest section.

References      These should only be the sources you have used in justifying your report – not the articles you have conducted your content analysis on.

Appendix        Additional information which supports your report should be included in your appendix. Information which is vital for understanding your report should not be placed in the appendix, but rather should be included in the main text of your short report. If you do not include information which is vital for understanding your short report in the main text of your short report, you will lose marks

Tips:

You might also like to consider the following points:

ü  Read the papers by Baumeister, Vohs, and Funder (2007); Sears (1986). You’ll notice that they spend very little time on their introduction, and almost no time on their actual findings. Most of their papers are devoted to detailing the implications of their work. Your assignment should follow a similar pattern.

ü  You probably don’t need to read anything more than the method sections of your chosen papers.

ü  If you have any difficulty classifying the response measure (is this a direct observation or not?) include a discussion of that in your write up, along with how you resolved the issue.