How to write a literature review without fluff and nonsense?

Some advice before diving in!

This article is born out of the bitter experience that me and my colleague Amjad Al Tobi had when writing literature reviews for academic papers and for our own PhD Theses. A good literature review is a considerable undertaking.

You should call your Literature Review (LR) by three words: ‘Critical Literature Review’ and not by two words and this is because what you call it makes a difference! It ingrains in your subconscious what a literature review really is. A Literature Review is also called “Related Work” and “Literature Search“.

All these terms highlights the  main aim of a literature review which is to critique, analyse, contrast and compare relevant works in the literature.

A lot of literature reviews that Masters students and PhD candidates normally write end up being laundry or shopping lists of research that they think is related to their work. I used to write literature reviews as laundry lists of relevant works so I was not an exception in any way. All students are showing to a marker, a reviewer or an examiner, is the fact that they know how to read, how to summarise and rephrase and how to cite sources. These are basic skills. Nothing impressive from the point of the view of a marker or a paper reviewer. One of the basic requirements for a good literature review is that it should engages critically and scholarly with ideas and works of other researchers.

Please finish reading this article, you will learn a lot! I promise you! Many colleagues, staffs and supervisors from the University of St Andrews and several others Scottish universities such as the University of Dundee, University of Edinburgh among others are very happy with the advice given in this article and they are suggesting it to their students. All comments are welcomed although I mediate all comments and do not publish comments that are just praising me or praising the article or the comments that do not add much. You can not imagine how much I hate praise. Please write a comment only if you have any tangible skill, technique or advice that is germane to writing high quality literature review chapters and sections so we can all help struggling fellow students.

Please check out also a recent article I am writing on PhD requirements, and Viva preparation titled “How to write a PhD thesis the way examiners want? & Viva Voce essential preparations“. By the way, “the critical engagement with the literature requirement  you read in the article, is an essential requirement checked by examiners for literature reviews in Masters dissertations and PhD theses.

The main aim of a literature review or related work section or chapter is to defend your work and to justify your research questions and build your research design. It is NOT meant to show us how much academic papers and books you can read or cite (a lot of students including my old “me” actually think that this is a good thing). It is not quantity that makes a literature review good, it is quality. The Literature review is considered the ‘litmus test’ or ‘acid test’ of the quality of a thesis [9,10].

As a side note: do you know why many students and researchers think that they are urged to put a lot of references of papers/books/articles they invested so much time reading and summarising. It is simple really and the urge is unconscious not conscious: it has to do with an extremely common and tacit logical fallacy that the absolute majority of people fell into easily which is called the  Sunken-Cost fallacy. Students usually say to themselves: Waw! Damn!!! we have spent all this time on reading those papers and books so that they do not end up in the literature review, we do not like that at all!!!! so let us try to shove them in despite they are either weakly relevant or do not add anything to the argument that the Literature Review is trying to get across.

In a literature review, you MUST show the examiners or reviewers that you have a good understanding of the relevant literature, that you are “not reinventing the wheel” and that you have summarised, analysed, categorised/classified, and interpreted the literature on the topic or research problem in question in order to draw needed insights and conclusions. Always remember that a Literature Review chapter is a chapter purposed to defend your choices, your research outcomes, and your research methodology/research method(s)!  The chapter/section is supposed to tell us also if your work is contributing to the knowledge in the field and whether this contribution is significant really and how much it is significant. Always keep in mind also the methodological aspect of the literature review as part of the whole process because and this is important in the methodology chapter you have to talk about the research methods used in other relevant work and say why you chose the research methods that you chose (whether you differ from them or not). My supervisor told me that he spent 45 mins discussing this in a Viva Voce, he examined. Examiners expect to see in your Literature Review chapter, a lot of tables and diagrams that summarise and classify studies under research themes, that contrast them and compare them to what you want to do or what you have done.

In the beginning of your research, the literature review, known as “provisional literature review“, plays a different role than the role played at the end. The initial role of a Literature Review is to help you find “research gaps” and pull out “worthy research questions“. At the end of the research, the LR helps you defend your research decisions. It is a completely different aim. This is why the literature review of the first year of your PhD or that of the first few months of your Masters is tweaked so much that it ends up looking completely different when you submit your PhD thesis or Masters dissertation and this is normal. This is why the literature review chapter is a chapter that you keep enhancing and updating until you submit your thesis. It is not a first year thing which you complete and forget! Although I should mention that few types of research degrees require from you that you finish and submit your literature review and then do the work you intend to do.

Masters and PhD students are not to be blamed for their lack of skills in writing good literature reviews. They were never taught the right skills to do a literature review in a systematic way. No real and descent Modus Operandi is taught to students.  What makes things even more miserable is the fact that supervisors/advisers,  actually think their students truly know how to write good literature reviews.

This article elucidates a systematic almost algorithmic method that help you write a good literature review. It helps you also to make judgements about what criteria governs the inclusion or exclusion of sources.

Due to the sheer size of the article, the “right referencing etiquette” using a particular referencing style (Harvard, APA, MLA, MHRA, AMA, Chicago, Vancouver, IEEE etc.) which is suitable or recommended for a particular research project, or field or institution, is not discussed as I believe there is already a plethora of information on the topic out there. I strongly advise you to be aware  as soon as possible of the referencing style you need to use and read a guide book on that. The reasons for reading a referencing style book before you embark on any research are numerous – examples: you need to know when you should compact to the “et al. form”, is it with 3 authors or 4 authors or more? How do you cite a work for which the year of publication can not be identified? How do you cite different works of the same author from the same year? How do you cite a DVD, a phone call with an expert in the field, a lecture, an anthology, an act of parliament etc… I could go on and on. Bare in mind each referencing style require specific rules that you should follow to the letter. Usually software such bibliographical management systems (explained later) and LaTeX/BibTeX/BibLaTeX (explained later) facilitate this task for you to a certain degree since you only need to specify what referencing style you are using. Nevertheless, be aware that these automatic tools always make mistakes and can generate in many cases wrong or incomplete citations.

I would also assume that you know very well the severity of committing academic misconduct and plagiarism so I will not discuss this in the article. You should always paraphrase and summarise the ideas and arguments of others and of course you can quote verbatim from them but always use the right formatting to designate that (either use single/double quotes or indentation) and cite the sources properly. It is still considered plagiarism if you have quoted verbatim a paragraph without using single or double quotes or any form of indentation to specify that it is a verbatim quotation to the reader even if you have cited the verbatim material adequately. So pay attention! Quotations when absolutely necessary should not be out of context of what the author is claiming and should be not be corrected by you if you find misspelling or grammatical errors. Use [sic] to specify to the reader that there are errors (please see my article on Academic writing tools and skills) for more information. I should mention here that you should use verbatim quotations only judiciously. Excessive use of verbatim quotations in academic writing is frown upon because it tells the reader that you do not have any voice and that your whole work is just a collection of extracts from other authors.

As much as possible avoid using secondary referencingSecondary referencing means referencing the original source while paraphrasing or summarising or quoting what a secondary source has talked about the first source without actually reading the first source and this is usually happens because researchers might not have access to the first source for a lot of legitimate reasons (source is out of print, inaccessible, source in foreign language etc…) or sometimes they are just sneaky 🙂 The danger here is that a secondary reference might misquote or misrepresent the first original source. Secondary references are considered as weak references in academic writing [18] and you only need to resort to using them in absolute cases of being unable to access the first sources.

Secondary references are not forbidden as long as they are cited correctly as such. Each referencing style (Harvard, APA, Chicago, IEEE, MLA, AMA…) obliges you to adopt a specific way  of citing secondary references. Per example, in the Harvard style, you would normally write something like (Cassell 2012, cited by Lanham-New 2014). You are saying to the reader: Hey! I have read Lanham-New 2014 and paraphrased what has been said about Cassel 2012 work which I did not read. Pay attention: referencing the first source while in actuality you were paraphrasing or summarising or quoting the secondary source instead of the first source, is considered a form of academic misconduct [18].

Have a look at two well recommended books on how to cite right: Cite them right & the complete guide to referencing and avoiding plagiarism.

Do not forget also to check the section in this article on bibliography applications and languages that aim to help you to automate the process of referencing.

Literature as ‘Data’

I want you now to consider the literature as ‘raw data’: actually literature review is gathering a lot of data and you need to analyse that data, criticise it and deduce links and patterns in it. Once you get this particular idea into your conscious and subconscious mind, suddenly, literature reviews are not papers/books and articles. Literature reviews become something else! They become a litle bit enjoyable! 🙂

It is the same when you analyse data and you draw graphs of your data to understand your data better. The literature review has the same purpose as data plots (like a scatter plot, a histogram, a box plot etc…). Actually the whole idea of drawing data plots is not about being fancy schmancy but these plots were created in the first place to allow us to make sense of what the data is really saying from different perspectives and in a compact way. Likewise, build the habit of visualising your Literature Review. Draw it!  Draw it!  Draw it!  A good Literature Review is an agglomeration of easy to understand plots of your topic.

A lot of examiners/ reviewers like seeing many summaries (of research tackling a problem), tables, drawings of timelines (for historical Literature reviews), mind maps, Venn diagrams. The sky is the limit for your imagination in this regard.

I have seen some well-cited papers where authors draw 3D diagrams (3 axes of space), where they represent each point in these diagrams as a paper (in the form ‘author et al.’) and where each axis is an angle of the research. The idea is quite amazing, don’t you think?

You should have something similar in your Literature Review chapter. I mean let your imagination flow. Literature Reviews don’t have to be boring.

The method

Part of the hypocrisy in academia is that no one can really give you good and practical advice at the same time. A lot of the time, you get the opposite, and on some rare occasions, you get good, but not practical, advice. You might read a lot of books on how to write a Literature review or how to write a PhD Thesis or Master Dissertation. But you are in a hurry and want to write a systematic review.

You are probably reading this article because you need an algorithmic, methodical, systematic way to get things done, as fast as possible and as efficient as possible.

Now granted, the task of scanning and reading the literature takes a lot of time no doubt. It depends on your reading speed and your apriori understanding of the topic (both are linked, by the way, since you normally read faster what you already understand). But what if I told you I have a technique you would thank me later for or maybe hate me for it. 🙂 In either case, please leave a comment!

Questions for a ‘Related Review’ to answer

You should write down in bullet points, or even better in a mind map, what are the questions that your literature review is trying to answer or tackle precisely.

Sit down with your Adviser/Supervisor and discuss all these points. They should mirror the research problem, research questions and research objectives. The following are a list of some important questions you need to consider before you start to do anything really:

  1. From the Literature, why is the issue or Problem X significant or important? (Why should the examiners or reviewers bother reading your LR?) What are the reasons for X importance? State the studies. (This also serves as good material not only for the Literature Review but also for the Introduction/Motivation chapter and Conclusion).
  2. From the Literature, what do we know about the causes of Issue/Problem X? What are the gaps in our understanding?
  3. From the Literature, what are the studies that have been done to address problem X if they exist? Why shouldn’t we be satisfied with current solutions or approaches?
  4. From the Literature, what are the motivations/contributions/advantages/disadvantages of these studies when addressing problem X?
  5. From the Literature, what are the research methods used to investigate X; are there any methodological flaws in these methods?

PS: I would assume you know already that there is a difference between research methodology and research methods?  You would be amazed by how many PhD candidates I know that had a heated argument with their external and internal examiners in their Viva Voce and are embarrassed because they don’t know that both terms are not the same (they are related but not the same), especially in Computer Science, since our discipline is kind of a hip non-conformist new science. If you are interested about Research Methods and Research Methodologies in Computer Science, I wrote this article which is still a work in progress (like all my articles 🙂 since I keep always adding new material and resources).

I have reconstructed a mind map template of questions your literature review needs to answer precisely. The asterisk * in the image are questions where you need to find as much as possible studies from the literature. The hash sign # is used to denote that you need to do some deducing, some judgement on your part, so that major parts of your literature review link, at it should be, to other sections and chapters of your Master Dissertation or PhD thesis.

Reconstructed and amended to suit any topic from D.R Rowland, the learning Hub, Student Services, The University of Queensland and  from Mary Carr – University of St Andrews ELT Workshops.

You can download the Draw.io , XML file (link) for the mind map above. This allows you to go to the site and edit the mind map to suit your precise topic.

A good technique here: is to take the mind map diagram after you do some amendments to it to suit your discipline and your specific topic  – in other words, fill out the template with your subject.

Afterwards, near each rectangle especially with the ones with an asterisk (*), put as many studies as humanly possible in the form of Smith et al.(ID), Bakri et al.(ID), etc… (with an assigned id), near all the rectangles. So make sure you have at least some peer-reviewed papers and books that tackles all the angles needed by each rectangle in the mind map. I would guarantee if you do that, you would write a good and strong literature review that is cogent.

When to stop?

When to stop? How much papers/books should you include? There is simply too much!  Be aware of information overload! The literature is like the ocean. Can you capture all the water in the ocean? Of course you can not! Saunders et al. [16] compared any literature on any topic to a giant river with the water always flowing  fast (meaning newer papers about your topic are always being published).

The best literature review you can do is what is similar to a good representing snapshot in time (photo) of a small part (i.e. the relevant part) of that river! Also that image is bound by whatever your camera can physically capture! You can not be omniscient, can you?

Cooper [1]  helps us answering the questions I asked in the beginning of this section, by advising us to ask ourselves what are we really tracking? Is it the previous research outcomes of the studies? Is it classifying the research methods used in every study? Is it classifying applications and practices?

We need to think how your literature review fits into Cooper taxonomy (Focus, Goal, Perspective, Coverage, Organisation and Audience).

You should investigate first the major conferences and journals, a characteristic mentioned by Cooper as Coverage particularly exhaustive with selective citations.

The idea  here is to have ONLY a representative sample of articles that cover all possible categories not necessary all articles that exist (otherwise you need a complete PhD dedicated only for the Literature Review). Your main goal is to prove to your Viva Voce examiners or Masters examiners that the sample of relevant articles chosen is representative and present a point of saturation for the specific sub-field or topic or idea.

In lay terms, it is meant here to have just enough representative papers/books to represent a certain category. In other words, if we return back to the mind map idea, include just enough representative relevant papers/books to represent a branch of the tree or a rectangle node. As an analogy, every district that has per example, a population of a hundred thousand citizens are represented by 1 or 2  or 3 Members of Parliament (MP). You literature review has to include only the MPs  of the different angles and the specific categories of your research topic. These angles of course always mirror your research questions and your research objectives.

Side Note for PhD folks (Sorry Masters!): In competitive scientific fields,  PhD candidates keep doing a literature review at different stages of their research (for academic papers, for project proposals, for the write-up period vel cetera). Do not be surprised! This is supposed to be the normal natural order of things in other words, if you are part of those sensitive scientific fields (Computer Science is among them!), you should keep an eye on the literature (i.e. on that giant flowing river!) all the time till the day of the submission of the thesis. Normally in such disciplines, first year is dedicated for a provisional literature review which aims to spot the gaps in the knowledge. Those gaps will be your justification (your tickets!) of why you are doing your work. Another reason to keep always an eye on the literature is: suppose you have published your research (which you should always aim for as soon as ready), there might be other scholars at a later stage that have cited you. On the one hand, they might have confirmed your results or findings.  This is called ‘Confirmation‘ and it is the mother of all jack pots in PhDs. It raises your work two levels high in the sky!: The first level is raised by you publishing in a reputable conference or journal in your field – you are telling the examiners: Hey! This work is original, is of academic quality, it had contributed to knowledge and was peer reviewed by at least 3 academics in the field.  The next level is the Confirmation achievement which say hey! others have confirmed what I have found.

On the other hand, whoever cited you might have criticised your work  instead of confirming it and that should also end up in the PhD thesis. Students update their literature review again in the write up period (usually last 6 to 8 months) either by re-doing it (not uncommon by the way – since research shifts normally to something useful or something that actually works and more importantly the aim of the literature review itself at the end is different (i.e. a defensive aim)  from the beginning (i.e. finding research gaps and formulating research questions aim). Literature reviews are works in progress till the submission date.  I strongly advice PhD candidates to use the technique presented in this article from the moment they start their PhD.

Methods to help you know what to exclude/include in a LR

These are called: the Inclusion/Exclusion decisions and criteria. In some disciplines, those criteria are elevated to the level of being extremely important in the sense that you have to defend why a particular study was or was not included (both ways).

First: Look at the best journals/conferences on your topic.  Your Adviser/Supervisor should help you in this regard (don’t be shy to ask simply because s/he is supposedly more experienced than you and knows which conferences/journals are worthy and which are not). If you are starting your PhD or Master, make these journals or conferences your daily or weekly digest and continue this behaviour through all the years of your study. You should subscribe to these conference proceedings or journals, like you do for your favourite car magazine or gaming console magazine. Many famous journals and conferences have email alerts and social networks’ alerts that inform you when a new issue is published. Use the technique explained in this article to store and document the papers you read.

IMPORTANT WARNING: There are many spurious, fake and low quality conferences and journals out there (open access and paid), so you MUST always check that you do not cite any paper published in such trashy places in your Masters dissertation or PhD thesis or academic paper. Thanks to good Samaritans such as Jeffrey Beall (an American librarian) who dedicated a website (scholarlyoa.net) just to  track spurious open access conferences and journals which only contain trashy papers.

Second: Search online databases and scholarly search engines such as Google Scholar, Semantic Scholar, Microsoft AcademicWorldWideScience, Science.gov,  and Refseek. There are some very fancy academic search engines such Wolfram Alpha that actually give scientific answers. You can also look for papers on search engines pertaining to specific publishers for specific fields. For example for Computer Science, the most common publishers and online libraries are: ACM Digital Library, IEEE Xplore, ScienceDirect, Springer, Elsevier/Elsevier Scopus and dblp CS Bibliography. For the humanities and social sciences, most common publishers and online libraries are Sage Publisher, John Wiley and SonsRoutledge , ProQuest (ABI/INFORM), EBSCO, Emerald Insight and Social Science Citation Index (SSCI) Web of Science platform in addition to many others shared with the scientific fields such as Elsevier and JSTOR. There is  also IngentaConnect which is a massive online database listing many papers from many publishers in natural, social and medical sciences. For medical fields, online databases such as MEDLINE/PubMed, Elsevier EMBASE and cochrane library are quite famous. In Psychology, for example, you would have PsycINFO database for American Psychological Association (APA), and PsycPORT.  For legal cases, you have Westlaw & LexisLibrary. Please check out what are the most famous electronic online databases that are pertinent to your field and what publishers your university library is subscribed to. These subscriptions are getting more and more expensive every year putting a huge burden on educational institutions. You can also use in your search, specialised library catalogs such as COPAC (for UK and Ireland), and the Online Computer Library. Despite the enormous amount of resources you can find, you will realise quickly that less than 10% of the articles covers really what you want at the end of the day [2].

  • Procedure of Search: First, identify search keywords and all their possible combinations, possible synonyms, and possible spelling (UK/US). These keywords should be relevant and must be taken from your research questions, or your research space. Pay attention!!! you might have many words that describe the same concept: An example in Computer Science: “multi-user virtual environment“, “multi-user virtual world“, “avatar-based scene” etc… all expressions mean the same thing, so in this case, you should use them all as search terms. This is not as easy as you think it is. Please spend some good amount of time (I advise few complete days) on finding the right and effective search keywords. It is totally worth it! Do not use phrases, search engines do not care about glue words, they care only about keywords. Base your search terms and search scope on written and very well-defined keywords and inclusion/exclusion criteria (please write them down literally in a report and discuss them with your supervisor). A example of an exclusion criterion would be something like “older than 2000” so that all papers older than the year 2000 are excluded from the review. Use disciplines’ dictionaries and thesauri to look out for the right terms!! Why? research communities are sometimes weird so a term might fell out of use in favor of another more “hippier” or usually more “hedging” term, so you will not find newer studies with the old term. Per instance, in the management field, little recent research studies use the terms “personal management” but the usage of the term “HRM” is more common, “employee evaluation” is now “employee appraisal” [11]. Examples of field dictionaries and thesauri are explained in the article discussing arsenal of academic writing. Keywords of relevant academic papers also contain additional related search terms or synonyms that you can also use. An amazing technique that is quite cool that I use sometimes which is to generate  word clouds from the most relevant academic papers and use the terms with the highest frequencies as additional search terms. I am falling in love recently with CAQDAS systems such as NVivo and MAXQDA and I am using them to do Literature Reviews (see the corresponding sections that cover the topic later). Second, Learn how to use Boolean search engines conjunctions (AND, OR, NOT etc.) and special characters to filter search. Per example: “Virtual AND World NOT OpenSim” will get you all papers that contain both words: “Virtual” and “World” but do not contain the word “OpenSim”. There are many similar and cool commands. Characters such as *  and ? are called wild card characters. The star a.k.a asterisk denotes one or more characters unknown. The question mark denotes zero or only one character is unknown. Per example, using: Innovat* will give you all papers containing words such “Innovation” , “Innovating” or “Innovate” etc.  Using recogni?e will give you all articles containing the word “recognise” (UK Spelling) or “recognize” (US Spelling). There are usually in all search engines and publishers search facilities, myriad advanced search features that you can use. Third, with each relevant paper you discover via the search process, you need to extract the keywords from such paper (usually found in the keywords section of the published paper) and therefore enhance further your search terms. This should be a repetitive process until you and your supervisor are satisfied completely by the sample of papers retrieved, by their quality and by their relevance.

Third: Seek the opinion of supervisors/advisers and other academics or researchers in the field concerning what to include and what to exclude from the literature review. Show the papers, books and articles that you have found relevant to your supervisor and seek his/her opinion. Although I know that supervisors sometimes are just weird to say the least, so don’t rely solely on them. It is good if you and other PhD candidates or Master students who are doing research in the same specific field/topic to share a common database or excel sheet containing metadata of relevant academic papers. I will discuss later a method that involves storing metadata about papers such as authors, publisher, relevance, summary of research outcomes, summary of research methods, limitations among others. I think it would be a great idea if a research group members could share a database or an excel document containing such metadata.

Fourth: Look at previous Theses or Distinction Level Masters Dissertations. Look also for survey papers in your field. ACM Computing Surveys (CSUR)  and IEEE Communications Surveys & Tutorials are great examples of places where you can find many surveys for computer science fields. If there is no new survey published on the topic, Well! Bob’s your uncle!! That means the door is open to you to publish a survey of your specific topic which will strengthen your thesis or dissertation.

Fifth: Use brainstorming techniques and visualisations such as mind maps and relevance trees to come out with search terms  and to bound the literature review. Share and discuss those with your supervisor/advisor and other academics. Relevance tree diagrams are extremely important in a literature review. As their name suggest, they show the relevant ideas out of your research questions in a hierarchical tree-like structure. So you start with a node at the top (called root note) which is usually one research question or one research objective and then you would divide it into branches and nodes containing keywords or phrases that come out of the root node and then you continue with the same process going down the tree until you reach absolute atomic relevant ideas. Relevance trees diagrams are trophies of search terms. Software that allow you to create mind maps usually allow you to draw relevance trees.

Tables and Diagrams in your Literature Review

Tables and Diagrams are super important. In this section, I present some comparison tables, Summary Tables, Timelines and nice diagrams that might help you ignite those ideas of diagrams and tables you might use in your own literature review. I took many of them from survey papers and well written PhD Theses. I will keep adding material to this section and enhance it. Please if you have new ideas of diagrams people have used in their Literature Reviews, please put them in the comments.

Summary Tables

An Example of Summary Table taken from [3]. I like the fact that the authors used Correct and dash signs to delineate the existence or absence of a feature.

Another example of a summary table taken from [5].

Summary Table taken from [6]

Summary Table taken from [6]. In this example, the author presents the Pros and Cons & gives few examples as studies.
Summary tables are very important to situate your work in the Literature. They give the reader an overview of major concepts that you are trying to shed the light on.

Timeline Diagrams

Timeline Diagram taken from [4].

Graphs

As much as it sounds crazy for the humanities postgraduates reading this article, it is quite a beautiful idea.

Graph-Like Diagrams in Literature Reviews
Diagram showing Class Accuracy of different datasets from the Literature (Deep Learning field) taken from [7].
Luxton-Reilly et al. [14] created many histogram graphs that show statistics pertaining to the publications they have included in their literature review. Have a look at their paper for ideas. These are amazing graphs that you can make yourself and they show the examiners and reviewers that you have engaged with the literature not merely writing a borring narrative. The more diagrams/tables you create that summarizes, classifies, and contrasts studies in the literature, the higher the quality your literature review will be!

Notes on Citations

Note 1 (more relevant to PhDs but also applies to Masters): it is assumed by your examiners (internal and external) that you would have already read every citation you have used in your thesis expect maybe very long books. So make sure you do that; especially before submission and before the Viva Voce. Do not cite something you did not read. This is considered a form of academic misconduct. If you have used resources that are relevant with relevancy= 4 or 5 according to the technique presented in this article, you MUST have done that already.  External examiners are experts in the field so you do not want to be in a situation where the external mentions some content in a paper which you are clueless of it.

Note 2 (Both Masters and PhDs): Try to use recent published peer-reviewed citations as much as possible. Examiners look for sure at the your references list. This proves to them that you are au courant of what is happening now in the field. A good advice given by Keshav [17] is to consider the highly cited articles from your search. These are normally pertaining to the key researchers in the field.  Although I will inject myself here and warn that sometimes a highly cited article might not be for the good reasons you are hoping for. Researchers might be citing it for the bad reasons (wrong theory, some ridiculous claims etc…). Keshav advises (and I strongly concur with him) to go to these top researchers websites and look for the conferences and the journals that they are publishing in. These would be the top conferences in your field of research. Top researchers publish always in top conferences and journals.

Note 3 (Both Masters and PhDs): DO NOT use Wikipedia articles as references for anything really, Wikipedia is not reliable at all for any academic setting (since anyone can edit the article); but a good trick is to check out Wikipedia references if they contain good relevant journal papers or books. Wikipedia articles are normally a good starting point to understand a general topic (nothing more than that). Same applies to other Wiki systems EXCEPT maybe the wikis or official online documentation of a software that you are using in your research since the wiki is written usually by the creators of the system (ex: CTAN web Wiki, Tensor Flow Python Package Wiki, OpenSim Wiki etc…).

Note 4 (For both Masters and PhDs): Avoid as much as possible using websites as references in your PhD Thesis or Master Dissertation.  Use always published  and/or peer reviewed material as references unless there are no published material on a very new topic and you are obliged to use a web site. The reasons for not using web resources are many: they are unreliable (anyone can write things on the web) and they are very ephemeral or volatile: i.e. they always disappear. I have read many good theses where the authors put websites in footnotes and reserve published or/and peer-reviewed materials to the bibliography or references list. Remember that websites tend to become unavailable as I said due to the nature of the web. The excellent theses and dissertations that I have read have at least 95% of their references lists as published works (Books, papers). Use also a URL Shortener like Google URL Shortener or Bitly for your web site URLs, this is a good trick to minimise the overall number of words in the thesis or dissertation, and more importantly some URLs are too long and too ugly to stay like that in your thesis (either in footnotes or in your references). Do not forget to create a note on web resources where you mention all this in the front matter of your Master Dissertation or PhD Thesis. Also do not forget to mention the date you have accessed the web resource.

Note 5 (For PhDs):  Some people try to cite the internal or external examiners’ papers in the thesis. Now this is fine if and only if the papers are really really really relevant. As an advice, do NOT DO that if they are not relevant or mildly relevant so that you and your supervisor/advisor can make a pathetic statement (some supervisors/advisors even ask candidates to do that). I saw a thesis where the author references an external examiner paper which was not relevant (since the external was one of the authority figures in the field coming from the US to UK for the Viva Voce). What the hell? In my opinion, this is extremely pathetic behaviour. But hey! this one is quite common in academia. What kind of deranged low level academics want to see his/her name in the references lists of PhD that they will examine? You have to have a big bloated ego or serious vitamin D deficiency or a pathetic unfulfilling life to want that if you are an examiner!

Stage 1: Literature as a Data Store / Database

Now we move to the beefy part. How can we make the ‘Relevant Literature’ as a Data Store or Database ready for mining useful stuff in a fast way? What is our magickal method really?

Copyright Image: Harry Potter.  🙂

Method

  • Step 1: Scan the Literature for the specific topic you need and begin to download papers that have some relevance.

Now wait! don’t begin to download hundreds of papers. Remember you are writing a related literature not writing ALL the literature — In this step you can use the quick [title + abstract] reading and eliminating strategy to figure out what is relevant and what is not. Only read Titles and Abstracts: nothing more, nothing less!

Relevant papers’ references will definitely lead you to more relevant papers. The Literature Review sections of your relevant papers points you normally in the direction of other interesting papers to read. Looking at the Literature Review section of a relevant paper is a technique I call  “Checking the Past of the Paper” since the current relevant paper is reviewing and criticising previous Literature which is relevant to it from the past (Get it!). The other way is even more cooler. I call the technique  “Checking the Future of the Paper“. If your current relevant paper is published let us say in 2002. Surely other published work have cited it after this date, otherwise it is probably a terrible paper :-). A big probability of this work is extremely relevant to your topic. So go to Google Scholar, copy the title of the relevant paper and check the “Cited By” field where you can get all papers that cited it. A lot of future papers might be either praising or criticising your relevant paper which constitute an essential material to cite and discuss in your literature review. The cited by technique does not work well if the paper is new.

  • Step 2: Put all papers that are somehow relevant to the topic into a folder. Try to copy from the online databases (Google Scholar, ACM Digital Library etc…), the title of the paper and make this title, the name of the file with a number at the start (a sort of an ID). You can download a BibTeX or EndNote file for each paper (Google Scholar allows you to do that if you wish). PS: when I said ‘somehow relevant’ [I meant they could be mildly relevant to super relevant (1 to 5)].

Now wait a minute! What could be mildly relevant? Well, suppose you find a paper that is not too specific to your topic but is in the parent field or topic. From these papers, you can get a lot of quotations that situate your topic in its context. Some types of Literature Review move from General to particular (a cone fashion). Papers that have the relevant field = 1, 2 or 3 are ideal for the general part of a Literature review.

  • Step 3: Create a Master Excel File (the columns of the excel sheet you will be creating are explained later) OR create a LaTeX Master File (fields of the LaTeX you will be creating are explained later) which will be filled out with information and metadata about papers/books. I would refer to these files as master files. You will also may need another Master MS Word Document accompanying the Master Excel File or Master LaTeX Document and this serves as a container for all images and tables you want to copy from papers. Make sure in this case to assign an ID for each paper and to reference the images and tables with the ID of the paper, so you can cite them correctly in your Literature Review.

Now, if you are writing your PhD thesis, Master Dissertation or an academic paper in LaTeX. It is better to go with the LaTeX Master file idea (no harm of using Excel of course). If you are an MS Word person, go with the Excel Master File idea.

If you decide to go with the LaTeX idea, make sure you create a BibTeX (.bib) file which contains the references of papers you are investigating (if these papers/books are relevant you will need those references anyway!).

The Purpose of the Master Files

These master files will contain a lot of fields that you gather while you read the papers many passes. The purpose is that you need to get rid of papers/books that you have already read during your research journey (You should make sure with this technique that you do not need them anymore). You will thus have all the information needed to construct your literature review [See Stage 2] in those Master Files. This is specially helpful for PhD students, as they start their literature review in the first year. By the last year (3rd or 4th year) and with their topic normally shifting, they would have these master files to save the day!

Make sure when you write information in the fields inside your master files, that these latter files are parse-able, search-able and filter-able. In other words, use same acronyms or words in all your documents. Be consistent!

In this step, if you are in much hurry,  you need to read most importantly in any paper:  Title + Abstract + Introduction + Conclusion => Termed as the ‘Essential Four’. Some argue the order in which to read  (1- Title, 2- Abstract, 3- Conclusion, 4-Introduction).

Any paper that you don’t get good material from reading the ‘Essential Four’ is not worth mentioning, Do you know why? it is written in a terrible way! So be assured then, it is not worth to be in your thesis! and let us move on.

Of course, If you have the luxury of time, you can read other sections in each paper like results, literature reviews etc.. Only papers of relevance of 4 or 5 needs to be read completely even when you are in a hurry to do a “quick and dirty” Literature Review.

Fields in Master Files

Make sure you are consistent throughout the Master document with what you say. In excel, per instance, you want to be able to filter or search easily. Example, filter all papers who have the relevant field = 5.

  1. Field 1 – [ID]: This is an identifier. Start with 1 if you want. Now you can include the year of when you are reading the paper like 20180001 per example (this is helpful for PhD students – knowing when they found this paper). Even better you can include the code of the Paper that Google Scholar gives to a particular paper in BibTeX format (example: bakri2015http).
  2. Field 2 – [Keywords] : Copy all the keywords of the paper to this field. This is a useful field to have because you can use the power of Excel to search or filter based on keywords.
  3. Field 3 – [Authors]: Write this field in the form of  ‘Smith et al.’, if there are many authors. This makes life a lot easier when copying the field information over to your actual literature review document.
  4. Field 4 – [Relevancy]: The values can be textual as ‘Relevant’, ‘Mildly Relevant’, ‘Super-Relevant’ etc…  or I prefer better to use numeric values from 1 to 5 with 1 being very mildly relevant and 5 being the most relevant (very close). Relevancy is gauged based on the degree of closeness of the paper to your research questions and objectives. If a study contradict or support your research questions and objectives, it is relevant (both cases). The rule of thumb here: always ask yourself: Why this paper is relevant to my research questions/ problem space? How this paper is relevant to my research questions/ problem space?
  5. Field 5 – [Reading Status]: You can say per example ‘Read’ or 1 pass’ or ‘A/I/C’ (meaning Abstract, Introduction, Conclusion)
  6. Field 6 – [Publisher/Conference]: The publisher  or conference. Publisher reputation is important. You should care also about the value and authenticity of the sources you will be using in your literature review.
  7. Field 7 – [Year]: Year of publication
  8. Field 8 – [# Citations]: Number of citations (make sure you use seminal papers in your literature review – i.e. peer-reviewed papers of value – markers/examiners like that).
  9. Field 9 – [Theme]: What is the theme of the paper?
  10. Field 10 – [Taxonomy]: This is an optional field. It is useful if you decide to create or propose taxonomies in your literature review.
  11. Field 11 – [Sub-Taxonomy]: This is an optional field. It is useful if you decide to create or propose sub-taxonomies under major taxonomies in your literature review.
  12. Field 12 – [Research Outcomes]: is an extremely important field. Summarise in two to three paragraphs or even more, all research outcomes of the paper you have just read. The bigger this part the better but not too big because it will slow you down if you write excessively in this field for every paper (imagine!).
  13. Field 13 – [Research Methods/Methodology]: is another super important field. What did the author(s) use as a research methodology and as research methods to arrive at the outcome: Observations, user studies, questionnaires, measurement experiment, development of a software etc…  What is their research methodology? This field is called a ‘Methodological Review’.  When you read be critical please! This field is mandatory. It will also be of use when you write your methodology chapter. Bare in mind, you MUST defend the methodology/method(s) used in your Master dissertation or your PhD thesis by comparing and contrasting them with what were used in the relevant studies in the literature. So include in this field the research method(s) of the paper and give your critique of them if applicable to field 15 (“the critique field”) which will be mentioned later.
  14. Field 14 – [Limitations]: A very important field. What are the limitations of the work in the paper from the authors perspective (not yours!!!). They usually mention this in the paper. Sometimes this is hidden under future work section.
  15. Field 15 – [Similarities and Differences from current work/Gaps-Critique field]: This field is super super super important and obligatory to be filled out. In it you mention how the work is similar to what you are researching or doing and what is different. You criticise the work in the meanest ways possible!!! This field MUST be a critical field, your own opinion.  An interesting exposition on the Critical Field 15 is explained in section How to be critical in Field 15? A lot of us don’t know how to be critical or are afraid to be critical of “published work” or think it is weird!. Well! that is wrong! this field defines and explains clearly the difference between a real Literature Review and a Laundry List. Writing in this field, is like you are grading or marking the paper and you are a   unpleasant high school teacher.  This field is very important also especially when you are trying to find a gap in the literature for your topic (at the beginning of a PhD) or even at the end of the PhD when you are comparing what you have done with what other researchers have done.
  16. Field 16 – [Quotes/Explanations/Expressions]: In this field you can quote from the paper. You can mention something that the authors have said in some section. Trust me! this field is very important as well. Now how is Field 16 different from Field 12? Well, Field 12 contains the actual research outcomes. Field 16 might contains quotes from per example the Literature review of the authors (might not be related at all to any actual research outcome). PS: Make sure you put quotes ‘ ‘ or ” ” –  so that you can differentiate between your own language and what is taken literally from the paper/book.
  17. Field 17 – [Has Useful Images-Tables:] In this field you either specify Yes or No, this tells you quickly later to lookup the MS Word Document dedicated for Images / tables taken from papers.

Papers of categories of relevancy 4 and 5 needs to be read completely. Actually it depends on your time. If you have plenty of time, even papers of relevancy 3 normally contains a trophy of useful material and quotes you might use which you might find in places other than the major four.

The excel sheet should and will be a massive file so back it up often. As  said previously, It should be easily searchable  and can be easily filtered by year/relevancy etc… Each paper (relevancy 4 and 5 maybe also 3) should at least have the most important fields filled out by two to three summary paragraphs in your own language (mainly fields 12 to 17).

You can add an optional additional field concerning extracts of language expressions.  I call it usually “language store“. This is a good exercise and is more relevant for newbie international students who did not yet mastered the  “academic language“. The idea here is to extract the “language expressions or academic vocabulary“, especially if they are quite interesting and put them in this additional field if you want. The authors do not own the language itself or the English words used so if you saw an expression in an academic paper that states per example “there is copious literature on X“, take that expression (not the idea just the language expression) and put it in this field. This is actually how I developed my academic language by filling my head with a massive vocabulary of academic English phrases extracted from reading a lot of papers. Later you might want to say that the literature has a large number of studies on a topic X so you might use such an expression. You would never imagine how much these language expressions come in handy when constructing the LR!! It is  similar to the Manchester Academic Phrasebank idea but you are the one who is gathering the academic expressions while reading for your LR.

How to be critical in Field 15?

Recall, Field 14 contains the limitations that the authors mentioned about their own work which is something I extremely respect in authors, although sometimes they hide limitations under Future Work or Further improvements, Future Measurements /Future Directions sections or any other title like that.

Field 15 is the King of all the fields in my technique (your harsh criticism). You know I mentioned before the fact that if you can NOT write field 15, for whatever reason,  you still do not know how to write a Literature Review. A literature review MUST be critical otherwise it is a laundry list or at best it is an annotated bibliography.

Well! How can I be critical? How can I be critical of something that was peer-reviewed?  You must be saying that to yourself, give me something tangible! In academia, you are expected to be critical and to question everything and not necessary agree with the papers or books that you read.

This following list is important to consider as a crucial step in learning how to be critical when you read a paper or a book – if you find difficulties using the questions in the list, please leave a comment, it would be interesting to learn from your experience.

Now put the following questions in front of you after you read very well the “victim paper/Book” in question (probably of relevancy 5 or 4). Begin to ask yourself the questions in the list and think if you can criticise in a truthful transparent academic manner the academic paper in front of you. Bring your knowledge of research methods and statistics to the equation. On a side note, I strongly recommend you to read at least one research methods book in your particular research field, if you have not done that yet by this stage. Think about the external and internal validity of the work. I know that crtiticising is easily said that done! But this is how academia is supposed to function.  Do not be impressed by the outer quality of the paper or the beauty of the language or the ranking of journal or conference, all this will clutter your judgement. “All academic work have limitations WITHOUT EXCEPTIONS” – you would be surprised by how much limitations, weaknesses or serious flaws you can find in peer reviewed articles published in high ranking prestigious journals.

Remember! you want to find objectively as much as you can, flaws in the work. Please you need to do that in a truthful way – do not invent limitations that do not exist and please try always to use a hedging academic language (i.e a cautious language) while describing flaws and limitations. Your criticism should be objective, harsh and in the same time constructive.

This exercise of criticism that you are doing here is very helpful to develop your critical thinking skills and is a very essential skill to have in the academic life. In the case of peer-reviewed sources, the material in these sources was subject to a peer review process where the reviewers  maybe have asked some of the same questions that you are asking here (Are there any methodological flaws?, does the results support the conclusions?, is the analysis cogent? etc…). You will do the exact criticism if you were asked to participate in the peer-review process.  NB: a paper who has passed through a peer-review process can still be critiqued heavily for a zillion reason. Don’t have any wrong ideas of perfection because of peer-review!!!

The following lists are what I call as  “Hammer” critical questions. They give you a systematic or somehow algorithmic way to try to find flaws and weaknesses in a paper or a book. These questions (sort of magical mantras if you want) will help you ignite that critique component of your brain. The lists are not extensive (I will keep adding to them) but are nevertheless useful especially for papers of relevancy 5 and 4 ).

NB: X, Y, Z etc… are aspects/angles that depend on your specific topic of research) – also for more questions of this kind, there is a big literature covering peer-review of journal/conference papers, of grants, and of books. In addition, you can find a big literature covering dissertations and PhD thesis assessment and how to critique research. Please have a look at these for more advice on traditional critique questions that every researcher should ask and be aware of. You will have tons of these! Please also help out with your comments if you have in mind anything that is important to add:

Set of Critical Hammer Questions 1

  1. Have the current study dealt with the important concept X?
  2. Have the current study treated X in much detail? (look at the beauty of the question here, if the first question did not catch the victim paper, the second maybe will) – Have the exposition of the problem Y been satisfactory?
  3. Is the current study inconsistent with the current understanding of the topic in the literature? (maybe the paper is a justifiable diversion from the literature, maybe it is not). Do the hypotheses of the authors follow logically from what was done before (eg: literature reviews, acceptable understanding in the field…)?
  4. Does what the authors presented seem to be a good way of exploring the issues? a practical way of exploring the issues?
  5. Have the current study been lopsided in any way (meaning focused only on X instead of Y)? especially when X and Y are general but important categories. Per example, the current study created a virtual museum using declarative 3D Web technologies such as X3D/XML3D, not by using more common imperative languages such as WebGL.
  6. Did the study take into account the most important criteria of Q, K and L? Do the study include certain essential variables?
  7. Did it solve a contradiction between N and M?
  8. Is the work limited in some way?
  9. Generalisability or what is known as the problem of generalisability (Oh Gosh how much this is painful!!), are the results or findings of the paper really generalisable? Try to work on that!! (this is linked to the statistical hammer questions  (sample size especially when it is small, sample composition and criteria of selection, population etc…) – explained later.
  10. How old is this research? How old is the technology used? Is the technology deprecated? is it more than 10 years? (this is the easiest limitation). Per example, Computer Science is an extremely fast field. I chose an arbitrary number ( 10 years) . Yes! If a paper did the exact idea that you did but the authors used an old technology or procedure. That is a legitimate critique. Per example, the paper developed system used old Java Applets (which no one uses anymore – a dead technology). Bob’s is your uncle.
  11. This question is similar to above but you ask: Is this study future proof? Will the result be still valid in the next 5 years, 10 years? etc… This type of critique is very trivial and weak and is more useful in fields such as computer science, engineering etc.. since  research tries to solve a problem ( a new network protocol) that future improvements in technology (increase in Internet Bandwidth or processing powers) will eventually eradicate the problem.
  12. Did the authors use a technology or language that is not widespread or not democratic or not used by experts? The critique here plays on the thread of the impact of the work. Your project might have used something very widespread and friendly to use whether the authors have used something archaic from the age of the Dinosaurs. Pay attention to the nuances between this question and the questions before.
  13. Have the results been based on old data/datasets (Owch!!!) like over 20 years ago or flawed datasets? I mean it depends of course on what is ‘old’ in your discipline means. In CS everything is old 🙂
  14. Does the results really support the conclusions/findings?
  15. Is the experimental data/methodology controversial? Do you know anything about that from the Literature? Is there any general agreement concerning this? Are there any methodological flaws?
  16.  X and Y  (other papers per example) suggest that the approach used by the authors in the current work is inconsequential or useless  or whatever bad adjective. Is that really the case? You need proof from the literature. Please always use hedging language in criticising. Remember a PhD thesis or a MSc dissertation is an eternally published document (no need for wars with other authors)
  17. Is the evidence in the paper lacking, or inconclusive, or contradictory?
  18. Are there any ethical problems or concerns with the study? (this critique is more prominent with studies involving human or animal subjects). Did the study take the necessary ethical approvals? Does the study follow the ethical standards required in the field?
  19. … please contribute to more critique questions…

Set of Critical Hammer Questions 2

  1. Let us play a little bit now using the Ambiguity Hammer,  shall we?!! Does the study lack clarity in Z? Is the paper ambiguous in what concerns K? Have the study produced equivocal results? Is it made clear that the hypotheses are supported or refuted? Do the results appear to address the issues raised in the literature review? — if you master this hammer question, you will never feel thirst or hunger for critique! Also you will become one of the most annoying academics in your department! I saw a lot of papers published in good journals that are ambiguous, I am not screwing with you! Should I give an example? OK, a paper in  ACM Web3D. Well! mmm! Maybe I will mention it in cryptic way! I do not want to be sued for defamation. I will only say that the guy gave the impression that he solved a problem he did not solve at all (OpenGL is not the same as the mobile streamlined version of the library  OpenGL ES and in similar degree, not similar to WebGL (JavaScript bindings of OpenGL ES). First one is for Desktop PCs or hardware capable devices Vs Second one is Mobile Devices Vs Third one is for Web consumption.  All of these are not the same.
  2. Is the work mature yet? Is the technology mature? Is the solution provided to the problem X mature? You can critique the Maturity of frameworks, workflows, developed tools, technologies etc…
  3. Is the work reproducible? Now Wait! this is more important than you think. An ambiguous methodology make the work un-reproducible and by consequent non-scientific and void. If you manage to prove this, this will be the harshest critique.
  4. Do the study consider the potential and full impact on X?
  5. Do the analysis of data in the study seem appropriate? Does the data suggest that there is a need for maybe other forms of more suitable analysis which has been overlooked? Did you find any unexpected findings and did the authors explain these results apropos the research questions and objectives?
  6. Do the current work take the social/economical/… impact of Z into consideration?
  7. Do the authors have any design flows you can discover? Do their work have any experimental design errors?
  8. Do they have inconsistent definitions/results etc…?
  9. Do the current work have any poorly developed theory?
  10. Do the authors overemphasize on something trivial or not that important? Do they over-rely on something they should not over-rely on?
  11. Do the current work you are checking have methodological limitations or flaws?
  12. Do the authors use standardized measures?
  13. Do the authors based their work on following/supporting a standard?
  14. Do they have a strong theoretical framework?
  15. Do the work have any shortcomings in the methods used to select cases?
  16. Can the results or findings of the author(s) be interpreted differently?
  17. The “file drawer problem” is a very common problem that results from “publication bias” which is the kind of research that “works” or that “shows positive results with acceptable effect size” that end up normally being published in journals. In contrast, scientific studies that show  “what does not work” or “what is not positive” or “what does not support a theory” and which could still be equally scientific and valid end up not being published by journals. In other words, the reaseach that is so flowery and which claim to have marvelous positive results end up in publication due to the very known publication bias… This problem is usually more emphasised in literature reviews, surveys or meta-analyses (I will talk about meta-analyses later). So here you can crtique such surveys or meta-analyses etc.. based on that if you can find that some “negative results” studies (if I can say that) were been excluded becuase the authors wanted to preserve a certain narrative.
  18. … please contribute to more critique questions…

Set of Critical Hammer Questions 3 (Statistical Hammers)

I feel sometime life is unfair for people who are doing a work that involves statistics because you have no clue of how much hammers we can use to hit you hard on the head with… Please comment below if you have more critique questions of a work based on statistics…

  1. Why does the study have small sample sizes, inadequate sample sizes or huge sample sizes? What is the right size really? You know of course in statistics and in some disciplines both too big and too small sample sizes are terrible with one worse than the other. Too big sample size in medicine per example open avenues for criticism (see this). If you know the right size that should be chosen for such an experiment per example from the literature like a specific range of sample size recommended by a standard or a recommendation, and it happens the authors are not following the recommendation. Bob’s your uncle! Was the sample random? (in case randomization is required) What is the composition of the sample? Suppose that the research is targeted to a specific population, knowing the composition is as important as the size of the sample.
  2. Was there a control group (for user/participant based studies)? Where the hell is it? (this is very common).  Was there a baseline measurement to compare with?
  3. The P-HACKING problem!!! This is so common in a lot of published papers. Please Please Please have a look at the lovely Veritasium video: Is Most Published Research Wrong? . After all, the p<0.05 is just an arbitrary selected by Ronald Fisher as a cut-off for statistical significance in a book he published in 1925. Some fields consider it adequate other not in a zillion year. The shocking thing is not this, the shocking thing is that a lot of published material in very reputable journals having the p<0.05 (considered by the field of these journals significant) turn out to be insignificant when replicated. Yup! One of the hypocrisy in academia and in natural and social sciences these days is that conferences and journals are not accepting any more replication research since it is considered ‘boring‘ and take space from other new type  of research. The glorious question, How much of published research literature is actually false? Intuitively and logically it is 5% (since by definition “if everyone using a p<0.05 as a cut-off for statistical significance, we would expect at least 5 of every 100 results to be false positives“). The reality is way way worse than that. Please read the paper titled: “Why most published research findings are false” [8] and do not forget to watch the video:  Is Most Published Research Wrong?. A great book I also advise you to read is “Statistics Done Wrong” especially if you want to learn how to discover the “most popular statistical errors and slip-ups committed by scientists every day “.
  4. Correlation does not imply necessarily a causation! This is a great motto in Statistics. Pay attention to this since many researchers make the mistake of concluding that if there is a correlation between two variables that it means that there is definitely a causation, which might not be true.
  5. Why does the study have low response rate?
  6. Is this study just limited to what is known as ‘convenience samples‘?
  7. Do the study have any selection bias in the sample? Why the study choose this population? Students/staff from the school of computer science only?, why? What about population with gamers and 3D graphics experts?  What about a population of old people? or young people? ….will that change the results? Oh Gosh, horrible nightmares
  8. Do the work suffer from any other serious sampling problems (there are many sampling problems filling huge statistical books?
  9. Now you will like this. I see tests such as ANOVA, t-tests, ANCOVA etc…  or similar in a lot of papers where somehow the authors forget that these tests are based on obligatory assumptions. A lot of the tests used do not apply to the situation presented in the paper? Dig into that! Same for hypothesis testing.
  10. Knowing the sample size, population, methodology and analysis used in the current work, do these taken in account really make the results or findings generalizable?
  11. … I would really appreciate if you can contribute to more critique questions in this category…

In addition you can get very good criticism by other scholars of a certain work using the technique I proposed before based on the “Cited By field” or “Checking the Future of the Paper” technique. An important condition here is that the paper or book should not be too new since you won’t have yet papers talking about it.

Warning: if you criticise based on wrong standing; you open on yourself literally the gates of academic Hell.

Two very good classical books I recommend to read on general academic critical thinkingcritical reading and critical writing:

  1.  Critical Thinking Skills: Effective Analysis, Argument and Reflection (cost 8£).
  2. Critical Reading and Writing for Postgraduates by Wallace and Wray

Fields in the MS Word Master File that is dedicated for Images & Tables

Remember I said, you might need an MS Word Document Master file for putting images and tables taken from others papers and books.

Organise this Master MS Word file in terms of having images and tables under the ID field of the paper in question. Since you have in your Excel Master file a relevant field (Field 17) that links both documents which states whether a certain paper has interesting images or tables to take from.

Note that a lot of examiners/Reviewers and supervisors prefer to see the your own diagrams and tables throughout the literature review.  In a literature Review, this gives a very good impression and prove to them your command on the topic and on your ability to  summarise and classify the Literature and the relevant topics it tackles in a meaningful and clear way. Diagrams are so attractive, always remember that!  “A picture is worth a thousand words” .

Here is another idea, why not merge ideas from many Literature Review Diagrams into one that you yourself create, then you can cite that your diagram is ‘constructed‘ from sources A,  B and C. A good free online tool to create amazing diagrams is Draw IO.

An abridged example of a LaTeX Master file:

\subsubsection{3D Model Indexing in Videos for Content-Based Retrieval via X3D-Based Semantic Enrichment and Automated Reasoning}
\begin{description}
\item [Keywords:] \textbf{3D Semantics} , Content-Based 3D Model Retrieval, X3D, spatiotemporal reasoning, MPEG-7, CGI
\item [Relevance (1 to 5):] \hl{\textbf{5}}
\item [Reading Status:] Read all (only 1 pass)
\item [Theme:] 3D Models \& Semantic Web Ontologies
\item [Authors:] L. F. Sikos
\item [Year:] 2017
\item[Publisher/Conference:] ACM Web3D 2017
\item [Taxonomy:] 3D Web and Semantic Web
\item [Sub-Taxonomy:] 3D Web and metadata + QoS-related Metadata
\item [Research Outcome:] L. F. Sikos introduced a novel 3D indexing method that relies on X3D alignment. There is a difficulty by software agents to understand the audio-visual content from the extraction of low level features. 3D model repositories [maybe he means Sketchfab],  provide only descriptions and tags on technical characteristics (Whether the model is low polygon or high polygon, if it is rigged, file format .obj, .max, .3ds, .blend), classification categories like animals people and licensing.
Sikos claims that these descriptions although useful but not sufficient to make 3D Models searchable by actual 3D characteristics like Shininess, material, shape, size etc…
He argues that low level features descriptors and the statistics about them are essential for classification, objects matching, tracking and retrieval. His work  showcases the use of 3D modelling ontology to describe X3D based models and videos which allow stakeholders to describe 3D models in terms of number of edges, vertices and polygons in addition to other properties like shininess, material, colour etc…. This vocabulary is based on subject-predict-object.
\item [Research Methods:] Showcasing a 3D modelling ontology and usages.
\item [Limitations:] It is only based on declarative 3D Web like X3D
\item [Similarities/Differences from my work/Gaps] Although their approach can be used in Hannibal adaptive engine,they have only tackled declarative 3D Web approaches like X3D  but what about imperative ones like WebGL? How can we use their ontology in gLTF or OBJ? It is not clear from the paper!
Questions to ask: Can this ontology be used in Omeka? It is based on X3D, can it be used with other 3D File Asset  like Obj and gLTF?

Use of Bibliography tools

If you are doing things manually in terms of bibliography inside MS Word: per example  writing each reference manually letter by letter  or in LaTeX: using thebibliography environment and \bibitem and writing letter by letter the reference, stop it! this is too primitive, too manual, too pre-computers era and a complete waste of time. You need to use a bibliography management software or tool.

There are a lot of these tools. There is Mendeley, a good free tool that I used before moving to LaTeX/BibTeX/BibLaTeX, it has neat features and it is good for the MS Word folks out there. There is also Endnote and some Web Browser based tools. Here are some other Honorable mentions: ZoteroRefWorks and Citavi

All these tools integrate very well with the majority of word processors via plugins. You need per example to install the Mendeley plugin in MS Word.

I mean you can really choose any tool you want as long as it makes your life easier not harder.  Choose what suits you! These tools are created to automate and facilitate the referencing process.

Please do not forget to consult at least a referencing guide book even when using these automated tools since a lot of them produce in many instances incomplete or incorrect citations. You can also seek the help of websites such as Cite This for Me and citefast. They would help you to manually fill out the details of a citation and then to export that citation to a format suitable to your bibliography management software such as Endnote or Mendeley.

LaTeX, BibTeX and Biblatex

NB: You can skip this section, if you do not plan on using LaTeX/BibTeX or BibLaTeX. If you used heavily LaTeX in the past, you probably know much of the material presented in this section.

Technical note 1: LaTeX pronounced «Lah-tech» or «Lay-tech», is a typesetting system and a typesetting language. According to the LaTeX project website, LaTeX is the “de facto standard for the communication and publication of scientific documents”.

Technical note 2: BibTeX and BibLaTeX are two very famous references management software and languages.  BibLaTeX is more advanced, more fancier than the good old BibTeX. Actually, BibLaTeX is considered the successor of BibTeX. Both in a technical sense are TeX packages that come by default these days with all the major distributions of LaTeX. BibTeX has an executable or a compiler that is run behind the scenes when you compile your LaTeX document. This compiler is used to parse your BibTeX bibliography notations which represent your sources (papers, books, online material etc…) i.e. entries in the .bib file. Same for BibLaTeX but the executable or compiler is called Biber. You do not have to worry about any of this as both of them are available by default in major TeX distributions. BibTeX is very old but still used. BibTeX is great for numerical citation styles such as IEEE, ACM etc… but can be extremely annoying when it comes to using an (author, year) citation style  such as (Bakri,2016) [Harvard, APA etc…] or a label-based citation style such as [ABC95] (ex: alpha) or superscript-based citation style such as the one used by the nature journal. When using only BibTeX, you end up being forced to use another LaTeX package such as natbib in tandem with BibTeX.

Nowadays, students learn LaTeX then BibTeX then move on to learn and adopt BibLaTeX. I strongly suggest to drop the usage of BibTeX in favour of BibLaTeX,

If you will be using LaTeX & BibTeX or LaTeX & BibLaTeX for your Literature Review, as long as the bibliography is organised very well and as long as each entry in the bibliography file(s) [.bib] using either BibTeX or BibLaTeX notations are complete; things would be fine. Make sure you always include all essential fields. Each type (@articles, @inproceeding, @proceedings etc…) has specific mandatory fields and optional fields. Ideally you should fill out all mandatory fields and as much optional fields as is humanly possible. This might not be easy for many reasons: articles without author or without year of publication etc… Use always comments in the .bib file(s) to classify papers/books etc.. Any character outside an entry code is considered a comment (except if it is a special character for BibTeX) in other words anything outside the entry command starting with the @ symbol. I personally use the % inside the .bib files to write comments.  The % is the same character used for comments in LaTeX. There is also the @comment which can be used to comment a large portion of the bibliography if one is not keen on deleting a lot of entries.

BibLaTeX is a superset of BibTeX so not all fields or entry types of BibLaTeX are in BibTeX but the other way is true.

I know LaTeX is pain in the neck but it is worth to use it for the beauty of the document  that you will get at the end. But Hey! this is a just a preference nothing more! MS Word is also fine.

Manual Construction of the .bib (BibTeX) file

What the majority of people  end up doing when using this method, is they create a file with .bib extension in a normal text editor or in one of the  famous TeX Editors (like TexStudio, Texmaker or Texworks…). For each reference;  a lot of folks write it down manually  in the bib file which is a  primitive method but  produces nevertheless a better and a more complete bibliographical entry!

The following is an example of a BibTeX entry:

@inproceedings{bakri2016virtual,
  title={Virtual Worlds and the 3D Web--time for convergence?},
  author={Bakri, Hussein and Allison, Colin and Miller, Alan and Oliver, Iain},
  booktitle={International Conference on Immersive Learning},
  pages={29--42},
  year={2016},
  organization={Springer}
}

There is a faster way: Many of us go to Google Scholar, search for the paper/book in question, choose the Cite Icon (the two quotes icon), then choose BibTeX, which leads to a BibTeX code  similar to the code above. The code would be copied into the BibTex file (.bib). This is done for every reference needed. You can imagine the size and cheer amount of references at the end of a PhD thesis or a Masters dissertation per example. You have the possibility of having many .bib files each pertaining to a topic covered by your thesis/dissertation. I like to divide my bibliography into many .bib files and  to use a lot of comments in the bib files because it keeps the references more organised under specific themes.

There are websites that help you cite well according to a specific referencing style and allow you then to export the citations to either EndNote or a BibTeX or to any other bibliographical format. Few examples: Cite This for Me and citefast. The problem with these websites is that the method that is used is too primitive & too manual since you have to fill manually a lot of necessary fields for citing a resource compared to using a referencing software such as Endnote or using BibTeX/BibLaTeX export features in scholarly search engines and from publishers websites but the advantage is that you know that you are following exactly the citation style you need and that your citations are both correct and complete.

One of the major problems, with the manual approach to add citations to the .bib file is that you might add accidentally the same reference many times so there would be many redundancies. Of course you can detect them and remove them and there are also a lot of tools (ex JabRef) and Python/Perl scripts that clean up the .bib file(s) for you. Some TeX editors have features that do that also.

Be Very Cautious: Google Scholar does not always give you a complete and correct BibTeX entry code. As I said, your citation should contain if it is possible  ALL the required BibTeX/BibLaTeX fields depending on the type of the citation (book, article, booklet, inproceeding…) + as much as possible of the optional fields (the more the better). Please consult a citation guide book for how to appropriately cite a specific source if these automatic tools give you something weird. I had many publishers contact me about missing fields in references. References which I have copied the BibTeX code verbatim from Google Scholar engine.

The right way to obtain a bib code: Either go to the publisher site or use DOIs. You probably have some familiarity with DOIs (Digital Object Identifiers). Every published paper/book… has a DOI number (aimed to identify digital online resources). A DOI looks something like this: 10.1145/1595496.1562908. You need to know what is the DOI of the paper in question and then go to  http://dx.doi.org/[PUT HERE THE DOI]. This will redirect you normally  to the publisher related web page of the paper where on it you will be told normally how to cite the paper properly (there is always an export format feature for BibTeX somewhere on the page). The difference here is you are following what the publisher wants you to do when citing a paper and not what Google Scholar gives you.

To illustrate this. The following is the Google Scholar version of the generated BibTeX Code of the survey paper titled (“A survey on service quality description”):

@article{kritikos2013survey,
title={A survey on service quality description},
author={Kritikos, Kyriakos and Pernici, Barbara and Plebani, Pierluigi and Cappiello, Cinzia and Comuzzi, Marco and Benrernou, Salima and Brandic, Ivona and Kert{\'e}sz, Attila and Parkin, Michael and Carro, Manuel},
journal={ACM Computing Surveys (CSUR)},
volume={46},
number={1},
pages={1},
year={2013},
publisher={ACM}
}

The publisher BibTeX code of the same survey paper (from ACM Digital Library):

article{Kritikos:2013:SSQ:2522968.2522969,
author = {Kritikos, Kyriakos and Pernici, Barbara and Plebani, Pierluigi and Cappiello, Cinzia and Comuzzi, Marco and Benrernou, Salima and Brandic, Ivona and Kert{\'e}sz, Attila and Parkin, Michael and Carro, Manuel},
title = {A Survey on Service Quality Description},
journal = {ACM Comput. Surv.},
issue_date = {October 2013},
volume = {46},
number = {1},
month = jul,
year = {2013},
issn = {0360-0300},
pages = {1:1--1:58},
articleno = {1},
numpages = {58},
url = {http://doi.acm.org/10.1145/2522968.2522969},
doi = {10.1145/2522968.2522969},
acmid = {2522969},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {QoS, SLA, Service, description, life-cycle, metamodel, model, provisioning, quality, service-level agreement},
}

You can see clearly that the code given by the publisher is a lot more detailed and precise than that of Google Scholar. A lot of publishers allow you to export the citation in different formats with or without the abstract (Yes!, you can include all the abstract text inside your BibTeX code (abstract={…}).

The smarter Way: JabRef

JabRef is one of the tools that you say to yourself when seeing it for the first time: Why the Hell I don’t know about this??

JabRef is an amazing open source Java Graphical User Interface (GUI)  bibliography reference management system similar to Mendely, Endnote, Zotero etc.. but dedicated for the LaTeX/BibTeX/BibLaTeX folks.

If you load your existing (.bib) file(s) in JabRef (by using File/Open Library). It will detect all BibTeX redundancies and mistakes in the bib file (Amazing! right!).

JabRef version: 4.3.1 - BibTeX entries Redundancies checker
JabRef version: 4.3.1 – BibTeX entries Redundancies checker
JabRef 4.3.1 - Main Window
JabRef version 4.3.1 – Main Window

You can add manually BibTeX entries depending on different types, or copy BibTeX raw text or BibTeX Code from other places. Export references to RDF, HTML, Endnote, MS Office, OpenOffice, LibreOffice….among others file formats. You can generate a BibTeX key or use the ones provided by publishers websites or Google Scholar.

You can open the DOI link of any paper/book which has it in your database. You can rank your references, mark them and give them certain colours to differentiate them. It allows you to edit/create/delete Essential, Optional and Deprecated fields. You can add an abstract for a paper and you can write comments. There is an amazing feature called ‘Related Articles’ which fetch you all related articles/papers you might need to consider.

The tool has good integration with Open source office suites like OpenOffice/LibreOffice and can link to your favourite text editor or LaTeX Editor (mainly TexStudio and Texmaker). In addition, you can search and fetch from the tool itself papers/books (such as searching ACM Digital Library and the like…)

BibDesk

BibDesk  is a graphical bibliography manager for BibTeX/BibLaTeX file management. Unfortunately, BibDesk is only for Mac OS Users. It is not as powerful as JabRef.

BibDesk MacOS image
BibDesk on my Mac OS

Systematic literature reviews and meta-analyses are gaining momentum!!!

Many scholars [11, 12] are investigating ways to approach the literature using explicit systematic procedures. You can think of it as making the process of doing the literature review more scientific, procedural and algorithmic so that it can be replicated by other scholars. Many research communities in many fields are getting sick and tired of all the fluff and nonsense in research, annoyed by all the literature reviews in academic papers and surveys that lack thoroughness and from biased literature reviews. Yes! you heard correctly: a literature review can be made biased while looking very professional :-)! Systematic literature reviews minimise bias by forcing the researcher to leave a clear “audit trail” of the researcher’s decisions, procedures and conclusions.

Many surveys in many disciplines especially the medical and pharmaceutical ones do that already  from a zillion years ago but now there is a trend emerging in other disciplines such as business [12], software engineering [13], Computer Science Programming Education [14] among others where  you give the methodology of how you conducted your literature review describing the research questions, the procedures, the sample of papers, the search inclusion/exclusion criteria, the search keywords & Boolean phrases and all keywords combinations used, the relevancy, the significance, the quality (i.e.specifying quality criteria) etc. All of these are dubbed as the “review protocol“. The review protocol must be documented and clearly defined so that to allow others to replicate exactly the literature review process done. In medicine, the review protocol is subjected to a rigorous peer-review process [13].

The  review protocol mainly the classification of papers and the inclusion/exclusion criteria and the synthesis of findings/results could be conducted in the form of groups of pairs of many academic members and would involve calculating the inter-rater reliability using something such as the Fleiss’ kappa [14]. This is because the process of classifying and filtering thousands or tens of thousands of academic papers is too overwhelming to be conducted by only one researcher. This is why fully-fledged professional systematic literature reviews are becoming group-based research projects. The demand on publishing systematic literature reviews  is extremely high at the moment especially in the fields where this was not part of the decorum of literature surveys.

For the avid reader, please read Kitchenham’s guidelines [15] on the procedures used to create good systematic literature reviews. There are complete books written on the topic. In addition, a whole set of visualisations are specifically used in systematic literature reviews & meta-analyses (especially in medical fields) such as Funnel plots & Blobbograms or Forest Plots.

Few interesting jargon terms to know concerning systematic reviews [11]:

  • Meta-analysis is a systematic literature review that contains only quantitative studies and summarises them quantitatively/statistically. Such reviews would show percentage of studies that have examined an issue X or adopted perspective Y. So they usually aggregate results across many studies. Suppose author E, mentioned drug 1 has risk of 2% of certain disease as a side effect, author K, mentioned drug 1 has risk of 6% of certain disease as a side effect etc. So in a systematic review you might show averages, mode, median etc… of percentages of such risk…  This type of reviews is gaining a lot of momentum recently in fields outside the medical and pharmaceutical ones.
  • Meta-ethnography is a systematic review that tackles qualitative studies.

The following provide tools and computer applications that help researchers in doing systematic reviews:

Specialised  Literature Review Applications

Now these types of applications are used by many for writing meta-analysis/systematic reviews. Meta-analysis is statistical analysis that combines & analyse results and findings from many studies in the literature. Many of the tools presented in this section do not have to be used specifically for a systematic literature review or a meta-analysis. Per instance, I have used Docear and two CAQDAS systems such as NVivo and MAXQDA for writing literature reviews that are not meta-analysis or anything akin to that. CAQDAS stands for Computer Assisted Qualitative Data AnalysiS.

CAQDAS are extremely powerful and enjoyable to use. They can import data from PDFs, word processors files and textual documents of all types, Microsoft Excel, Comma Separated Values (CSV), surveys results (including from services such as SurveyMonkeyQualtrics…), from SPSS, pictures, e-mails, memos, audio recordings, videos, web articles or wikis, bibliographic metadata of citations from Mendeley, Refworks, Zotero, Citavi, Endnote, BibTeX etc.., Notes from software such as Evernote, Microsoft OneNote etc. They can scrape, import and code textual data from social media web sites such as Twitter (importing tweets, #hastags, @authors etc.), YouTube (i.e video comments & video transcripts) and Facebook. Data  imported to these tools could be interview transcripts, academic papers, survey results, audio recordings or their transcripts etc …

These tools do very powerful analysis on your qualitative data in whatever form you want it and then visualise the data as mind maps, project maps, concept maps, word clouds (a.k.a tag clouds), cluster analysis graphs among many other  graphs that aggregate/summarise qualitative data and show them in a meaningful manner.

That has been said, these software do not do that automatically like magic! It is not that you throw your academic papers and then these tools would “magically” show you the patterns discovered or would aggregate and statistically analyse findings from these papers! We did not reach that level yet! It would be cool if we did!

You have to “code your qualitative data”. You have to code your academic papers. Coding in qualitative research/literature reviews has nothing to do with programming languagesCoding actually is a whole science and there are many books on the topic. In simplest terms, coding in qualitative research/Literature Reviews is classifying and categorising textual data by using codes in the sense of semantic labels to define a piece of text. Coding is facilitated by CAQDAS software such as NVivo, MAXQDA, ATLAS.ti, QUALRUS among others.

Per example, suppose you have an academic paper imported into Nvivo, you would pass through the paper and code parts of it under semantic categories. You then do that for the next and so on and so forth. Now the beauty is not in this task, actually it is daunting task. The beauty afterwards because CAQDAS software contain many powerful analysis tools both visual and non visual to study patterns, interrelationships and to aggregate findings. In addition, they provide you with a complex code querying system.

For classical books on coding, you are advised to consult:

  1. The Coding Manual for Qualitative Researchers by Johnny Saldana – this is a classical book recommended by a lot people who I have asked concerning this topic – a must read before you start coding your qualitative data.
  2. A Step-by-Step Guide to Qualitative Data Coding by Philip Adu

There are tools that are specific/ “bespokely” designed for scientific reviews and data-sets. I could not find yet any application under this category that is  purely free and open source. Please if you know any, kindly leave a comment. A lot of the following tools use advanced text mining, machine and deep learning techniques. The majority of these tools understandably are very expensive.

Docear – the awesome academic literature management suite

Docear is a free and open source management suite for academic literature. It helps you organise and discover academic literature. It integrates with MS Word (via an addon), OpenOffice/LibreOffice (via addons), LaTeX/BibTeX and PDF software. It has an academic search engine and a literature recommender system. You can draw mind maps, sort annotations created on PDFs of papers you have included among many other features.

CAQDAS Systems

CAQDAS systems are not made for “systematic literature reviews” only but can be used to facilitate doing any type of literature review.

NVivo

NVivo is a qualitative data analysis tool produced by QSR International. NVivo is a quite expensive piece of software but your educational institution might have licenses for it. Understandably, it is expensive because to tell you the truth the tool is extremely powerful. You can import you data from PDFs, documents, Surveys, Pictures, Emails, Memos, Audio recordings and Video.  Your data that you want to analyse could be interview transcripts, academic papers, survey results, audio recordings … The software do very powerful analysis techniques on your textual data in whatever form and then visualise as mind maps, project maps, concept maps, cluster analysis graphs among many other graphs that aggregate qualitative data and show them in a meaningful manner.

The cheapest student license per year cost 60 quid (so it is not cheap!). Many universities have subscriptions for this tool since it is quite famous. The following are few resources that could help get started with NVivo.

Books

Online video courses/YouTube channels

MAXQDA

MAXQDA is super powerful. I have used it myself and used NVivo in a course aiming to familiarise myself with the myriad CAQDAS systems out there. I can tell you right away between the two, I would go with MAXQDA. NVivo is famous and this is why a lot of folks recommend it but I found personally MAXQDA 2018 is a lot more powerful than NVivo 12. Before you decide please take the advise of your supervisor/advisor and other PhD candidates and other researchers who did similar qualitative or mixed methods research as you will be doing in your PhD. You can have a look at what the software can do for you: hereMAXQDA allows you to analyse qualitative and mixed methods data.

Online Video Course/ YouTube Channels

  1. Qualitative Data Analysis using MAXQDA Analytics: Hands On –  is a Udemy online video course that is quite good. The accent of the instructor is a litle bit hard to understand and you will feel he is wasting time in the beginning of the course explaining the features of the software but despite that the course is rich. This is an absolute beginner course in my opinion, it barely scratch the surface of what you can do in MAXQDA.
  2. MAXQDA VERBI YouTube channel contains many video guides created by the MAXQDA team that teaches you how to use MAXQDA. Please have a look, you will be amazed by what you can do with MAXQDA.

ATLAS ti

ATLAS ti is Qualitative and Mixed Methods Data Analysis tool. It allows you to code, and annotate your textual data and to visualise relationships between them.

Video guides

Books

EPPI-Reviewer 4

EPPI-Reviewer Software is a online systematic review software.  It was created to deal with the challenging task of doing a very challenging Literature Reviews especially reviews conducted in the medical disciplines (way more difficult than Computer Science). The tool has a powerful reference management system which allows you to search online scholarly databases or import  references in different formats. You can store the original document with the reference. EPPI-Reviewer 4 is not free (but you get to try it for 1 month).

Comprehensive Meta-Analysis

Comprehensive Meta Analysis analyses metadata from studies in the literature.

Distiller SR

Distiller SR is a massive systematic review software.

Mix 2.0 Add on for Microsoft Excel

Mix 2.0 is statistical add-on for doing meta analysis in Microsoft Excel

RevMan 5

RevMan 5 “facilitates preparation of protocols and full reviews, including text, characteristics of studies, comparison tables, and study data. It can perform meta-analysis of the data entered, and present the results graphically” {taken verbatim from official page}.

How to use CAQDAS Systems for Literature Reviews

CAQDAS systems are just awesome when it comes to creating Literature Reviews and not necessary LRs that are meta-analyses/systematic literature reviews. I find that coding all the papers retrieved facilitates the job of filling all the metadata fields explained in this article. In similar vein, it facilitates comparing/contrasting/summarizing and criticizing the relevant literature. How? You see! if you use a well-studied coding framework pertaining to your problem space across all the papers of your Literature Review, the CAQDAS software then allows you to compare a topic/a theme/a problem/ research gaps (used as codes) across all papers and thus amazingly allows you to “construct” a very good critical literature review when it comes to actually writing it up (a section that will be discussed later).

Per example, you can code or use memos to designate what author X said on a topic Y, what author K said on topic Y, what author D said on a topic Y and so on and so forth and then compare/contrast their opinions across all papers with the help of the software.

In addition, it is advisable to follow a coding framework that is based on the fields presented in this article mainly field 12 [Research Outcomes], field 13 [Research Methodology/Methods], field 14 [authors own limitations] and field 15 [your own critique]. Please use different colours for different fields (CAQDAS systems allow you do that). You can write memos or notes linking it to a code and an academic paper where you write in your own language what should go into the content of a particular field similar to the Excel method. Codes can have hierarchies such as parent/child relationship so you can use that to your advantage by creating a parent code as per example a problem X and then the different children codes would be opinions by different authors under this parent code. There are many other techniques of course. See what suits you and be consistent!

There is also another advantage: CAQDAS allows you to check the relevance of the papers or the sample of papers  that you have chosen to use in your Literature Review chapter. In other words, suppose you end up with 300 relevant academic papers as a sample for your LR. I find that creating word clouds in CAQDAS systems such as MAXQDA allows me to check that the papers retrieved are really relevant since the terms of the word clouds should mimic to a big degree the essential keywords that constitutes my research questions.

Free/Paid Tools & skills needed for Language

I wrote an article: “Skills you must know before you step into a PhD in UK” , that contains many skills /knowledge of tools etc..   that you need to know before stepping into a PhD.  Some of the skills in the following list can not be acquired overnight, but you have to start somewhere. For a dedicated article that I wrote on writing skills and tools (still a work in progress), please refer to Your Arsenal of Academic Writing Tools. The following list provide tools/skills that you need for Academic language mastery:

  1. Academic Phrasebanks: There is a cool website that you should definitely bookmark and always use especially if you are not a native English speaker (well even if you are!). The University of Manchester Academic Phrasebank, website contains all those needed academic phrases and expressions that authors use for introducing work, reviewing the literature, describing methods and methodology, reporting results, discussing findings and writing abstracts and conclusions. In addition, the website contains tons of academic phrases to contrast ideas, to express being cautious or critical, to classify/contrast and list etc… mainly these phrases are compiled and taken from around 100 postgraduate dissertations/theses (completed at the University of Manchester), and from hundreds of academic papers. By the way, apparently the PDF booklet contains more phrases than the website, in addition to tips on academic style, grammar and sentence structure but unfortunately it costs money unless if you are a staff or student of University of Manchester. Nevertheless, I totally recommend the website and it is worth even to buy the PDF (around 5£). If you are a Kindle user, there is an E-Book titled “The Only Academic Phrasebook You’ll Ever Need: 600 Examples of Academic Language“. There is also a paper version, but hey! you should begin to save the planet. I always prefer buying Kindle books than buying Paper/Hardcover books. The cost of the E-book is 2.99£ (the cost of a cup of coffee).
  2. Thesaurus: Invest in a good Thesaurus Book or Thesaurus Software/Service! Some are available free online of course and all major word processors contain Thesauri.  Nevertheless, the ones that cost money are more extensive and richer. Trust me it makes a difference! They provide you not only with synonyms and antonyms but complete replacements of expressions used in academic context. Pay attention not to use too much sugary language (such as using some weird synonym just so that you can look fancy!), this is might piss off your readers, mainly your most important readers: examiners. But a good Thesaurus helps you avoid repeating yourself but most important try always to strike a good and gentle balance in the usage of words.
  3. Fancy Language Services 🙂  If you have some cash to throw, invest in services like ProWritingAidGrammarly and WhiteSmoke and I never thought my clean conscience would allow me to recommend such tools but now things are different. These tools are getting more and more smarter so they are NOT like your good old Microsoft Word Spelling and Grammar checker or your TexStudio Spelling checker. They are far more advanced. I subscribed for 6 months to ProWritingAid , and it is amazing. It has many features (helps you find the right words, checks readability of your document (Microsoft Word has only basic readability check),  helps you avoid clichés and glue phrases etc…). It has even a plagiarism checker (not bad but not too good either). I love this tool and so far I am happy with it. I totally recommend it.
  4. Academic Rhetorical Devices: You should master the usage of Rhetorical Devices especially the ones dedicated to persuasion in academic writing. Maybe the humanities and probably social sciences know about this but for natural sciences fields; in my experience, knowing rhetorical devices in academic writing is like consuming exotic fruits that no one tells you about  before especially if you are not a native English speaker like me (in other words, you did not study this in school). What the hell is the fuss about this? You must be asking? Before I tell you the benefit, I should mention what the average Joe believe Rhetoric is . Rhetoric might either mean for him: (1) something only used by writers of poetry, fluff and novels (2) a weapon used by politicians for misinformation, untruthful communications, propaganda and for playing with minds of people.  Well, Rhetoric could be used that way. Per example Donald Trump uses a lot of  Apophasis as a rhetoric device for raising  ad hominem attacks on his opponents (this of course could never be detected by the white angry farmer voter with poor education in Texas) . In academic writing,  Rhetorical Devices have different and more useful role. They help persuade and argument. Per instance, You should learn to detect “Hyperbole”  in your writing and avoid using it. I used to be terrible, maybe I  am still 🙂 ,at constructing good parallelism. 2 goods books you should have on your shelf: Rhetorical Devices: A Handbook and Activities for Student Writers (kindle version is 7£) and Sixty-Nine Tools: Sixty-Nine Useful Rhetorical Devices Which Will Assist in Vastly Improving Your Presentations and Writing (kindle version is only 0.99£).
  5. Academic Style in a particular field/Analysis of academic Prose Skills: In computer Science Skills courses (University of St Andrews), 3 books are always recommended to students: The elements of style (Kindle version 2.5 £), Writing for computer science (Kindle version : £14), and A handbook of reflective and experiential learning: theory and practice (somehow more expensive – Kindle Version £31).  There is a big probability these books are available in your University Library so you won’t pay a penny. Ask senior academics or your own adviser/supervisor about similar books for your own field. You need to read such books when you start your PhD or Master.
  6. Logic and Critical Thinking Skills: This set of skills you need not only for writing but for all aspects of academia and even for your own life. Do you know these fallacies Ad hominem, Tu quoqueAppeal to authority, Slippery slope, straw man, and red herring? There are tons of formal and informal fallacies  that authors commit knowingly and unknowingly in their writings. You can know when unethical and hypocrite academics commit such fallacies. A good book to recommend for general critical thinking and writing in academia: Critical Thinking Skills: Effective Analysis, Argument and Reflection (cost 8£) and Critical Reading and Writing for Postgraduates by Wallace and Wray

An Idea for a software!

It would be nice to have a tool that is inspired by the workflow suggested in this article. I mean a bibliography tool (we already have that) + an Organiser/Classifier + A database where you can store the metadata fields the way I have discussed in this article or add additional custom fields and link all of these metadata to the actual papers/books semantically and programmatically over the web. The tool should also have good integration with Microsoft Word, Open Source Word Processors and a LaTeX editor. This would be a very cool Project!!! Come on honours students or Taught Maters out there!

Stage 2: Constructing the Literature Review

This is the stage similar to the analogy of cleaning/organising the data then running statistical tests, then plotting the data and finally interpreting it.

Possible Organisations

1- Intersecting Areas: your topic is an intersection of 2, 3 or more areas. So you have to get the best of all worlds.

Source: ‘LR Getting Started’ workshop given by Jane Brooks

2 – General to Specific Literature Review

This is a very common organisational pattern used by many. It is also known by the Cone structure organisational pattern.

Source:  ‘LR Getting Started’ workshop given by Jane Brooks

3 – Patchwork Literature Review

Source: ‘LR Getting Started’ workshop given by Jane Brooks

Your excel sheet is what you need to focus on now. The next phase will involve constructing sections/subsections by copy paragraphs from your master files with their citations to your literature review document. You will need to seek feedback from advisor on the argumentation, flow and structure of the LR.

Putting it all together

This phase of the process is called “narrative synthesis“. You have now a big database of papers and books which contains all you need. Begin with papers/books of relevancy of 5 and 4, but first follow the following steps:

Pass 1: Papers/Books/Articles of relevancy 5 and 4.

Step A: Create a detailed skeleton of headings, sub-heading and sub sub-headings… for your Literature Review Chapter. The headings should mirror exactly your research questions and research contributions. It should also by nature, mirror all major material in your chapters and sections. Remember you are doing a related work. Seek feedback on this detailed skeleton from your advisor and from other academics before proceeding. If you are a last year PhD candidate, you need to have your literature review supporting your chapters material (nothing more, nothing less). If you are a Master or PhD student who is just starting, you already should have a plan of work and research questions to address so mirror the topics/problems covered in your initial proposal or initial literature review according to your research questions and research objectives (do not be tempted to write fluff, generalizations or laundry lists – common pattern we see always in newbie Literature Reviews).

Some Literature reviews have a cone structure (from General to Specific), the “General” here is not usually what you think. In other words, if you are following this organisation, the “General Part” is not too General at all. It is just the parent topic of your specific tiny topic. Per example: if your research topic is: “Evaluating and testing which machine learning model is the most accurate for intrusion detection methods“, your “General”  here is “Intrusion Detection Systems”  or even better than this your General part is “Machine learning used in Intrusion detection methods” not Computer Security nor hacking :-), Now I know some supervisors/Advisors  they might ask you to write a section or two about  the general topic like computer security per example, just in order to situate the topic and these sections normally should be brief (sort of motivational sections only).

I saw excellent PhD Theses which were only 25000 words  (the maximum word limit of a PhD thesis is 80000). A lot of students write fluff and padding and non-sense in their PhD Theses thinking that this will impress the examiners who are extremely busy and get pissed off quite easily 🙂

Step B: Copy Fields 3, 12, 13, 14, 15, 16 from the database to the relevant parts of your Literature Review document. You should use different colours for each field text (so that you can differentiate what is your opinion or critique from fields that state material originating from the paper). Pay attention to the relationship that will be a benefit to you between Fields 12 and 13 VS Field 14 and 15. The trick here, you talk about what the authors did or proposed or created which is the material in Field 12 and Field 13. Then you critique/contrast/compare authors’ work with your own work (or planned work) using the power of the material you should have in Fields 14 and 15.

It helps here if you are using a CAQDAS system such as MAXQDA to view the different codes and memos pertaining to the major fields. The software will help you in comparing them and this by consequence will help you in writing a narrative combining the different fields’ material.

Step C: Organise and tidy up all the material, write it in a understandable English at least in this step.

Step D: Now Pull out the big guns! your language and argumentation arsenal. I will include later as many techniques/phrases that I gathered from reading many Literature Reviews from many PhD Theses. Eventually you need to (1) compare/contrast work(s) from the literature to other work(s) from the literature and (2) compare/contrast relevant work in the literature to your work (how you work differ,  how supposedly yours is better or easier or more holistic or whatever than other authors’ work).  Please see my Phrases/Ideas Argumentation bank later.

Step E: Tidy up and link everything together through mainly Logical Connectors and Connector Phrases of contrast/Agreement/addition etc… depending on the logic of the argumentation. Don’t despair here!, nobody in the whole world, get this right from the first time. We need enormous number of iterations and drafts to begin to have something descent.

Pass 2: Papers/Books/Articles of relevancy 3, 2 and 1.

It depends on the time dedicated for the Literature Review of course, since these papers are normally higher in number than papers of relevancy 5 and 4. Papers 3, 2 and 1 are typically the papers, books where you have material from the parent topic or expressions that explain concepts or emphasise the importance of the problems etc… Re do the steps of Pass 1 above, on all the sections and sub sections of your literature review. Mainly this will enrich probably introductions of sections (General parts of sections of your Literature Review) to situate better your topic.

Phrases/Ideas Argumentation Bank

Logical connectors/phrases, discourse markers and words (will keep adding to the list…): 

first/second/third/fourth/finally, however, furthermore, otherwise, in addition, moreover, in similar vein, in similar timespan, likely, likewise, similarly,  in a different vein, nevertheless, nonetheless, notwithstanding, conversely, in contrast, in contradiction, for instance, hitherto, hence, thus, therefore, consequently… erstwhile,… in other words, as a result, for/per example, for/per instance, in particular, due to,  provided that, in general both X and Y have U in common… However, the former do Z whereas, the latter do Y. On the one hand,…. On the other hand,….Supporting this view, Smith (2001) did X… Adopting a similar position,  Smith (2001) argues that…. Adopting a similar stance, Smith (2001)…,

Connector Phrases:

1) No harm with the usage of expressions like “The approaches presented in studies [45, 56] make them relevant to chapter X“. It is a clear expression which tells the examiners that the studies which you should have explained and criticised before you say this, are relevant to a certain chapter. You can of course say this more succinctly like saying “The approaches presented in studies [45, 56] are of great pertinence to chapter 6“.

2) I like this! so suppose you mentioned and explained several papers (using of course the fields I talked about before) so that every paper is stated in a paragraph or two. Then let us say 5 or 6 of these papers have something in common (they address maybe a common problem). I like the expression I see it quite often in well written theses:  “A common denominator of the aforementioned work [90,93,92] is the fact that X“.

A common denominator of the aforementioned work [90,93,92] in their use of technologies in museum contexts, either to create digital replicas of museum exhibits or to create virtual environments which can then be deployed as an on-site installation[taken verbatim from the Ph.D Thesis of my friend Adeola Fabola]

3) Expressions to note paucity of research in an area: No previous study has investigated X… Y has received scant attention by scholars… The field of Z is understudied… The field of Z is underresearched… A systematic understanding of how X contributes to Y is still lacking… the issue of X has attracted little attention… To the best of my/our knowledge no research studies have addressed Z… Very little research investigated Q.. There have been very few empirically published accounts of C.. – (Look at the careful/cautious academic language that you should master!!! I took many expressions from the University of Manchester Phrasebank – More goodies there)

…to be continued (still work in progress…), I will keep adding cool stuff here, return back!

How and Where Literature Review links to other chapters

Literature Review in the Introduction Chapter

Yes, that is true! A well-written Master dissertation or PhD thesis  should have in the Introduction chapter, a small literature Review. Don’t worry, you have already written your literature review, didn’t you? just write down the most important relevant previous work.
As a slight deviation from the topic, I have included a list of what your Introduction chapter should contain (taken from University of Manchester Phrasebank Guide and amended). Please if you are writing your PhD introduction chapter you should try to tick all the below – otherwise do not go even to the Viva Voce): PS: The Introduction Chapter is the last chapter you should write (just before the abstract which is the absolutely last thing you should write). The Introduction Chapter is the first impression! you have probably dated before so you should know how much this is important!

  1. Establishing context, background and importance of the topic (motivation) – Explain the significance or value of your work, the stakeholders affected (Yep! Why the hell should we care?!)
  2. Identify very clearly the problem or challenge (a lighter word 😉 – Identify the Knowledge Gap or also known as the ‘niche’ in the field.
  3. Giving a brief and most important review of the relevant Literature  – So yes!!! you need a small part of that Literature Review you worked on so hard before to go in the Introduction Chapter. What part! Well! The most important and most relevant (a very limited summary of important LR). The Literature Review in the Introduction, would normally address what is already known about the challenge (situate the theme of the thesis/dissertation) or you could use the Literature Review to show that there is a Knowledge Gap showing the paucity of research in a specific area or highlighting inadequacies or weaknesses of previous studies etc…
  4. Define Terminology / terms used in the thesis/dissertation (from Literature Review and even define your own terms, if you are suggesting new terms based on your research).
  5. Identify the Aim of research, Research Questions/hypotheses, Research Objectives – Link them to all chapters of the thesis/dissertation (where have you addressed them??)
  6. State clearly and with succinct academic prose the Thesis Statement (PhD only). Decompose that magical normally long phrase you wrote, into its constitute parts in a narrative and link it to the different parts of your Thesis. In other words, where have you addressed the different parts of your Thesis statement in your chapters? Remember you are doing a Doctor of Philosophy (PhD or D. Phil). I met a lot of PhDs that do not understand why the word ‘Philosophy‘ is in the title of their degree. Writing thesis statements is a science and an art so seek help from your advisor/supervisor, this is why these people are there for – If you are a Master this point is not relevant to you, so ignore it.
  7. Research Contributions (More relevant to PhD candidates) – Examiners look at that and they like to argue here. Important: Link them to the chapters of the Thesis also
  8. Synopsis of research design, research methods and methodology (i.e. a small and succinct version of your Methodology Chapter)
  9. Thesis/Dissertation outline or structure (a classical section).

Some people add Limitations of the research to the list above in the Introduction chapter. Waw!!! Yes I know right!!! they are the honest Masters/PhDs. Normally supervisors/advisors do not like that and thus they oblige candidates to put limitations in the Conclusion Chapter LOL 🙂 (so maybe the external and internal examiners are too bored to reach the end of the thesis or too impressed by what they have already read all the way that limitations seems not worthy). LOL!!! The Hypocrisy in academia!

Some supervisors/advisors even change Limitations section to Further Measurements/Further Future Work or whatever – See the magic of language here!!!! You see; I guess they are sometimes right and are teaching you an academic secret indirectly. This secret nobody dare to talk about.

I will tell this secret, a lot of awesome papers with massive contributions get rejected because the author(s) “undersold” themselves or their work. Hypocrite reviewers would say: Well! There are a lot of limitations in this paper, why we should accept it? why not address them and return back to us? A friend of mine, wrote a massive section called Limitation in her paper and submit it to a prestigious conference. The Paper was rejected. She changed only the title of this section to Future Work, did not change anything else and submitted it too an even more prestigious conference with higher ranking and lower acceptance rate  than the first one (actually submit it to the highest ranking conference in her topic of research ) and guess what  THE PAPER WAS ACCEPTED!. And you tell me there is no hypocrisy in academia, Really????!!!!

Every research has limitations (no exceptions), every academic paper has limitations (no exceptions), every Master Dissertation has limitations (no exceptions), and every PhD Thesis has limitations (no exceptions). You should always strive to be completely honest as an academic since that what attracted us to the sciences and to academia in the first place. The search for the truth should be our major objective. In addition, it is pertinent to never undersell yourself or your work and never oversell it.

I have been told once a story of a PhD candidate in the school of physics in our university (University of St Andrews), who was extremely brilliant, he discovered something awesome. The point I am driving home here is that the school thought he will have a Viva Voce of 1 hour or even less  than that (Imagine!) since his work was at a “Genius Level”. The guy spent many  many hours  in the Viva. The supervisor was very worried and also the school. What the hell is going on in there! It turns out the guy (which is from an eastern background like me), was ‘ very humble’ in his viva and in his thesis since he ‘underestimated‘ the breakthrough he has just achieved. The examiners wanted to emphasize this idea in the Viva since they were convinced that this person does not even know what he just did and the magnitude and impact of what was discovered. This comes from underselling a person’s own work.

Literature Review in the Methodology Chapter

The main objective of a Methodology /Methods chapter is Replicability especially in the sciences. A second objective is to show your markers/examiners/readers what is the procedure you used in your research. In other words, this chapter should be clear and detailed enough so that any other academic in your field can repeat your research using your methodology and your method(s) and reach the same deductions and results.

Yes!  Literature Review plays a role here. Your research methodology should be based on an established methodologies in the Literature. Your research methods should be based on an established methods in the Literature. In addition, your Literature Review might have addressed methodological flaws in other people’s work, if this is the case you have also to base your argument on how methods used in your thesis/dissertation, are better than other researchers’ methods; relying on a strong support from the Literature. You should also specify clearly or justify why have you used/adopted such methodology or such method(s) and rejected other suitable ones from the literature. Normally Books and academic papers that address research methodology and research methods in a discipline, present many pros and cons of each method and provide the ideal situation or research design they should be used in.

Again you already wrote a massive, hopefully very good literature review following my technique  cool. So utilise some material but this time ONLY the material from the Literature Review that talks about Methodology and Methods (+ flaws or weaknesses in other relevant work methods/ contrasted to your work). This is also called a methodological review.

For CS only: Ok! I know, I know!!! Computer Science is very poor when it comes to formalising Research Methods or Research Methodology since it is a hip new science with a chronic severe hangover (clueless and happen to be awake always in a dumpster). When you reach the stage of Methodology in your research, you ask yourself and others, for the LOVE OF GOD, where the hell are all the books/papers that talks about Research Methods and Research Methodology used in Computer Science (all fields)?

Computer Science relies on any research methodology it can have her hands on from the sciences (saying that without any shred of hypocrisy of course!). CS has refactored many research methods from other scientific disciplines and invented zero research method. Have a look at the article I am writing on How to write a Research Methodology Chapter in Computer Science without fluff and nonsense? – it is still a bone structure.

Since this article is not dedicated solely for Computer Science students, it is unfair to drag this article in this direction. Maybe first order of business after I finish my PhD is to write a descent academic textbook which will be an encyclopedia of research methods and methodologies for major computer science fields (which does not exist yet, believe it or not!).

If you are not a Computer Science student, you should have no problem locating good well-established papers and books about Research Methods/Methodologies in your discipline.

As a PhD candidate or as a Masters student you should read at least one book that covers the research methods in your specific field before you start your research. There are many cross discipline books that cover many disciplines but I mean if you can find a book tailored to your specific field of research why bother with high-level books!

The following sections present few suggestions of methodologies books that are cross-disciplines and methodologies/methods books for few fields:

Good Books that are cross disciplines

You can of course buy one of the following list (they are good!) but I believe if you are doing a specific field of research, it is better to buy (or borrow from the library) a book that covers directly your field or even your particular sub-field instead of reading a book that is too general  when it comes to covering research methods. Ask your supervisor/adviser, other academics or PhD candidates about the best research methodologies/methods textbooks they recommend:

  1. Research methodology: Methods and techniques by Kothari
  2. Research methodology: A step-by-step guide for beginners by Kumar [Covers health, education, psychology, social work, nursing, public health, library studies and marketing research]
  3. How to research by Loraine Blaxter
  4. Introduction to Research – Understanding and Applying multiple strategies by Elizabeth Depoy
  5. The Good Research Guide – For small-scale social research projects by Martyn Denscombe (6th Edition) – a very good book for the social sciences folks
  6. Quantitative Methods in Social Science by Stephen Gorard [cross discipline book but only for social sciences such as economics, sociology etc.]
  7. Keywords in Qualitative Methods by Michael Bloor et al.
  8. Your Research Project by Nicholas Walliman

Computer Science Methodologies/Methods Books

There is a scarcity of books that cover Research Methods in Computer Science as a whole (as kind of an encyclopedia or textbook that “rule them all” if we can say that). Nevertheless there are few awesome people who wrote books on research methodologies/methods  for their respective fields in CS. “Chapeau Bas” and a big thank you for them and for their great job!!

Human Computer Interaction
  1. Research Methods in Human Computer Interactions by Jonathon Lazar et al. – a amazing book recommended by the majority of HCI supervisors. A must read if you are in HCI. It also helps people studying Quality of Experience, Usability/Accessibility and other similar in nature user studies.
  2. Research methods for human-computer interaction by Paul Cairns and Anna L. Cox
  3. Doing Better Statistics in Human-Computer Interaction by Paul Cairns
Artificial Intelligence
  1. Empirical Methods for Artificial Intelligence by Paul Cohen
System Performance, Simulation and Quality of Service Methods

Many supervisors in UK universities recommend Raj book to their students to read. If you are doing a PhD in Computer Networks’ Quality of Service (QoS) or any system performance studies, the following list is for you:

  1. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling by Raj Jain – This book is the holy grail on the topic.
  2. Measuring Computer Performance: A Practitioner’s Guide by Lilja
  3. Systems Performance: Enterprise and the Cloud by Brendan Gregg
  4. Performance Modeling and Design of Computer Systems: Queueing Theory in Action by Mor Harchol-Balter

Business Research Methodologies/Methods Books

The best and greatest book on the topic which is the classic of the classics: the book of Bryan & Bell – a must read for the folks in Business and even economics. The following list elucidates few recommended books for Business research Methods:

  1. Business Research Methods by Emma Bell, Alan Bryman and Bill Harley – newer edition Bill Harley wrote few chapters – if you are in a hurry you can buy the old 2011 version.
  2. Research Methods for Business Students by Saunders et al. is a GREAT  book on research methods. If half of the supervisors in UK recommend Bell & Bryman book, the other half would recommend this one.
  3. Mathematics for Economics and Business by Ian Jacques – Now this book, according to a friend of mine who is doing a PhD in Economics, was strongly suggested for reading by her supervisor together with Bell and Bryman book.
  4. A dictionary of business research methods by John Duignan (kindle version and paid online version)
  5. Research methods for managers by Gill and Johnson
  6. Handbook of qualitative research methods in marketing by Russell W. Belk.

Psychology Research Methodologies/Methods Books

The lovely psychology discipline has a plethora of books on research methodologies and research methods. Ask your supervisor to suggest a book. It should be noted that Computer Science borrowed a lot  from psychology research especially in the fields of Human Computer Interaction, Quality of Experience, Accessibility and Usability. The best of the best books on research methods in the field of psychology are stated below (please if you have any suggestions, write them in the comments):

  1. Research Methods and Statistics in Psychology by by Hugh Coolican – Buy the new edition (2018) – it is amazing!
  2. Research Methods and Statistics in Psychology by by Haslam and McGarty
  3. Statistical Methods for Psychology by David C. Howell

Political Science

  1. Political Science Research Methods by by Johnson and Reynolds
  2. Political Research: Methods and Practical Skills by by Sandra Halperin and Oliver Heath
  3. Research Methods in Politics by Burnham et al.

… There are many other suggestions of famous methodologies books for a lot of disciplines, you can find them in the PhD skills articles or Computer Science research methodology article.

Literature Review in the Results/Implementation or Analysis Chapter(s)

Well there is not much literature review in these kind of chapters other than in the introductions and summaries of those chapters, I usually found that a lot of people link back to the literature review and talk about the most relevant studies and how the current work  addressed, enhanced etc… on previous works and contributed to the literature. So the literature review in such chapters should be very directed and aiming at showing contributions.

Literature Review in the Discussion/Evaluation Chapter(s)

Some people call the chapter: Discussion, some call it Evaluation chapter. Some have even more than one of these. One would be called Evaluation (they actually evaluate a system implementation or experimental results) and the other would be called Discussion. Call them whatever you want.  These chapters are the most important chapters really because they normally have the following three major purposes: (1) To actually evaluate results/system implementation etc.. ;  (2) To connect everything in the whole thesis together (very important! You want the Aha moment here!!), (3) To evaluate your work on grand scale Vis a Vis of the Literature. A lot of times, I begin reading a PhD thesis or Master Dissertation from the evaluation/Discussion chapter(s). I know it is weird but I will tell you why!

If these chapters are well-written, they can tell me everything I need  to know about the work.  Well! I might not know the intricate details of the results and the analysis (since I did not read those chapters) but I would know what is the core argument of the thesis or dissertation.

Questions you might have asked yourself while writing the literature review

How can a LR defend the scarcity/absence of research in a certain Topic X?

Oh! Lord! what a struggle!  your literature review aims to defend your research and shows to examiners and readers why your research questions are pertinent. That has been said, how do you prove an argument such as: “Although a lot of research has been done on topic X, little or none has been done on topic Y”?  X and Y are closely related of course. This is not easy at all by the way and you need to be very convincing in your literature review by surveying X to a degree and then stating something such as: “To the best of my knowledge no research tackled Y”. Of course it helps a lot if you can find researchers in the literature mentioning the scarcity or absence of topic Y or that topic Y is under-studied or under-researched.

How can a LR defend using a research method that is not established or not common?

This is even harder for your LR to defend than the previous question. I discussed once that with my PhD supervisor and I remember he said that normally if you use research methods outside the one commonly used (or commonly established ones used in the literature, you have to have a good defense for that in the thesis mainly in the methodology chapter and to be prepared to defend that in the Viva Voce. It is uncommon to see that, but usually PhD candidate have already strong reasons why they chooses such research methods. So you need to ask yourself: Why have you used such research methods and not the common ones? What makes the methods you used so special?

He said in the form of joke that probably the first person who came out with the concept of using mixed research methods had to defend such choice vehemently. Now it is very common!

———————————————————————————————————————–

This work is still in progress, please comment and share anything important which I did not mentioned or tackle in this article

———————————————————————————————————————–

Compiled based on divers techniques and ideas used and taught by my lovely mentors Jane Brookes, Mark Carver and Mary Carr (ELT Team of the University of St Andrews); and from many muddy experimental techniques used by Hussein Bakri and Amjad Al Tobi.

The jokes and weirdness in expression in this article and any errors serious or small are only the result of my own shortcomings.

If you are a student of the University of St Andrews and you are struggling with your Literature Review or with any part of your dissertation or thesis. I advise you to seek help from the English Language Teaching (ELT) team – They are AMAZING!

Every year they organise a Dissertation/Thesis writing week (for both Masters and PhDs) with tons of advice like the one you have just read in this article and I would say a lot better that my lay advice. I am actually a very sloppy writer!

I totally recommend you to seek their advise. You know supervisors/advisers are sometimes sloppy, cryptic, not helpfully in in the best cases, they think you know these things or too lazy and hypocrite to tell you anything useful! ELT are the people you need!!!

In addition, some supervisors/advisors give you grandma feedback meaning nothing of substance to your topic or give you language feedback (Language feedback is not really their job, It is good if they give you that but if you realize that is the only advice they are giving you then you have to read  an article that I wrote that helps you deal with supervisors/advisors neglecting you or giving you no feedback or little feedback or giving you feedback of no substance. I will pinpoint statements from University policies of essence in this regards  (for UK universities – Sorry) minimizing the time you need to look for them and file a winning complaint. The article will teach you how to protect yourself (by audio recordings or minutes of meetings among other techniques..) which are your rights so that you always have a proof. In addition, the article will deal with harassment, discrimination, bullying, blackmail among many others bad behaviour that we are witnessing even in the most prestigious and well-respected universities. This article is a work in progress and contains the vast experience of many people that will contribute to it who either were victims of such behaviour or who dealt with it by disciplinary actions.

Back to my suggestions of people that might  help you out  also in writing Literature Reviews, we have also in the University of St Andrews, a team usually referred to as  CAPOD, which they organise a lot of workshops on many topics like statistics (R, Statistical Package for Social Science [IBM SPSS]…), writing and presentation skills among others. I attended a lot of CAPOD workshops, they are quite interesting and useful but with complete sincerity, and it is my opinion of course, the ELT workshops were truly amazing in what concerns writing theses and dissertations!

References

[1] Cooper, Harris M. “Organizing knowledge syntheses: A taxonomy of literature reviews.” Knowledge in society 1.1 (1988): 104.

[2] Randolph, Justus J. “A guide to writing the dissertation literature review.” Practical Assessment, Research & Evaluation 14.13 (2009): 1-13.

[3] Muhamad Risqi U. Saputra, Andrew Markham, and Niki Trigoni. 2018. Visual SLAM and Structure from Motion in Dynamic Environments: A Survey. ACM Comput. Surv. 51, 2, Article 37 (February 2018), 36 pages. DOI: https://doi.org/10.1145/3177853

[4] Britta Meixner. 2017. Hypervideos and Interactive Multimedia Presentations. ACM Comput. Surv. 50, 1, Article 9 (March 2017), 34 pages. DOI: https://doi.org/10.1145/3038925

[5] Adrien Maglo, Guillaume Lavoué, Florent Dupont, and Céline Hudelot. 2015. 3D Mesh Compression: Survey, Comparisons, and Emerging Trends. ACM Comput. Surv. 47, 3, Article 44 (February 2015), 41 pages. DOI=http://dx.doi.org/10.1145/2693443

[6] Shu Shi and Cheng-Hsin Hsu. 2015. A Survey of Interactive Remote Rendering Systems. ACM Comput. Surv. 47, 4, Article 57 (May 2015), 29 pages. DOI=http://dx.doi.org/10.1145/2719921

[7] Ioannidou, Anastasia, et al. “Deep learning advances in computer vision with 3d data: A survey.” ACM Computing Surveys (CSUR) 50.2 (2017): 20.

[8] Ioannidis, John PA. “Why most published research findings are false.” PLoS medicine 2.8 (2005): e124.

[9] Golding, Clinton, Sharon Sharmini, and Ayelet Lazarovitch. “What examiners do: What thesis students should know.Assessment & Evaluation in Higher Education 39.5 (2014): 563-576.

[10] Golding, Clinton. “Advice for writing a thesis (based on what examiners do).” Open Review of Educational Research 4.1 (2017): 46-60.

[11] Bell, Emma, Alan Bryman, and Bill Harley. Business research methods. Oxford university press, 2018.

[12] Tranfield, David, David Denyer, and Palminder Smart. “Towards a methodology for developing evidence‐informed management knowledge by means of systematic review.” British journal of management 14.3 (2003): 207-222.

[13] Budgen, David, and Pearl Brereton. “Performing systematic literature reviews in software engineering.Proceedings of the 28th international conference on Software engineering. ACM, 2006.

[14] Luxton-Reilly, Andrew, et al. “Introductory Programming: A Systematic Literature Review.” (2018).

[15] Kitchenham, Barbara. “Procedures for performing systematic reviews.Keele, UK, Keele University 33.2004 (2004): 1-26.

[16] Saunders, Mark, Philip Lewis, and Adrian Thornhill. Research methods for business students. Pearson education, 2016.

[17] Keshav, Srinivasan. “How to read a paper.” ACM SIGCOMM Computer Communication Review 37.3 (2007): 83-84.

[18] Smyth, T. Raymond. The principles of writing in psychology. Macmillan International Higher Education, 2017.

Suggested Readings (very good books, no nonsense!)

I know books of that type might be boring but the following are really good books on how to write Literature Reviews:

  1. The Literature Review: A Step-By-Step Guide For Students by Diana Ridley – I found this book to be the best on the topic. It is very easy to read. I love this book. I read it cover to cover.
  2. The Literature Review: Six Steps to Success by Lawrence A. Machi et al.
  3. An Introduction to systematic Reviews by David Gough et al. – the book is amazing although the first few chapters contain some fluff but this book contains tons and tons of advices. It is an algorithmic book it gives an actuall appoach you can actually use not advices from the clouds.
  4. Conducting Research Literature Reviews from internet to paper by Arlene Fink
  5. Seven Steps to a Comprehensive Literature Review by Anthony J. Onwuegbuzie & Rebecca K. Frels

“Dum spiro spero prudentia et candore” (“While I breath I hope for wisdom and purity”)

Leave a Reply

Your email address will not be published. Required fields are marked *