R |

Quarto: a library to run them all? A collaborative exercise to use, learn and assess quarto for authoring reproducible documents in different scenarios

Thu, 01 Sep 2022 13:00:00 +0000

Using literate programming is a widespread practice amongst data scientists. This practice not only encourages data scientists to produce transparent, rich and reflective accounts of their analysis without the extra overhead of switching between tools, but also leads to artefacts (i.e., notebooks) that are increasingly becoming a medium for dissemination, reproducibility and education. Rmarkdown or Jupyter notebooks, are two of the most well known and used options. While both solutions can be used with multiple programming languages, the decision of whether to use one or the other is almost certain to be exclusively based on that. At least until now, with Quarto being mature enough to become a game-changer.

Quarto is a language-agnostic software based on Pandoc to render files combining markdown and code into multiple ranges of formats and outputs. As a result, it can be used with either R, Python or Julia without any other dependencies.

This collaborative workshop will be structured as follows:

a brief theoretical introduction and instructions;
a task that participants may choose from the different use cases provided (i.e. generating a single document, migrating from Rmarkdown or Jupyter, creating a book or generating interactive content); and
a group discussion and conclusions.

Participants in this workshop have fun while gaining enough depth of breath and practice to evaluate how feasible it is to use Quarto in different scenarios and, ultimately, if it can become the one tool for authoring reproducible scientific or technical documents, regardless of your language of choice.

Expertise level

Practitioner
Expert

Audience

This workshop is targeted to anyone interested in reproducible publishing, especially if they are looking for a single solution to use regardless of their programming language and are willing to explore with others quarto’s pros and cons, contributing to a group discussion.

Managing R script dependencies: automagic and renv

Thu, 14 Jul 2022 13:00:06 +0000

At the end of the we had an interesting discussion on the need for reproducibility. This talk introduces the concept of reproducibility and focuses on one of it’s many facets: that of managing R script’s dependencies.

In order to do so, two different methods will be presented, compared and discussed: the simple yet works-out-of-the-box {automagic} and the more complex, yet backed up by RStudio, {renv}

At the end of the session, points in common as well as limitations will be highlighted, hopefully leading to a discussion and opening the door for the second talk of the reproducibility series.

Visualisation tool for a grounded FEW Nexus

Fri, 20 May 2022 00:00:00 +0000

Streetscape Perception Modelling – Theoretical Considerations and Methodological Possibilities

Fri, 17 Dec 2021 13:00:06 +0000

The influence of features and properties of the urban built-up environment on people’s sense of safety and perception of beauty, social vibrancy, and walkability is a topic of interest of urban geographers, designers, planners, and environmental psychologists alike. Along with emerging forms of data and the computational paradigm of Artificial Intelligence, current GIS, citizen science, and sensor technologies offer exciting technical and methodological possibilities for extracting, representing, and modelling aspects of people’s perception of streetscapes. This workshop aims to explore these possibilities, exchange research experiences, and discuss the theoretical grounds based on which we can operationalise, i.e., model, streetscape perception, particularly based on geospatial technologies.

The workshop will offer an open and interactive environment for researchers of all levels to discuss questions including, but not limited to:

What theoretical frameworks can guide us in extracting, representing, and modelling different streetscape perception aspects of people from different social groups and cultural backgrounds?
Which variables, parameters, and indicators these theoretical frameworks suggest and how can they be reliably extracted based on different digital technologies and participation methods?
Which GIS interfaces, data structures, and visualisations are effective in representing people’s perception of streetscapes as to foster theory development and inform planning and policy design towards more sustainable and inclusive urban spaces?

An outcome of the workshop will be a written summary of the discussed ideas authored by all participants and published in the PLATIAL’21 proceedings. The indicative programme is as follows:

Introduction by the organisers (20 min)
Crash presentations by participants (optional) (20 min)
Break-out group discussions (30 min) + World Café (30 min)
Paper writing outlook (20 min)

Creating Interfaces

Mon, 11 Oct 2021 00:00:00 +0000

An open source prototype for a visual interface to support research and Nexus engagements, designed collaborativelly as part of WP4, developed by the at the .

Aim

The aim of this tool is to provide an interface capable of understanding the implications of our decisions regarding food, and how can meals in kindergartens be turned into drivers for positive change for the health and the environment.

The visualisation tool may be helpful to perform tasks such as:

Identify highly rated meals and the lower rated meals
Identify meals with higher footprint
Identify ingredients with higher footprint

Which, ultimately, will lead to discussions and reflections on how food, energy and water are interlinked and how small changes in food can make a big impact.

Online demos

Food in Kindergartens, Slupsk (Poland):
Food waste, Wilmington (USA):
Local producers, Tulcea (Romania):

Source Code

Source Code:
Licence: GPL

Acknowledgements

This study was conducted in the Creating Interfaces project, funded within the framework of the Sustainable Global Urban Initiative (SUGI) Food-Water-Energy Nexus program. This program has been set up by the Belmont Forum and the Joint Programming Initiative (JPI) Urban Europe and has received funding from the European Union’s Horizon, 2020 research and innovation program under grant agreement # 730254 and the following national funding agencies: United Kingdom Research and Innovation funding was received through the Economic and Social Science Research Council (ESRC) grant ES/S002235/1; the National Science Center (NCN) of Poland funded this work under grant #UMO-2017/25/Z/HS6/03046.

Considerations on the importance of data and science in data science

Sun, 05 Apr 2020 00:00:00 +0000

I must confess: during these days of lockdown, I have been toying with dashboards and infographics about COVID-19 outbreak, just like everyone else. with nice plots, some formatted tables and even some (basic) maps. It’s shiny and nice, you can spend some time playing with it turning layers on and off, hovering, zooming… And, honestly, I have to admit that I am proud of some of the results I have achieved. And yet, it is flawed. Just like everyone else’s (or almost). And yet, I will keep improving it, even though I’m afraid it will always be flawed and even though I acknowledge that it will never be a contribution to improve knowledge on the topic.

Why, then, am I persisting on keeping working on it if I know I cannot change its fate? Admittedly, at some point, I asked myself that very question and I even considered quitting. Not only I didn’t want to lose my time (even in these days when we are locked down at home there are plenty of things we can do), but I didn’t want to contribute to generating noise, misinformation and even more dramatism about an already important drama. Because that’s what flawed graphics do. But in the end, I realised that working on a dashboard like that could be a great opportunity for learning by doing.

Many types of lessons can be learnt from it, like those related to technical skills, or those related to how data is gathered, visualized and analyzed. Today, when data and figures on COVID-19 are everywhere, I want to share some reflections on the science (or lack of it) in data science.

1. Get the (right) data.

As obvious as it sounds, there is no data visualization nor data science without data to visualize or analyze. Therefore, the first thing that anyone who wants to make any type of data visualization or data analysis is to get the data. Second: we cannot use any type of data. We need to use good data, and by good I mean reliable, usable (in terms of licences, formats and structure), up-to-date and frequently updated, and, hopefully, official data that is representative enough to explain the phenomenon we are studying. While this is usually non-trivial, it is even more crucial if we are to explain a completely new phenomenon that it is happening as we speak and it does at a global scale like COVID-19.

Usually, there are only two possible options¹: either gather data by ourselves or rely on others’ data. Whereas gathering our own data might be the best choice for some scenarios, in the case of COVID-19, it is unlikely that we are in a position to gather the kind of data that might be useful for us (in fact, even governments are struggling to do so, as we will see). Therefore, we are left to just one option. Of course, we cannot rely on some random person or institution, we need to rely on someone we can trust, like universities (because they tend to provide rigorous data), governments (because they provide official data) or organizations (like the ). But where do we get the data from?

WHO’s COVID-19 dashboard (Screenshot from 02-04-2020)

While most governments provide data in open licences that allow its use and reuse for any purpose, they usually fail to address another important issue: the file format and data structure. As a result, in countries like the UK or Spain, data is not ready to consume as it is, without manual work². Others, such as WHO, provide nice dashboards with data, but they do not provide the raw data to be consumed by 3rd parties. The good news is that there are people and institutions who are working in releasing clean data with open licences, such as , that gathers data from different governments and WHO, and even that fetches data from UJH and provides a nice data frame for developers and data scientists to use in their projects.

As a result, it is no wonder that most infographics and dashboards worldwide rely on the same data sources. It seems a sensible decision: not only we get data which is ready to use, but we do it from trustful sources. And yet, as I will argue, it is because of that reason that most of them are wrong. But what could possibly go wrong?

2. Don’t take data too seriously

Now that we know where to get the data from, there is something we have to be aware of: by relying on data generated by others we have not solved the (main) problem of data gathering, we have simply transferred the responsibility to somebody else, but someone still has to deal with what we have been trying to avoid. And, surprise, not every country gathers the data in the same way.

Take the case of the most basic and crucial question: how are the number of confirmed cases defined. Since COVID-19’s symptoms are very similar to those of influenza and the only way to know if someone is infected by it is by testing positive in the tests³. This seems a great definition: we have an objective test which is same the for everyone and all countries seem to use the same criteria. Unfortunately, those tests require equipment which is scarce (compared to the current worldwide demand), can only be made in hospitals, and require up to two days to get the results. Therefore, there are many other scenarios that are not considered within this test, such as those shown in figure 2a. So yes, every government provides that figure, yet all of them are much lower than the real figure. How much lower? There is no way to know.

Confirmed figures ≠ actual figures. There are many more cases than the official ones. How many more? We can’t possibly know

Let’s focus on another example: the number of deaths by COVID 19. Apparently, this should be easier. Every country keeps a record on the number of deaths per day and its cause of death, so it should be easy to filter those who died from COVID-19 amongst all the possible causes. Well, no. As we have seen, if we can’t define with precision the number of people who are infected by COVID-19, we will not be able to know the number of people who have died as a result of it.

But it can be even trickier if we consider that every country has different criteria on how they count the number of deaths of people who were tested positive⁴. Take the case of UK’s definition:

The figures on deaths relate in almost all cases to patients who have died in hospital and who have tested positive for COVID-19.[…] These figures do not include deaths outside hospital, such as those in care homes, except as indicated above.⁵

So, again: real figures are much higher than those reported, no matter which country made the measurements. All of them are wrong. Some people argue that governments do not want to provide real figures not to create even more social alarm, lose popularity with their voters or even to look better than other countries. However, often the simpler answer is the most probable one⁶: it is not that governments want to hide information from us, it is just that no country has the means to face this outbreak, nor to mention to take accurate metrics. And here lies another drama of COVID-19 that goes beyond the personal tragedy: no country in the world is prepared for the stress test that COVID-19 represents.

It is not that Governments are conspiring hiding information, it is just that they do not have the means to get better figures. And here lies the tragedy!

But let’s go back to our path: that of data and figures. At this point, we have to acknowledge that all the data is flawed and we cannot get a perfect picture of the real situation out of them. If we wanted to do so, we would need to use other data sources, like comparing the record of total daily deaths⁷ with the same period last year(s). Of course, this will need more time, and, in turn, this also has other implications and problems (for example, it will not give an accurate number of deaths by COVID-19, as there is no way to know their cause of death, but the significant difference between periods could be a good proxy).

So we have two options now, either losing faith completely in all COVID-19 infographics and metrics or to acknowledge their limitations and assume that they are just a rough approximation to reality.

We have two options: either losing faith completely in all COVID-19 infographics or assuming they are no more than a rough approximation to reality

3. Choose the right figures and visuals

Great! If you are reading this it means that you are ok assuming that reality is (as always) far more complex than what nice dashboards can ever show, no matter how fancy they are. And speaking of that: beware of fancy visuals!

I’m sure that at this point you may have seen plenty of neat infographics of any type, like the ones above: some of them use boxplots, other lines, other scatterplots… some of them have smooth edges, others have axis with logarithmic scales. Even there are some that want to introduce geospatial analysis and present maps of different kinds (choropleths, bubbles, sizes)… and if you are like me, you can enjoy watching them and interacting with them during hours. But are they really effective to display useful data? Unfortunately, most of them are not (even some of my own).

One of the most basic yet frequent is to display a big figure of the total cases within a country. Big figures are really catchy and easy to understand, but they usually lack some context to make them really meaningful. Of course, anyone can understand that a 7-figure number is a big one, but it is really difficult to know how big it is. We need to compare it to something else to wholly grasp its real magnitude. Also, since we are dealing with a cumulative figure, knowing the analysed time span is a must, as it is not the same to reach a certain figure in one day, one week, one month or one year.

A common variation of that is to provide a line plot of cumulative cases, with the dates on the X-axis and total cases in Y-Axis. There are several variations of that: displaying relative data (eg: number of cases/population), with logarithmic scale (in order to make it easier to see the variations of the first days when compared to most recent ones), displaying several categories, either representing different regions or type of cases… Whereas they are really effective and most of them are right from a technical standpoint (especially considering that some of these variations make a great difference), it is the representation of figure itself that may be of little or no use. Is it really representative of something? What kind of questions can we answer by providing a number that, by definition, will always grow?

Line plot displaying cumulative cases. Is it really meaningful? Source:

It is for that reason that some have decided to use other indicators in order to assess the evolution of the pandemic, such as the number of new cases per day, as they allow us to easily identify if figures are better or worse than the previous day. Again, the same variations of the previous plots can be made in order to make this even more insightful, such as the following barplot, which displays the daily variation of cases, grouped by types. Not only we can see that they are starting to lower, but also, that the number of recovered cases is increasing over the deaths or active cases.

Daily cases, by type. Source:

This is indeed more useful than cumulative cases, although it is somewhat volatile and can lead to confusion. Take the image below, as an example:

Can you see that weird unexpected variation? It is the weekend effect! (That’s why in the next dashboard version I will be highlighting the weekends). Source:

Can you see how all figures suffer from a dramatic fall on 29th and 30th March just before peaking again the day after? This is indeed an unexpected behaviour and really difficult to explain. Unless we realise that those days were Saturday and Sunday, and due to the fact that there are fewer people working at hospitals, data is not taken as fast as usual and therefore, accumulates on Monday. This phenomenon has been called “the weekend effect” (Did I mention that you should not take data too seriously? 😉)

It is because of that that some others, such as from Financial Times, prefer to use a rolling average of a fixed period (such as 3 or 5 days), which is a more stable figure, such as that shown in the figure below.

Source:

Other techniques used to fix those outliers and display tendencies are to use smooth line plots based on the actual data, like the following plot made by .

Smoothed lines based on actual data. Source:

4. Do not make hasty comparisons

Surely, the most frequent type of visual is that comparing how COVID-19 is affecting different regions, either within a country or comparing different countries (usually using China or Wuchan as a reference -after all, it is where it all started). While this kind of plots could provide answers to questions such as how a specific region is doing regarding another one (and therefore, replicating or avoiding their measures against COVID-19, for example), those comparisons are really problematic. For starters, the fact that population or size is largely different invalidates any comparison in absolute terms.

But even when using relative values (eg: number of cases per inhabitant), there are other key factors that have a direct impact on the evolution of the disease and its effects are assumed to be the same, while in reality can differ in several orders of magnitude, such as demography⁸, geography, urban settlements or health systems (in terms of human and financial resources), just to name a few.

Histogram of cases and mortality rates by age and country. Most cases in Italy were from 70-90, as opposed to 20-30 in South Korea, hence the enormous difference in the total of deaths. Source:

Amongst all those differences, though, the most relevant well may be the number of tests done to detect COVID-19, a figure that is not just different but also usually unknown. This is by no means trivial, as we have seen that the number of cases is defined using this parameter. Therefore, if a country performed a really low number of tests, it will also have an extremely low number of cases. Out of sight, out of mind.

So, the biggest problem here is that without taking into consideration those factors, comparisons may render plenty of biased conclusions that have nothing to do with reality, such as come countries may be dodging COVID-19 or are kind of immune, or that COVID-19 only affects countries with a bad health system, unorganized governments, or undeveloped countries. As a result, there is a risk of developing a narrative of moral superiority⁹ based on totally wrong foundations like what some politicians have started to do in their own self-interest¹⁰.

Don’t be like Wopke Hoekstra doing hasty comparison, or you risk reaching repugnant conclusions like him. Photo:

5. Take predictions even less seriously

The last group of plots, and the most complex ones, are those that make predictions. While they are really appealing and, apparently, provide answers to one of our main concerns (“When is this going to end?” / Will this last any longer?) in a very understandable way, they are really tricky. There are several ways to make predictions, such as using linear regression or models. While a model can be as easy¹¹ or as complex as we want it to be (and as a result, their accuracy will differ dramatically), they mostly rely on having a good set of historic data or knowing the logics of the phenomenon they want to describe. Unfortunately, since COVID-19 is a new phenomenon, we are lacking of both, and therefore, predictions at this stage are prone to errors. Some predictions are based on what has happened in other places where the outbreak started before, but we have seen how problematic this can be.

If we can’t see (nor understand) the model behind a prediction, it can be as useful and reliable as that of a fortune teller.

Last, but not least, we should not forget that they require a large amount of knowledge of a particular field something like most of the people who are doing those nice visualizations (including myself) lack of. Therefore, since I have openly admitted that this is something beyond my knowledge, at this point I can only but recommend to be sceptical about any prediction that is not made by an authority on the field. And even in that case (provided I could understand it), I would recommend caution. Or even better: just rely on predictions if they were made by epidemiologists and you are one of them.

Wrapping up

As argued, if data visualization is never easy, it is even less so in the case of a novel phenomenon such as COVID-19. Therefore, when facing any type of visuals, we should proceed with caution. If you are doing (or planning to do) any type of visualization, ask yourself what question do you want to answer and which is the best way to do it, take into account the aforementioned considerations and make them evident to your readers¹². Also, make your analysis reproducible, so anyone could tell you if you did something wrong or even fix it by themselves. If you are simply watching them, look for all those explanations, and if you can’t find them, ask for them, help the author or simply ignore it and look for an alternative. But in any case, you should always remember not to take data too seriously or too blindly. Data by itself is not what really matters, is what we do with it and how we do it in order to achieve knowledge what really matters. And here’s when science plays its role.

There is a third scenario: to infer or calculate the data from other datasets. ↩︎
For example, , without a clean structure. Not only they use a proprietary format, but they mix data with metadata on the same file, even in the same sheet, instead of using a wide format where every column is a field and every row is an observation or a long format where keys and values are stored as in a dictionary. On the other hand, Spain decided to release the data in PDF. PDFs are great for visualization because they are an ISO format and can be opened with plenty of different softwares. Unfortunately, it is not great for consuming data, as it is not structured in any way. ↩︎
At the time of writing this post, and according to ’s page, the . The standard method of testing is real-time reverse transcription polymerase chain reaction (rRT-PCR), typically done on respiratory samples obtained by a nasopharyngeal swab and results are generally available within a few hours to two days. ↩︎
This article from El Pais (in Spanish) ↩︎
↩︎
This is the commonly phrased version of principle. ↩︎
In Spain ↩︎
Refer to this ( -in Spanish) which focuses on the fact that coronavirus mortality rate differs enormously according to the age. ↩︎
Refer to (Publico, in Spanish) ↩︎
Take as an example the unfortunate words by Dutch finance minister Wopke Hoekstra, who suggested the EU “should investigate countries like Spain that say they have no budgetary margin to deal with the effects of the crisis provoked by the new coronavirus in spite of the fact that the eurozone has grown for seven consecutive years”, a statement that was later qualified as “repugnant” by Portugal’s Primer Minister, António Costa. (Source: ) ↩︎
Just to give you an example of how easy can be to implement a prediction in R, refer to this article: ↩︎
Also, reading this post may be useful: ↩︎

Manipulating dataframes in R and Python

Sat, 08 Feb 2020 17:46:34 +0000

While I have been using R for many years now (mainly for data manipulation and visualization), and I am extremely happy with some of its features (like how easy is to deal with data or to create interactive reports that can be exported in plenty of different outputs, such as pdf, documents, slides, dashboards or blog posts like this one). However, I have always wanted to learn python, mostly because it is a multi-purpose language that I would be able to use in other aspects of my everyday life such as web development, QGIS or Academic research. It is for that reason that I have recently started to learn python’s pandas.

In the following blog post I will be comparing how to perform the same tasks using pandas and tidyverse. This mainly serves two learning outcomes:

To generate a cheat sheet that can work as a reminder (for myself): I know there are pages like , but I learn by doing, so I needed to write the code myself)
To use reticulate package, which allows running both, R and python within the same document (a Rmarkdown file to be more specific)

Loading environment

Since I want to use Python and R from a .Rmarkdown file, I first need to load reticulate for this, which is a python interface for R. Also, since pandas is not a standard library module, I need to load a python environment¹ with the required packages. reticulate makes it possible to load environments created with anaconda.

The last step is to load some data about COVID-19 provided by coronavirus package, which I will be using in this blog post.

library(reticulate)
library(DT)
use_condaenv('osm_imports_preparations', required = TRUE)

Loading data

First thing we are doing to do is to read a CSV file and turn it into a dataframe which we are going to manipulate in the next steps.

R

# We will be loading a CSV file from RamiKrispin's coronavirus' package.
csv_url = "https://raw.githubusercontent.com/RamiKrispin/coronavirus/master/csv/coronavirus.csv"

# Read the CSV, convert it into a dataframe and store it in a variable.
r_df <- read.csv(csv_url)

# Explore the first on the dataframe.
head(r_df, 12)

## date province country lat long type cases
## 1 2020-01-22 Afghanistan 33.93911 67.70995 confirmed 0
## 2 2020-01-23 Afghanistan 33.93911 67.70995 confirmed 0
## 3 2020-01-24 Afghanistan 33.93911 67.70995 confirmed 0
## 4 2020-01-25 Afghanistan 33.93911 67.70995 confirmed 0
## 5 2020-01-26 Afghanistan 33.93911 67.70995 confirmed 0
## 6 2020-01-27 Afghanistan 33.93911 67.70995 confirmed 0
## 7 2020-01-28 Afghanistan 33.93911 67.70995 confirmed 0
## 8 2020-01-29 Afghanistan 33.93911 67.70995 confirmed 0
## 9 2020-01-30 Afghanistan 33.93911 67.70995 confirmed 0
## 10 2020-01-31 Afghanistan 33.93911 67.70995 confirmed 0
## 11 2020-02-01 Afghanistan 33.93911 67.70995 confirmed 0
## 12 2020-02-02 Afghanistan 33.93911 67.70995 confirmed 0

Python

import pandas as pd

# We will be loading a CSV file from RamiKrispin's coronavirus' package.
csv_url = "https://raw.githubusercontent.com/RamiKrispin/coronavirus/master/csv/coronavirus.csv"

# Read the CSV, convert it into a dataframe and store it in a variable.
py_df = pd.read_csv(csv_url)

# Explore the first elements on the dataframe.
py_df.head(12)

## date province country lat long type cases
## 0 2020-01-22 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 1 2020-01-23 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 2 2020-01-24 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 3 2020-01-25 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 4 2020-01-26 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 5 2020-01-27 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 6 2020-01-28 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 7 2020-01-29 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 8 2020-01-30 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 9 2020-01-31 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 10 2020-02-01 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 11 2020-02-02 NaN Afghanistan 33.93911 67.709953 confirmed 0

So far, there are no significant differences between both, but note the following differences:

R works with regular functions, whereas python uses methods instead.
In order to work with dataframes in python, pandas module has to be imported beforehand, whereas it is a base feature from R.
Both commands have read the same file but, apparently, they display different data. We will need to explore further the imported data to make sure that both are what we expected and, therefore, the same.

Exploring a dataframe

In this step we are going to evaluate what kind of object have we created, as well as a very basic data exploration.

R

# Explore what kind of object r_df is.
str(r_df)

## 'data.frame': 218276 obs. of 7 variables:
## $ date : chr "2020-01-22" "2020-01-23" "2020-01-24" "2020-01-25" ...
## $ province: chr "" "" "" "" ...
## $ country : chr "Afghanistan" "Afghanistan" "Afghanistan" "Afghanistan" ...
## $ lat : num 33.9 33.9 33.9 33.9 33.9 ...
## $ long : num 67.7 67.7 67.7 67.7 67.7 ...
## $ type : chr "confirmed" "confirmed" "confirmed" "confirmed" ...
## $ cases : int 0 0 0 0 0 0 0 0 0 0 ...

# Basic statistics
summary(r_df)

## date province country lat
## Length:218276 Length:218276 Length:218276 Min. :-51.796
## Class :character Class :character Class :character 1st Qu.: 6.428
## Mode :character Mode :character Mode :character Median : 22.041
## Mean : 20.561
## 3rd Qu.: 40.182
## Max. : 71.707
## long type cases
## Min. :-135.00 Length:218276 Min. :-16298.0
## 1st Qu.: -12.89 Class :character 1st Qu.: 0.0
## Median : 21.75 Mode :character Median : 0.0
## Mean : 25.01 Mean : 331.8
## 3rd Qu.: 84.25 3rd Qu.: 12.0
## Max. : 178.06 Max. :140050.0

# Explore unique values within a variable.
levels(as.factor(r_df$date))

## [1] "2020-01-22" "2020-01-23" "2020-01-24" "2020-01-25" "2020-01-26"
## [6] "2020-01-27" "2020-01-28" "2020-01-29" "2020-01-30" "2020-01-31"
## [11] "2020-02-01" "2020-02-02" "2020-02-03" "2020-02-04" "2020-02-05"
## [16] "2020-02-06" "2020-02-07" "2020-02-08" "2020-02-09" "2020-02-10"
## [21] "2020-02-11" "2020-02-12" "2020-02-13" "2020-02-14" "2020-02-15"
## [26] "2020-02-16" "2020-02-17" "2020-02-18" "2020-02-19" "2020-02-20"
## [31] "2020-02-21" "2020-02-22" "2020-02-23" "2020-02-24" "2020-02-25"
## [36] "2020-02-26" "2020-02-27" "2020-02-28" "2020-02-29" "2020-03-01"
## [41] "2020-03-02" "2020-03-03" "2020-03-04" "2020-03-05" "2020-03-06"
## [46] "2020-03-07" "2020-03-08" "2020-03-09" "2020-03-10" "2020-03-11"
## [51] "2020-03-12" "2020-03-13" "2020-03-14" "2020-03-15" "2020-03-16"
## [56] "2020-03-17" "2020-03-18" "2020-03-19" "2020-03-20" "2020-03-21"
## [61] "2020-03-22" "2020-03-23" "2020-03-24" "2020-03-25" "2020-03-26"
## [66] "2020-03-27" "2020-03-28" "2020-03-29" "2020-03-30" "2020-03-31"
## [71] "2020-04-01" "2020-04-02" "2020-04-03" "2020-04-04" "2020-04-05"
## [76] "2020-04-06" "2020-04-07" "2020-04-08" "2020-04-09" "2020-04-10"
## [81] "2020-04-11" "2020-04-12" "2020-04-13" "2020-04-14" "2020-04-15"
## [86] "2020-04-16" "2020-04-17" "2020-04-18" "2020-04-19" "2020-04-20"
## [91] "2020-04-21" "2020-04-22" "2020-04-23" "2020-04-24" "2020-04-25"
## [96] "2020-04-26" "2020-04-27" "2020-04-28" "2020-04-29" "2020-04-30"
## [101] "2020-05-01" "2020-05-02" "2020-05-03" "2020-05-04" "2020-05-05"
## [106] "2020-05-06" "2020-05-07" "2020-05-08" "2020-05-09" "2020-05-10"
## [111] "2020-05-11" "2020-05-12" "2020-05-13" "2020-05-14" "2020-05-15"
## [116] "2020-05-16" "2020-05-17" "2020-05-18" "2020-05-19" "2020-05-20"
## [121] "2020-05-21" "2020-05-22" "2020-05-23" "2020-05-24" "2020-05-25"
## [126] "2020-05-26" "2020-05-27" "2020-05-28" "2020-05-29" "2020-05-30"
## [131] "2020-05-31" "2020-06-01" "2020-06-02" "2020-06-03" "2020-06-04"
## [136] "2020-06-05" "2020-06-06" "2020-06-07" "2020-06-08" "2020-06-09"
## [141] "2020-06-10" "2020-06-11" "2020-06-12" "2020-06-13" "2020-06-14"
## [146] "2020-06-15" "2020-06-16" "2020-06-17" "2020-06-18" "2020-06-19"
## [151] "2020-06-20" "2020-06-21" "2020-06-22" "2020-06-23" "2020-06-24"
## [156] "2020-06-25" "2020-06-26" "2020-06-27" "2020-06-28" "2020-06-29"
## [161] "2020-06-30" "2020-07-01" "2020-07-02" "2020-07-03" "2020-07-04"
## [166] "2020-07-05" "2020-07-06" "2020-07-07" "2020-07-08" "2020-07-09"
## [171] "2020-07-10" "2020-07-11" "2020-07-12" "2020-07-13" "2020-07-14"
## [176] "2020-07-15" "2020-07-16" "2020-07-17" "2020-07-18" "2020-07-19"
## [181] "2020-07-20" "2020-07-21" "2020-07-22" "2020-07-23" "2020-07-24"
## [186] "2020-07-25" "2020-07-26" "2020-07-27" "2020-07-28" "2020-07-29"
## [191] "2020-07-30" "2020-07-31" "2020-08-01" "2020-08-02" "2020-08-03"
## [196] "2020-08-04" "2020-08-05" "2020-08-06" "2020-08-07" "2020-08-08"
## [201] "2020-08-09" "2020-08-10" "2020-08-11" "2020-08-12" "2020-08-13"
## [206] "2020-08-14" "2020-08-15" "2020-08-16" "2020-08-17" "2020-08-18"
## [211] "2020-08-19" "2020-08-20" "2020-08-21" "2020-08-22" "2020-08-23"
## [216] "2020-08-24" "2020-08-25" "2020-08-26" "2020-08-27" "2020-08-28"
## [221] "2020-08-29" "2020-08-30" "2020-08-31" "2020-09-01" "2020-09-02"
## [226] "2020-09-03" "2020-09-04" "2020-09-05" "2020-09-06" "2020-09-07"
## [231] "2020-09-08" "2020-09-09" "2020-09-10" "2020-09-11" "2020-09-12"
## [236] "2020-09-13" "2020-09-14" "2020-09-15" "2020-09-16" "2020-09-17"
## [241] "2020-09-18" "2020-09-19" "2020-09-20" "2020-09-21" "2020-09-22"
## [246] "2020-09-23" "2020-09-24" "2020-09-25" "2020-09-26" "2020-09-27"
## [251] "2020-09-28" "2020-09-29" "2020-09-30" "2020-10-01" "2020-10-02"
## [256] "2020-10-03" "2020-10-04" "2020-10-05" "2020-10-06" "2020-10-07"
## [261] "2020-10-08" "2020-10-09" "2020-10-10" "2020-10-11" "2020-10-12"
## [266] "2020-10-13" "2020-10-14" "2020-10-15" "2020-10-16" "2020-10-17"
## [271] "2020-10-18" "2020-10-19" "2020-10-20" "2020-10-21" "2020-10-22"
## [276] "2020-10-23" "2020-10-24"

Python

# Explore what kind of entity py_df is.
py_df.info()

# Now, calculate basic statistics for the numeric columns in the DataFrame.

## <class 'pandas.core.frame.DataFrame'>
## RangeIndex: 218276 entries, 0 to 218275
## Data columns (total 7 columns):
## # Column Non-Null Count Dtype
## --- ------ -------------- -----
## 0 date 218276 non-null object
## 1 province 63433 non-null object
## 2 country 218276 non-null object
## 3 lat 218276 non-null float64
## 4 long 218276 non-null float64
## 5 type 218276 non-null object
## 6 cases 218276 non-null int64
## dtypes: float64(2), int64(1), object(4)
## memory usage: 11.7+ MB

py_df.describe()

# List unique values in the df['date'] column

## lat long cases
## count 218276.000000 218276.000000 218276.000000
## mean 20.561062 25.011408 331.749812
## std 24.759252 69.572388 3045.547043
## min -51.796300 -135.000000 -16298.000000
## 25% 6.428055 -12.885800 0.000000
## 50% 22.041450 21.745300 0.000000
## 75% 40.182400 84.250000 12.000000
## max 71.706900 178.065000 140050.000000

py_df['date'].unique() # I prefer using this notation to prevent problems with columns with a dot inside.

## array(['2020-01-22', '2020-01-23', '2020-01-24', '2020-01-25',
## '2020-01-26', '2020-01-27', '2020-01-28', '2020-01-29',
## '2020-01-30', '2020-01-31', '2020-02-01', '2020-02-02',
## '2020-02-03', '2020-02-04', '2020-02-05', '2020-02-06',
## '2020-02-07', '2020-02-08', '2020-02-09', '2020-02-10',
## '2020-02-11', '2020-02-12', '2020-02-13', '2020-02-14',
## '2020-02-15', '2020-02-16', '2020-02-17', '2020-02-18',
## '2020-02-19', '2020-02-20', '2020-02-21', '2020-02-22',
## '2020-02-23', '2020-02-24', '2020-02-25', '2020-02-26',
## '2020-02-27', '2020-02-28', '2020-02-29', '2020-03-01',
## '2020-03-02', '2020-03-03', '2020-03-04', '2020-03-05',
## '2020-03-06', '2020-03-07', '2020-03-08', '2020-03-09',
## '2020-03-10', '2020-03-11', '2020-03-12', '2020-03-13',
## '2020-03-14', '2020-03-15', '2020-03-16', '2020-03-17',
## '2020-03-18', '2020-03-19', '2020-03-20', '2020-03-21',
## '2020-03-22', '2020-03-23', '2020-03-24', '2020-03-25',
## '2020-03-26', '2020-03-27', '2020-03-28', '2020-03-29',
## '2020-03-30', '2020-03-31', '2020-04-01', '2020-04-02',
## '2020-04-03', '2020-04-04', '2020-04-05', '2020-04-06',
## '2020-04-07', '2020-04-08', '2020-04-09', '2020-04-10',
## '2020-04-11', '2020-04-12', '2020-04-13', '2020-04-14',
## '2020-04-15', '2020-04-16', '2020-04-17', '2020-04-18',
## '2020-04-19', '2020-04-20', '2020-04-21', '2020-04-22',
## '2020-04-23', '2020-04-24', '2020-04-25', '2020-04-26',
## '2020-04-27', '2020-04-28', '2020-04-29', '2020-04-30',
## '2020-05-01', '2020-05-02', '2020-05-03', '2020-05-04',
## '2020-05-05', '2020-05-06', '2020-05-07', '2020-05-08',
## '2020-05-09', '2020-05-10', '2020-05-11', '2020-05-12',
## '2020-05-13', '2020-05-14', '2020-05-15', '2020-05-16',
## '2020-05-17', '2020-05-18', '2020-05-19', '2020-05-20',
## '2020-05-21', '2020-05-22', '2020-05-23', '2020-05-24',
## '2020-05-25', '2020-05-26', '2020-05-27', '2020-05-28',
## '2020-05-29', '2020-05-30', '2020-05-31', '2020-06-01',
## '2020-06-02', '2020-06-03', '2020-06-04', '2020-06-05',
## '2020-06-06', '2020-06-07', '2020-06-08', '2020-06-09',
## '2020-06-10', '2020-06-11', '2020-06-12', '2020-06-13',
## '2020-06-14', '2020-06-15', '2020-06-16', '2020-06-17',
## '2020-06-18', '2020-06-19', '2020-06-20', '2020-06-21',
## '2020-06-22', '2020-06-23', '2020-06-24', '2020-06-25',
## '2020-06-26', '2020-06-27', '2020-06-28', '2020-06-29',
## '2020-06-30', '2020-07-01', '2020-07-02', '2020-07-03',
## '2020-07-04', '2020-07-05', '2020-07-06', '2020-07-07',
## '2020-07-08', '2020-07-09', '2020-07-10', '2020-07-11',
## '2020-07-12', '2020-07-13', '2020-07-14', '2020-07-15',
## '2020-07-16', '2020-07-17', '2020-07-18', '2020-07-19',
## '2020-07-20', '2020-07-21', '2020-07-22', '2020-07-23',
## '2020-07-24', '2020-07-25', '2020-07-26', '2020-07-27',
## '2020-07-28', '2020-07-29', '2020-07-30', '2020-07-31',
## '2020-08-01', '2020-08-02', '2020-08-03', '2020-08-04',
## '2020-08-05', '2020-08-06', '2020-08-07', '2020-08-08',
## '2020-08-09', '2020-08-10', '2020-08-11', '2020-08-12',
## '2020-08-13', '2020-08-14', '2020-08-15', '2020-08-16',
## '2020-08-17', '2020-08-18', '2020-08-19', '2020-08-20',
## '2020-08-21', '2020-08-22', '2020-08-23', '2020-08-24',
## '2020-08-25', '2020-08-26', '2020-08-27', '2020-08-28',
## '2020-08-29', '2020-08-30', '2020-08-31', '2020-09-01',
## '2020-09-02', '2020-09-03', '2020-09-04', '2020-09-05',
## '2020-09-06', '2020-09-07', '2020-09-08', '2020-09-09',
## '2020-09-10', '2020-09-11', '2020-09-12', '2020-09-13',
## '2020-09-14', '2020-09-15', '2020-09-16', '2020-09-17',
## '2020-09-18', '2020-09-19', '2020-09-20', '2020-09-21',
## '2020-09-22', '2020-09-23', '2020-09-24', '2020-09-25',
## '2020-09-26', '2020-09-27', '2020-09-28', '2020-09-29',
## '2020-09-30', '2020-10-01', '2020-10-02', '2020-10-03',
## '2020-10-04', '2020-10-05', '2020-10-06', '2020-10-07',
## '2020-10-08', '2020-10-09', '2020-10-10', '2020-10-11',
## '2020-10-12', '2020-10-13', '2020-10-14', '2020-10-15',
## '2020-10-16', '2020-10-17', '2020-10-18', '2020-10-19',
## '2020-10-20', '2020-10-21', '2020-10-22', '2020-10-23',
## '2020-10-24'], dtype=object)

There are no significant differences, nor in the syntax nor in the ouput.

Note:

We prefer to use this notation in order to prevent problems with columns with a dot inside.

Sorting dataframe

library(tidyverse)

## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──

## ✓ ggplot2 3.3.2 ✓ purrr 0.3.4
## ✓ tibble 3.0.4 ✓ dplyr 1.0.2
## ✓ tidyr 1.1.2 ✓ stringr 1.4.0
## ✓ readr 1.3.1 ✓ forcats 0.5.0

## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag() masks stats::lag()

head(arrange(r_df, country))

## date province country lat long type cases
## 1 2020-01-22 Afghanistan 33.93911 67.70995 confirmed 0
## 2 2020-01-23 Afghanistan 33.93911 67.70995 confirmed 0
## 3 2020-01-24 Afghanistan 33.93911 67.70995 confirmed 0
## 4 2020-01-25 Afghanistan 33.93911 67.70995 confirmed 0
## 5 2020-01-26 Afghanistan 33.93911 67.70995 confirmed 0
## 6 2020-01-27 Afghanistan 33.93911 67.70995 confirmed 0

py_df.sort_values(['country']).head()

## date province country lat long type cases
## 0 2020-01-22 NaN Afghanistan 33.93911 67.709953 confirmed 0
## 147922 2020-01-26 NaN Afghanistan 33.93911 67.709953 recovered 0
## 147921 2020-01-25 NaN Afghanistan 33.93911 67.709953 recovered 0
## 147920 2020-01-24 NaN Afghanistan 33.93911 67.709953 recovered 0
## 147919 2020-01-23 NaN Afghanistan 33.93911 67.709953 recovered 0

Select and summarise

Let’s pretend that we want to have a table displaying the top 10 countries with the most number of confirmed cases until today (2020-10-25).

R

r_df %>%
 # select(Country.Region, cases, type) %>% 
 filter(type == "confirmed") %>%
 group_by(country) %>%
 summarise(total = sum(cases)) %>%
 arrange(desc(total)) %>%
 head(5)

## # A tibble: 5 x 2
## country total
## <chr> <int>
## 1 US 8575177
## 2 India 7814682
## 3 Brazil 5380635
## 4 Russia 1487260
## 5 France 1084659

# Or, even more succintly:
r_df %>%
 filter(type == "confirmed") %>%
 count(country, wt = cases, sort = TRUE, name = "total") %>%
 head(5)

## country total
## 1 US 8575177
## 2 India 7814682
## 3 Brazil 5380635
## 4 Russia 1487260
## 5 France 1084659

Python

py_df.loc[py_df['type'] == 'confirmed'].groupby('country').agg(
 {'cases':'sum'}
).rename(columns={'cases': 'total'}).sort_values(
 ['total'], ascending = False).head(5)

## total
## country
## US 8575177
## India 7814682
## Brazil 5380635
## Russia 1487260
## France 1084659

This is something funny. python takes pride in stating that is an elegant and easy to read syntax due to its strict indentation syntax. On the other hand, I have met several python proponents mocking about R’s code to be quite obscure and difficult to memorise. While I often share the same views (especially when dealing with base R’s syntax), I particularly find tidypverse’s syntax far easier to read and memorise than pandas’ . While the first makes use of the pipe operator ( %>%) to chain commands while preventing typing unnecessary data, the latter requires to concatenate up to six different methods in a single line, which becomes too long to read (and thus, not liked very much by )

Joins and calculations

But that’s not fair, we are comparing countries with very different number of population! If we are to compare them, we need to use relative values. For example, we would need to create a ranking based on the total number of cases per 1000 habitants.

In order to do so, we will need to do the following steps:

Load a dataframe with population data per each country
Add the population data by join our existing dataframes with the newly created one in the previous step (left join)
Calculate relative number of confirmed cases like this: \(confirmed~rel = \frac{confirmed}{population}\)

R

# Load population data.
countries19 <- read.csv("data/countries_pop19.csv")

r_df %>%
 filter(type == "confirmed") %>%
 count(country, wt = cases, sort = TRUE, name = "confirmed") %>%
 # Add population column.
 left_join(countries19, by = c("country" = "Location")) %>%
 # Calculate relative cases.
 mutate(confirmed_rel = confirmed / PopTotal) %>%
 # Select certain columns only.
 select(country, confirmed, confirmed_rel) %>%
 # Sort by confirmed_rel on descending order.
 arrange(desc(confirmed_rel)) %>%
 # Display everything on a nice datatable.
 head(10)

## country confirmed confirmed_rel
## 1 Andorra 4038 52.34231
## 2 Bahrain 79975 48.73066
## 3 Qatar 130965 46.24354
## 4 Israel 309413 36.31875
## 5 Holy See 27 33.12883
## 6 Panama 128515 30.26417
## 7 Kuwait 120927 28.74371
## 8 Peru 883116 27.16406
## 9 Montenegro 16629 26.47981
## 10 Belgium 305409 26.46680

Python


# Load population data.
countries19 = pd.read_csv('data/countries_pop19.csv')

py_confirmed_rel = py_df.loc[py_df['type'] == 'confirmed'].groupby('country').agg(
 {'cases':'sum'}
).rename(columns={'cases': 'confirmed'})

# Join population information.
py_confirmed_rel = py_confirmed_rel.join(countries19.set_index('Location'), on = 'country')

# Calculate relative confirmed cases.
py_confirmed_rel = py_confirmed_rel.assign(confirmed_rel = py_confirmed_rel['confirmed']/py_confirmed_rel['PopTotal'])

py_confirmed_rel = py_confirmed_rel[['confirmed', 'confirmed_rel']]

py_confirmed_rel.sort_values(['confirmed_rel'], ascending = False).head(10)

## confirmed confirmed_rel
## country
## Andorra 4038 52.342312
## Bahrain 79975 48.730657
## Qatar 130965 46.243544
## Israel 309413 36.318753
## Holy See 27 33.128834
## Panama 128515 30.264174
## Kuwait 120927 28.743710
## Peru 883116 27.164056
## Montenegro 16629 26.479805
## Belgium 305409 26.466797

Conclusion

Before concluding, I have to admit that I have been using tidyverse for several years, and I am very used to its syntax. Therefore, it is no wonder that I feel much more comfortable with it than with pandas’ . Being said that, I find the latter to be quite straightforward and relatively easy to use and memorise (I will need to check this post and for reference). However, I have two main concerns about pandas:

Compared to R, which is more succinct, pandas requires to type many times the data frame’s name. This makes it more prone-error and slow to type, but also more difficult to read. If you want to avoid typing it as much, you need to chain a number of methods that result in very long lines, which makes it difficult to comment and it is, again, more difficult to read.
I find it somewhat overwhelming that there are many ways to perform same task in pandas. I believe Ted Petrou’s advice on learning the is a good advice, as it makes things simpler.
I miss the magritte’s pipe operator, but I guess I should change my mindset when using python.

Regarding reticulate: I think it is very promising. While jupyter notebooks can be used with different kernels such as Julia, python or R (hence its name), there is no way to combine different kernels in the same notebook, at least that I am aware of. This means that only one language per notebook can be used. On the other hand, reticulate allows you to use different languages within the same document (a markdown file, which I prefer it over jupyter notebooks, by the way). Admittedly, I do not know if that is a common scenario, but it has proven to be very useful for a post like this one.

Being said that, I admit that I expected that I could use one variable from python and use it in R, if that makes any sense at all. However, that’s not possible, as both languages are isolated, which I assume is the logical way (I assume is not straightforward at all to convert from one python list to an R vector, for example).

If there is something that I have fallen in love with python so far is how convenient and easy are to create virtual environments from a simple yaml file listing all the dependencies. This is something that would be very useful in R, too and I should explore in the future: I know packrat is there for this purpose, but it is not as fast and easy to deal with as conda environments. On the other hand, if I am not mistaken, conda also has R libraries, so it may be possible to create conda environments for R, too. ↩︎