Here’s an unusual way of representing population growth. Pedro M Cruz, from Northeastern University in Boston, takes two centuries of US census data and shows the increasing population as rings of a tree, one for each decade.
For a radical new perspective on immigration, picture the US as an ancient tree
According to Cruz, the tree metaphor ‘carries the idea that these marks in the past are immutable’ and it ‘embodies the concept that all cells contributed to the organism’s growth’. As with so many renderings of US history, indigenous populations are conspicuously absent from the tableau. Still, Cruz’s skilfully deployed data doubles as a resonant work of cultural commentary, offering a rich and often surprising look at the ever-evolving makeup of the country.
There’s more information on the video’s Vimeo page.
Simulated dendrochronology of U.S. immigration (1830-2015)
Trees in their natural setting have annual growth rings that reflect varying environmental conditions; the rings’ forms are neither perfect circles nor ellipses. The algorithm is inspired by this variation and accordingly deposits immigrant cells in specific directions depending on the geographic origin of the immigrant. Rings that are more skewed toward the country’s East, for example, show more immigration from Europe, while rings skewed South show more immigration from Latin America. With this, it is possible to observe the quantity of immigration through the thickness of the rings. The color of the cells corresponds to specific cultural-geographical regions.
This is a little alarming.
Anonymised data isn’t nearly anonymous enough – here’s how we fix it
We developed a machine learning model to assess the likelihood of reidentifying the right person. We took datasets and we showed that in the US fifteen characteristics, including age, gender, marital status and others, are sufficient to reidentify 99.98 per cent of Americans in virtually any anonymised data set.
Some more examples.
The simple process of re-identifying patients in public health records
In late 2016, doctors’ identities were decrypted in an open dataset of Australian medical billing records. Now patients’ records have also been re-identified – and we should be talking about it.
‘Anonymous’ browsing data can be easily exposed, researchers reveal
A journalist and a data scientist secured data from three million users easily by creating a fake marketing company, and were able to de-anonymise many users.
“What would you think,” asked Svea Eckert, “if somebody showed up at your door saying: ‘Hey, I have your complete browsing history – every day, every hour, every minute, every click you did on the web for the last month’? How would you think we got it: some shady hacker? No. It was much easier: you can just buy it.”
University application processes are in full swing, but here is some reputationally damaging news from Lancaster University.
Lancaster University hit by cyber attack, hundreds of students’ personal data stolen
The full scale of the cyber attack was revealed yesterday (July 22), when university chiefs confirmed that hackers had breached IT systems and accessed student records. […] It said it regretted that the breach has led to fraudulent invoices being sent to some undergraduate applicants demanding large sums of money.
Two days later, and the police have arrested someone for it.
Man arrested over UK’s Lancaster University data breach hack allegations
Names, addresses, email addresses and phone numbers were among the categories of data visible to the hackers. Fraudulent invoices were sent to some, the university admitted. With overseas applicants (of which Lancaster had 575 last year from non-EU countries and 375 from other EU countries) paying fees measured in the tens of thousands of pounds per year, the potential for high returns is great.
Our sources added that around half a dozen students had paid these fraudulent invoices. The highest undergraduate fees for overseas (non-EU) students is Lancaster’s Bachelor of Medicine, Bachelor of Surgery (MBChB) course at £31,540.
It’s more than a little embarrassing, as Lancaster University is one of a number of universities offering degrees in cyber security…
Cyber Security MSc – Lancaster University
In addition to the taught modules, you will also work on an individual research project, supervised by two academics from two of the four departments. Through this project, you will obtain an in-depth understanding of the theoretical and practical aspects of cyber security and technology. You will put the skills and knowledge you have developed throughout the year into practice and gain experience of tackling real-world cyber security issues.
Well, there’s a ‘real-world cyber security issue’ for you.
An introduction to what promises to be a fascinating new blog from Anna Powell-Smith, “about the data that the government should collect and measure in the UK, but doesn’t.”
Across lots of different policy areas, it was impossible for governments to make good decisions because of a basic lack of data. There was always critical data that the state either didn’t collect at all, or collected so badly that it made change impossible.
Eventually, I decided that the power to not collect data is one of the most important and little-understood sources of power that governments have. This is why I’m writing Missing Numbers: to encourage others to ask “is this lack of data a deliberate ploy to get away with something”?
By refusing to amass knowledge in the first place, decision-makers exert power over over the rest of us. It’s time that this power was revealed, so we can have better conversations about what we need to know to run this country successfully.
I sit and stare at Excel for a significant proportion of my day. I can’t believe I’ve not been aware of this simple trick with copying formulas without messing up cell references. It’s saving me an immense amount of time.
Copy Excel formula without changing cell references (or without file references)
It’s quite simple actually!
- Highlight the are you’d like to copy
- Go to Home / Find & Select / Replace (or press Ctrl + H)
- Search for = and replace with a text that’s not in your file – in this example I chose “notinfile” (note as mentioned in the comments in YouTube, you can also replace with ” =”, i.e. a space before the equal sign)
- Go back to Home / Find & Select / Replace (or press Ctrl + H) – search for your text – in my example “notinfile” and replace with =.
- That’s it!
Here are a few more tips and tricks.
10 easy Excel timesavers you might have forgotten
Microsoft has packed Excel with all kinds of different ways to get things done quicker. However, you can’t take advantage of these features if you don’t know about them. These ten techniques may only save you a few seconds every time you use them. That might not sound like much, but if you can integrate them into your workflow, you’re sure to reap the benefits over time.
Via FlowingData, here’s a witty visualisation of how we spend our days, on average. It’s just a stacked bar chart, but turning it into a comic “can allow the audience to identify with the story, sparking self-reflection: “Is this how I live my life? How am I different?””
A day in the life of Americans: a data comic
There are three settings in this comic (a bedroom, an office, and a bar), each serving as a metonym for an activity (sleep, work, and leisure). I have also included colors and positions as redundant, but clarifying, codes of classification. Such scenes allow for a novel method of highlighting data; a setting inside a panel is “lit up” by a light source if the activity for which it stands occupied those two hours of Americans the most.
The Economist’s charts are usually very clear and helpful, but that’s not to say they can’t be improved – as they themselves show.
Mistakes, we’ve drawn a few
At The Economist, we take data visualisation seriously. Every week we publish around 40 charts across print, the website and our apps. With every single one, we try our best to visualise the numbers accurately and in a way that best supports the story. But sometimes we get it wrong. We can do better in future if we learn from our mistakes — and other people may be able to learn from them, too. […]
Let’s start with the worst of crimes in data visualisation: presenting data in a misleading way. We never do this on purpose! But it does happen every now and then. Let’s look at the three examples from our archive.
Mistake: Truncating the scale
Hot on the heels of Facebook’s latest password problem, TechCrunch has news of another online service with a very shoddy approach to data protection – i.e. there wasn’t any.
The app, Family Locator, allows families to track each other’s movements, similar to the location sharing option in Google Maps. But it seems the backend database for their nearly a quarter of a million users wasn’t protected at all.
A family tracking app was leaking real-time location data
Based on a review of the database, each account record contained a user’s name, email address, profile photo and their plaintext passwords. Each account also kept a record of their own and other family members’ real-time locations precise to just a few feet. Any user who had a geofence set up also had those coordinates stored in the database, along with what the user called them — such as “home” or “work.”
They tried to get in touch with the developer, React Apps, but to no avail.
On Friday, we asked Microsoft, which hosted the database on its Azure cloud, to contact the developer. Hours later, the database was finally pulled offline.
In an attempt to get rid of the sour taste left in our mouths from yesterday’s post about the rise of populist politics, here are some more award-winning data visualisations via David McCandless and the Information is Beautiful people.
The winners of the World Data Visualization Prize
Conducted in partnership with the World Government Summit, the prize focuses on how governments are improving citizens’ lives. We asked entrants to use the power of data-visualization to illuminate data on the innovations and decisions – seen and unseen – that drive progress.
Here’s my favourite, an interactive overview of the different factors that contribute to happy countries (or not).
GOV|DNA — Discover the DNA of a good government
This interactive visualization enables the exploration of the DNA of a good government. You can analyze and compare multiple indicators to investigate their influence on countries and the related behaviour and performance of governments.
Each six months Andy Kirk of Visualising Data highlights some of the significant developments in data visualisation. It’s a great collection, but this one in particular caught my eye.
10 significant visualisation developments: July to December 2018
2. ‘Human Terrain’: A genuinely captivating project from Matt Daniels of ThePudding, ‘Human Terrain’ is a staggeringly detailed, explorable prism map of the world’s population that can trap you into browsing for far longer than you can realistically afford. It evokes memories of a classic graphic from 2006, created by Joe Lertola for Time magazine. There is also a wonderful companion piece, ‘Population Mountains‘, where Matt walks through ‘a story about how to perceive the population of cities’.
When you fly from one part of the world to another, it becomes very quickly apparent just how crowded some places must be, compared to others.
Human Terrain: visualizing the world’s population, in 3D
Kinshasa is now bigger than Paris. Guangzhou, Hong Kong, and Shenzhen are forming an epic, 40 million-person super city. Over the past 30 years, the scale of population change is hard to grasp. How do you even visualize 10 million people?
It puts those incredibly dense housing schemes in Hong Kong I mentioned earlier into context, doesn’t it?
Population growth, like charity, starts in the home, so here’s an animated chart on family sizes in the US.
How many kids we have and when we have them
The chart above shows 1,000 timelines, based on data from the National Survey of Family Growth. Each moving dot is a mother. Age is on the horizontal, and with each live birth, the dot moves down a notch. The green bubbles represent the total counts for a given age.
It’s interesting to watch the chart populate. You’ve got to wonder about the stories behind those outliers though.
One of the dangers at just looking at the numbers.
Progress 8 scores for most schools aren’t that different
There were over 300 schools with P8 scores between -0.05 and +0.05 – a difference of over 300 rank places (10% of schools) between the highest and lowest scoring of them. But what do these numbers mean?
Let’s say the score for School A was +0.05 and School B was -0.05. Taking the numbers at face value, one interpretation is that if you picked two pupils with the same KS2 attainment, the two pupils would have the same grades in seven of the subjects included in Attainment 8 but the pupil from School A would have one grade higher in one and only one subject than the pupil in School B.
Is this an educationally important difference?
And talking of Progress 8 confidence intervals…
xkcd: Error bars
Postcards are such simple things, really – just small rectangular pieces of thin card. No technology required. Perhaps that’s why they don’t seem as popular these days? But thanks to the Postcrossing project, I’ve been sending and receiving more and more — and from all over the world.
The Postcrossing project was created in 2005 by Paulo Magalhães as a side project when he was a student in Portugal. Paulo loves to receive mail and postcards in particular; from friends, family — or from anyone in the world. Finding a postcard in the mailbox always makes his day!
He knew more people shared the same interest, but there was no good way yet to connect everyone. And that’s how he got the initial idea of creating the online platform for this which he called Postcrossing. Its goal: to connect people across the world through postcards, independently of their country, age, gender, race or beliefs.
Here are some more postcard-related links.
Wish you were here? Postcards from the art world
“It’s possible to form a significant collection of extremely good and important works of art without being wealthy,” he says. “Anyone could decide to form a collection very close to mine with most of the same things – and I like that. It’s anti-exclusive.”
A postcard writing Rube Goldberg machine in a suitcase
As the sun regrettably sets on the art of letter writing, the inventive folks at design studio HEYHEYHEY have pieced together a clever contraption that promises to keep the art of travel postcards a thing of the present. Kind of. Melvin the Traveling Mini Machine is an elaborate Rube Goldberg machine that fits in a pair of suitcases that executes the simple task of “writing” and stamping a postcard of your choice, that is, if the absurdly elaborate sequence of steps goes off without a hitch.
Two women leading parallel lives are getting to know each other through data
Giorgia Lupi, who lives in New York, and Stefanie Posavec, who lives in London, are engaged in a long-distance, postcard-based data exchange in order to get to know each other better: “Dear Data.” They’ve only met in person twice, and they’re both interested in data, so they’re sending each other postcard drawings of data about their day-to-day lives.
Four Corners Books announces its next publication
Four Corners Books has just announced the next book in its Irregulars series titled, Leeds Postcards, a celebration of the independent postcard press. For the past four decades, independent postcard press Leeds Postcards has been making oppositional, inspiring images; activism by design. The cards are not of Leeds; the name represents a defiant rejection of the hegemony of London. The images cover a fascinating range of domestic and international politics, causes and campaigns, creating, in their own unique and graphically inventive way a record of the struggles as well as the progressive political triumphs from 1979 to the present day.
Some recent data protection stories that have caught my eye.
French data watchdog dishes out largest GDPR fine yet: Google ordered to hand over €50m
The French agency, CNIL, ruled today that the search giant had offered users inadequate information, spreading it across multiple pages, and had failed to gain valid consent for ads personalisation. […] The CNIL concluded that Google had breached the General Data Protection Regulation in two ways: by failing to meet transparency and information requirements, and failing to obtain a legal basis for processing.
Amazon, Apple and Google face data complaints
General Data Protection Regulation (GDPR) rules say EU customers have the right to access a copy of the personal data companies hold about them. However, privacy group noyb said it found that most of the big streaming companies did not fully comply. It has filed formal complaints, which if upheld could result in large fines.
Google accused of GDPR privacy violations by seven countries
Consumer groups across seven European countries have filed GDPR complaints against Google’s location tracking (via Reuters). The European Consumer Organisation (BEUC), of which each of the groups are a member, claims that Google’s “deceptive practices” around location tracking don’t give users a real choice about whether to enable it, and that Google doesn’t properly inform them about what this tracking entails. If upheld, the complaints could mean a hefty fine for the search giant.
The NOYB organisation gets mentioned a number of times there.
Max Schrems: The privacy bubble needs to start ‘getting sh*t done’
After years locked in numerous long, drawn-out and often bitter legal battles, Schrems decided to launch a nonprofit aiming to help people bring their own consumer privacy cases to court.
The plan is for NOYB (None Of Your Business) to take advantage of the incoming European Union General Data Protection Regulation, which offers more options for collective redress across the bloc, and harness the momentum Schrems has built up with various high-profile court cases.
Seems to be working. (Via)
Taking a break from their AI coverage, Artnome take us on an interesting journey through Van Gogh’s shifting colour schemes, busting a couple of myths as they go.
New data shows why Van Gogh changed his color palette
We were not convinced by the medical reasoning behind the shift in Van Gogh’s color palette and we could not think of any French Impressionists that painted with colors nearly as bold as Van Gogh, so we decided to take a look at some other possibilities.
Van Gogh was a restless soul and moved around quite a bit. He also spent a lot of time painting outdoors, especially in his later years. As someone who famously struggled with mood swings, we thought location, and more importantly, weather patterns may have impacted his use of color.
To test this, we created composite images averaging every painting Van Gogh created from each of the major locations he worked from and compared them to weather patterns from those regions. We think the results are quite remarkable.
Using data and technology like this is a fascinating way to learn more about works of art and the hidden narratives behind them.
I certainly enjoy reading about these voice assistants more than I do using them.
You bought smart speakers over the holidays. Now what are Amazon and Google doing with your data?
Ultimately, the choice to keep a smart speaker around comes down to what you’re getting out of the product. For some people with physical disabilities or intellectual differences, smart speakers can make household tasks easier or provide an engaging presence in daily life. For tech junkies like my friend, the sheer joy of commanding a smart home network might be enough. For Hoffman-Andrews, though, the benefits of a speaker don’t outweigh the costs. He bought a couple of products for testing, but he admits he couldn’t actually bring himself to set them up. Being able to ask a speaker to dim the lights or play a weather forecast just didn’t seem like a good enough tradeoff for giving companies access to his home.
“Is it normal to have cameras and microphones pointed at you and your guests? Currently the answer is mostly no,” he says. “These devices aim to change the answer to yes.”
Another end of year roundup, this time looking at data visualisation design.
Information is Beautiful Awards 2018: The Winners
Let’s raise a glass to dataviz that pushes boundaries, illuminates truth, and celebrates beauty. Thank you to everyone who joined us on the Information is Beautiful Awards journey this year – now see which entries took home trophies at tonight’s spectacular ceremony.
There is so much to pour over, here. Two that stood out particularly for me was this visual representation of a Beethoven string quartet and this unusual view of our lively planet.
Dynamic Planet Interactive Scientific Poster
The Interactive Scientific Poster „Dynamic Planet” was designed and developed for the exhibition „Focus Earth” of the GFZ German Research Centre for Geosciences in Potsdam. One of the main advantages of a digital poster is that it can display dynamic content. This is at the same time the essential statement of the scientific poster “Dynamic Planet”: our earth never stands still, is permanently shaken by earthquakes. These tensions are measured by three measuring points of the GFZ and their data is visualized in real-time in an interactive poster in the exhibition context. The viewer is given a direct impression, he can be a “witness” to current measurement and research.
This video demonstrates how people can interact with the poster, to navigate the large amount of data presented in an intuitive, visual manner.
Dynamic Planet – Scientific Poster
The challenge for the interface design was to ensure a clear overview, despite of the massively many events. The solution consists in an interactive graphical representation of the events by filtering the earthquakes by eg. magnitude and depth. A special visual feature of the scientific poster “Dynamic Planet” is the representation of the earthquakes depth in a transparent, rotating globe.
The economist and dataviz blogger Jonathan Schwabish took on an unusual challenge, to introduce his son’s primary school classmates to data visualisation.
I wouldn’t know where to start — I’m still not sure of the difference between a histogram and a bar chart — but cleverly, Jonathan begins with examples of diagrams everyone is familiar with. Maps.
Teaching data visualization to kids
I then introduced the term “choropleth” and showed them this map of graveyards in the US and this map of McDonald’s (a couple of kids actually tied the two together!). I also showed them a clip of Aron Koblins’ Flight Patterns project (my son loves this one)—the simple and intuitive animation, and black and white color scheme make it easy to follow. I also showed them a video of Martin Wattenberg and Fernanda Viegas’ Wind Map, again, something I think they could all relate to.
He then asks the children to draw their own maps, of their homes rather than the whole world, and to add in any data they liked.
I then passed out tracing paper and, bringing up the graphs I showed them earlier in which color, dots, lines, and bubbles were placed on top of the map, I asked them to plot any data they liked. … Could they add differently-sized bubbles to their favorite rooms? Could they draw lines showing their paths through the house? What about smiley faces for the most fun room?
What a fantastic idea. I hope others are similarly encouraged to spread the word in this way. As he says in his conclusion, helping children to understand graphs is a good thing for many reasons.
I’d love to see a way to make data visualization education a broader part of the curriculum, both on its own and linked with their math and other classes. Imagine adding different shapes to maps in their Social Studies classes to encode data or using waterfall charts in their math classes to visually demonstrate a simple mathematical equation or developing simple network diagrams in science class. The combination of the scientific approach to data visualization and the creativity it sparks could serve as a great way to help students learn.
More announcements of company data (our data) being stolen. The numbers involved each time are just incredible.
Hackers breach Quora.com and steal password data for 100 million users
Compromised information includes cryptographically protected passwords, full names, email addresses, data imported from linked networks, and a variety of non-public content and actions, including direct messages, answer requests and downvotes. […] In a post published late Monday afternoon, Quora officials said they discovered the unauthorized access on Friday. They have since hired a digital forensics and security firm to investigate and have also reported the breach to law enforcement officials.
Whenever these stories are reported, the articles often end with a little summary of other recent snafus. The one above ended with:
Quora’s post is only the latest disclosure of a major breach. On Friday, hotel chain Marriott International said a system breach allowed hackers to steal passport numbers, credit card data, and other details for 500 million customers. In September, Facebook reported an attack on its network allowed hackers to steal personal details for as many as 50 million users. The social network later lowered the number of accounts affected to about 30 million.
A post from The Register, about that massive Marriott breach, concluded with this reminder of previous losses.
Marriott’s Starwood hotels mega-hack: Half a BILLION guests’ deets exposed over 4 years
Few hacks of individual firm’s customer data have come close to the scale of this one. The Yahoo! breach in 2013 saw three billion email accounts breached, while Carphone Dixons, the UK electronics retail chain, managed to lose control of 5.9 million sets of payment card data. In the US, the US Government Office for Personnel Management (which handles sensitive files on millions of government workers) had the personal data of 21 million employees’ breached by hackers.
Following on from the Ofsted Chief Inspector’s comments about teachers being ‘reduced to data managers’, the Education Secretary has written to all schools this week, reiterating his commitment to “clamp down on teachers’ workload.” Here’s the DfE’s press release.
DfE: More support for school leaders to tackle workload
Secretary of State for Education Damian Hinds said: “Many teachers are having to work way too many hours each week on unnecessary tasks, including excessive time spent on marking and data analysis. I want to make sure teachers are teaching, not putting data into spreadsheets. That’s why I am stopping my department asking for data other than in the school’s existing format.
“I am united with the unions and Ofsted in wanting teachers to do less admin. I have a straightforward message to head teachers who want their staff to cut right down on collecting data to be able to devote energies to teaching: I will support you. Frequent data drops and excessive monitoring of a child’s progress are not required either by Ofsted or by the DfE.”
Here’s a link to the Teacher Workload Advisory Group report and government response.
It will be interesting to see how this plays out and what, if anything, changes. Perhaps less data collection, rather than less data analysis: I can see schools making judgements about students’ progress two or three times a year, instead of three or four, but will they really stop analysing progress by pupil premium, SEN, ethnicity, gender and so on and so on? Perhaps this is aimed at classroom teachers, rather than subject leaders, data managers and SLT data leads.
This looks to be an interesting response to the call to be more data literate. Via Flowing Data, a straightforward and potentially free way to get skilled up with R, without needing to install any software, it seems.
Chromebook Data Science – a free online data science program for anyone with a web browser
The reason they are called Chromebook Data Science is because philosophically our goal was that anyone with a Chromebook could do the courses. All you need is a web browser and an internet connection. The courses all take advantage of RStudio Cloud so that all course work can be completed entirely in a web browser. No need to install software or have the latest MacBook Computer.
Here’s some info on what the courses cover, including introductions to R and GitHub. Worth a look?