Data disasters

Check out this interactive ‘balloon race’ data visualisation from Information Is Beautiful, of all the major data breaches from the last ten years. Billions of records.

You can choose to highlight the items by year or data sensitivity, and filter for different sectors like academic, governmental or the media.

World’s biggest data breaches & hacks

Our data problems could get a whole lot worse, and not because of hackers this time, but politicians.

A no-deal Brexit may trigger a data disaster, and UK companies don’t have a clue
In the event of a no-deal Brexit, the Data Protection Act will ensure that personal information processed in the UK will keep enjoying the same level of protection they do now. Still, under EU law, the UK will be automatically considered a third country not bound by GDPR rules, and able to diverge from the current strong standards if parliament so decides. Consequently, data from EU countries would not be able to flow freely to the UK.

“Things will remain the same for organisations residing in the UK, and who need to transfer data to the EU,” says Cillian Kieran, CEO of privacy start-up Ethyca. “But you won’t be able to gather data from the EU into the UK. This is an issue for any company that processes information at any level.”

Poor performance

For such a small number, a school’s Progress 8 score can be quite a big deal. So the last thing we need is an exam board messing up the performance tables process by not sending complete data to the DfE.

Progress 8 error in performance table checking after BTEC gaffe
Peter Atherton, data manager at a school in Wakefield, told Schools Week some schools had received a “nasty surprise” when they went to check the website.

“It could be the case that, if all of these qualifications were missing for your school, that could affect your progress 8 score by quite a lot. Some schools are saying they’re -0.20 below what they were expecting.”

Gaffe is such an odd word, if you think about it. French, I guess. Would that make Pearson a gaffeur?

It is the second gaffe relating to BTECs to hit the exam board this year.

In August, Pearson was forced to apologise after it hiked grade boundaries for its BTEC Tech Awards just days before pupils were due to collect their results, meaning youngsters were handed lower grades than they were expecting.

Requires improvement, I’d say.

Excel errors are everywhere

I know that Excel is only trying to be helpful when it ‘corrects’ what it sees as formatting errors, but it really needs to pack it in.

An alarming number of scientific papers contain Excel errors
A team of Australian researchers analyzed nearly 3,600 genetics papers published in a number of leading scientific journals [and] found that roughly 1 in 5 of these papers included errors in their gene lists that were due to Excel automatically converting gene names to things like calendar dates or random numbers.

You see, genes are often referred to in scientific literature by symbols — essentially shortened versions of full gene names. The gene “Septin 2” is typically shortened as SEPT2. “Membrane-Associated Ring Finger (C3HC4) 1, E3 Ubiquitin Protein Ligase” gets mercifully shortened to MARCH1.

But when you type these shortened gene names into Excel, the program automatically assumes they refer to dates — Sept. 2 and March 1, respectively. If you type SEPT2 into a default Excel cell, it magically becomes “2-Sep.” It’s stored by the program as the date 9/2/2016.

A life in print

Last year, Facebook gave us the option to download all our data. Katie Day Good, an avid Facebook user since the early days, took them up on the offer and, perhaps because of her former interest in scrapbooking, decided to print it all out…

Why I printed my Facebook
Other files were less amusing. “Advertisers Who Uploaded a Contact List With Your Information” was a 116-page roster of companies, most of which I had never heard of, that have used my data to try to sell me things. The document called “Facial Recognition Code” was disturbingly brief and indecipherable, translating my face into a solid block of jumbled text—a code that only Facebook’s proprietary technology can unlock—about 15 rows deep. Some documents held secrets, too. “Search History” revealed an embarrassingly detailed record of my personal obsessions and preoccupations over the years. Crushes, phobias, people I have argued with and envied―this was the information I never wanted to post on Facebook, but instead had asked Facebook to help me find. This information, along with the facial recognition codes of my children (which were not included in the .zip file, but which I assume Facebook owns), is the data I most wish I could scrub from the servers of the world.

All told, my Facebook archive was 10,057 pages long.

The tree rings of US immigration

Here’s an unusual way of representing population growth. Pedro M Cruz, from Northeastern University in Boston, takes two centuries of US census data and shows the increasing population as rings of a tree, one for each decade.

For a radical new perspective on immigration, picture the US as an ancient tree
According to Cruz, the tree metaphor ‘carries the idea that these marks in the past are immutable’ and it ‘embodies the concept that all cells contributed to the organism’s growth’. As with so many renderings of US history, indigenous populations are conspicuously absent from the tableau. Still, Cruz’s skilfully deployed data doubles as a resonant work of cultural commentary, offering a rich and often surprising look at the ever-evolving makeup of the country.

There’s more information on the video’s Vimeo page.

Simulated dendrochronology of U.S. immigration (1830-2015)
Trees in their natural setting have annual growth rings that reflect varying environmental conditions; the rings’ forms are neither perfect circles nor ellipses. The algorithm is inspired by this variation and accordingly deposits immigrant cells in specific directions depending on the geographic origin of the immigrant. Rings that are more skewed toward the country’s East, for example, show more immigration from Europe, while rings skewed South show more immigration from Latin America. With this, it is possible to observe the quantity of immigration through the thickness of the rings. The color of the cells corresponds to specific cultural-geographical regions.

Re-thinking supposedly anonymous data

This is a little alarming.

Anonymised data isn’t nearly anonymous enough – here’s how we fix it
We developed a machine learning model to assess the likelihood of reidentifying the right person. We took datasets and we showed that in the US fifteen characteristics, including age, gender, marital status and others, are sufficient to reidentify 99.98 per cent of Americans in virtually any anonymised data set.

Some more examples.

The simple process of re-identifying patients in public health records
In late 2016, doctors’ identities were decrypted in an open dataset of Australian medical billing records. Now patients’ records have also been re-identified – and we should be talking about it.

‘Anonymous’ browsing data can be easily exposed, researchers reveal
A journalist and a data scientist secured data from three million users easily by creating a fake marketing company, and were able to de-anonymise many users …

“What would you think,” asked Svea Eckert, “if somebody showed up at your door saying: ‘Hey, I have your complete browsing history – every day, every hour, every minute, every click you did on the web for the last month’? How would you think we got it: some shady hacker? No. It was much easier: you can just buy it.”

Lancaster University’s student data stolen

University application processes are in full swing, but here is some reputationally damaging news from Lancaster University.

Lancaster University hit by cyber attack, hundreds of students’ personal data stolen
The full scale of the cyber attack was revealed yesterday (July 22), when university chiefs confirmed that hackers had breached IT systems and accessed student records … It said it regretted that the breach has led to fraudulent invoices being sent to some undergraduate applicants demanding large sums of money.

Two days later, and the police have arrested someone for it.

Man arrested over UK’s Lancaster University data breach hack allegations
Names, addresses, email addresses and phone numbers were among the categories of data visible to the hackers. Fraudulent invoices were sent to some, the university admitted. With overseas applicants (of which Lancaster had 575 last year from non-EU countries and 375 from other EU countries) paying fees measured in the tens of thousands of pounds per year, the potential for high returns is great.

Our sources added that around half a dozen students had paid these fraudulent invoices. The highest undergraduate fees for overseas (non-EU) students is Lancaster’s Bachelor of Medicine, Bachelor of Surgery (MBChB) course at £31,540.

It’s more than a little embarrassing, as Lancaster University is one of a number of universities offering degrees in cyber security

Cyber Security MSc – Lancaster University
In addition to the taught modules, you will also work on an individual research project, supervised by two academics from two of the four departments. Through this project, you will obtain an in-depth understanding of the theoretical and practical aspects of cyber security and technology. You will put the skills and knowledge you have developed throughout the year into practice and gain experience of tackling real-world cyber security issues.

Well, there’s a ‘real-world cyber security issue’ for you.

Known unknowns

An introduction to what promises to be a fascinating new blog from Anna Powell-Smith, “about the data that the government should collect and measure in the UK, but doesn’t.”

Missing numbers
Across lots of different policy areas, it was impossible for governments to make good decisions because of a basic lack of data. There was always critical data that the state either didn’t collect at all, or collected so badly that it made change impossible.

Eventually, I decided that the power to not collect data is one of the most important and little-understood sources of power that governments have. This is why I’m writing Missing Numbers: to encourage others to ask “is this lack of data a deliberate ploy to get away with something”?

By refusing to amass knowledge in the first place, decision-makers exert power over over the rest of us. It’s time that this power was revealed, so we can have better conversations about what we need to know to run this country successfully.

Excel timesavers

I sit and stare at Excel for a significant proportion of my day. I can’t believe I’ve not been aware of this simple trick with copying formulas without messing up cell references. It’s saving me an immense amount of time.

Copy Excel formula without changing cell references (or without file references)
It’s quite simple actually!

  1. Highlight the are you’d like to copy
  2. Go to Home / Find & Select / Replace (or press Ctrl + H)
  3. Search for = and replace with a text that’s not in your file – in this example I chose “notinfile” (note as mentioned in the comments in YouTube, you can also replace with ” =”, i.e. a space before the equal sign)
  4. Go back to Home / Find & Select / Replace (or press Ctrl + H) – search for your text – in my example “notinfile” and replace with =.
  5. That’s it!

Here are a few more tips and tricks.

10 easy Excel timesavers you might have forgotten
Microsoft has packed Excel with all kinds of different ways to get things done quicker. However, you can’t take advantage of these features if you don’t know about them. These ten techniques may only save you a few seconds every time you use them. That might not sound like much, but if you can integrate them into your workflow, you’re sure to reap the benefits over time.

A typical day, comically speaking

Via FlowingData, here’s a witty visualisation of how we spend our days, on average. It’s just a stacked bar chart, but turning it into a comic “can allow the audience to identify with the story, sparking self-reflection: “Is this how I live my life? How am I different?””

A day in the life of Americans: a data comic
There are three settings in this comic (a bedroom, an office, and a bar), each serving as a metonym for an activity (sleep, work, and leisure). I have also included colors and positions as redundant, but clarifying, codes of classification. Such scenes allow for a novel method of highlighting data; a setting inside a panel is “lit up” by a light source if the activity for which it stands occupied those two hours of Americans the most.

a-typical-day

Self-improvement

The Economist’s charts are usually very clear and helpful, but that’s not to say they can’t be improved – as they themselves show.

Mistakes, we’ve drawn a few
At The Economist, we take data visualisation seriously. Every week we publish around 40 charts across print, the website and our apps. With every single one, we try our best to visualise the numbers accurately and in a way that best supports the story. But sometimes we get it wrong. We can do better in future if we learn from our mistakes — and other people may be able to learn from them, too. […]

Misleading charts
Let’s start with the worst of crimes in data visualisation: presenting data in a misleading way. We never do this on purpose! But it does happen every now and then. Let’s look at the three examples from our archive.

Mistake: Truncating the scale

self-improvement-1

 

Another data protection failure

Hot on the heels of Facebook’s latest password problem, TechCrunch has news of another online service with a very shoddy approach to data protection – i.e. there wasn’t any.

The app, Family Locator, allows families to track each other’s movements, similar to the location sharing option in Google Maps. But it seems the backend database for their nearly a quarter of a million users wasn’t protected at all.

A family tracking app was leaking real-time location data
Based on a review of the database, each account record contained a user’s name, email address, profile photo and their plaintext passwords. Each account also kept a record of their own and other family members’ real-time locations precise to just a few feet. Any user who had a geofence set up also had those coordinates stored in the database, along with what the user called them — such as “home” or “work.”

They tried to get in touch with the developer, React Apps, but to no avail.

The company’s website had no contact information — nor did its bare-bones privacy policy. The website had a privacy-enabled hidden WHOIS record, masking the owner’s email address. We even bought the company’s business records from the Australian Securities & Investments Commission, only to learn the company owner’s name — Sandip Mann Singh — but no contact information. We sent several messages through the company’s feedback form, but received no acknowledgement.

On Friday, we asked Microsoft, which hosted the database on its Azure cloud, to contact the developer. Hours later, the database was finally pulled offline.

What makes good governance?

In an attempt to get rid of the sour taste left in our mouths from yesterday’s post about the rise of populist politics, here are some more award-winning data visualisations via David McCandless and the Information is Beautiful people.

The winners of the World Data Visualization Prize
Conducted in partnership with the World Government Summit, the prize focuses on how governments are improving citizens’ lives. We asked entrants to use the power of data-visualization to illuminate data on the innovations and decisions – seen and unseen – that drive progress.

Here’s my favourite, an interactive overview of the different factors that contribute to happy countries (or not).

GOV|DNA — Discover the DNA of a good government
This interactive visualization enables the exploration of the DNA of a good government. You can analyze and compare multiple indicators to investigate their influence on countries and the related behaviour and performance of governments.

what-makes-good-governance-1

Where is everybody?

Each six months Andy Kirk of Visualising Data highlights some of the significant developments in data visualisation. It’s a great collection, but this one in particular caught my eye.

10 significant visualisation developments: July to December 2018
2. ‘Human Terrain’: A genuinely captivating project from Matt Daniels of ThePudding, ‘Human Terrain’ is a staggeringly detailed, explorable prism map of the world’s population that can trap you into browsing for far longer than you can realistically afford. It evokes memories of a classic graphic from 2006, created by Joe Lertola for Time magazine. There is also a wonderful companion piece, ‘Population Mountains‘, where Matt walks through ‘a story about how to perceive the population of cities’.

When you fly from one part of the world to another, it becomes very quickly apparent just how crowded some places must be, compared to others.

visualising-populations-2

Human Terrain: visualizing the world’s population, in 3D
Kinshasa is now bigger than Paris. Guangzhou, Hong Kong, and Shenzhen are forming an epic, 40 million-person super city. Over the past 30 years, the scale of population change is hard to grasp. How do you even visualize 10 million people?

visualising-populations-3

It puts those incredibly dense housing schemes in Hong Kong I mentioned earlier into context, doesn’t it?

Population growth, like charity, starts in the home, so here’s an animated chart on family sizes in the US.

How many kids we have and when we have them
The chart above shows 1,000 timelines, based on data from the National Survey of Family Growth. Each moving dot is a mother. Age is on the horizontal, and with each live birth, the dot moves down a notch. The green bubbles represent the total counts for a given age.

visualising-populations-1

It’s interesting to watch the chart populate. You’ve got to wonder about the stories behind those outliers though.

Statistically insignificant?

One of the dangers at just looking at the numbers.

Progress 8 scores for most schools aren’t that different
There were over 300 schools with P8 scores between -0.05 and +0.05 – a difference of over 300 rank places (10% of schools) between the highest and lowest scoring of them. But what do these numbers mean?

Let’s say the score for School A was +0.05 and School B was -0.05. Taking the numbers at face value, one interpretation is that if you picked two pupils with the same KS2 attainment, the two pupils would have the same grades in seven of the subjects included in Attainment 8 but the pupil from School A would have one grade higher in one and only one subject than the pupil in School B.

Is this an educationally important difference?

It depends?

And talking of Progress 8 confidence intervals…

statistically-insignificant

xkcd: Error bars

Bringing back postcards

Postcards are such simple things, really – just small rectangular pieces of thin card. No technology required. Perhaps that’s why they don’t seem as popular these days? But thanks to the Postcrossing project, I’ve been sending and receiving more and more — and from all over the world.

Postcrossing history
The Postcrossing project was created in 2005 by Paulo Magalhães as a side project when he was a student in Portugal. Paulo loves to receive mail and postcards in particular; from friends, family — or from anyone in the world. Finding a postcard in the mailbox always makes his day!

He knew more people shared the same interest, but there was no good way yet to connect everyone. And that’s how he got the initial idea of creating the online platform for this which he called Postcrossing. Its goal: to connect people across the world through postcards, independently of their country, age, gender, race or beliefs.

Here are some more postcard-related links.

Wish you were here? Postcards from the art world
“It’s possible to form a significant collection of extremely good and important works of art without being wealthy,” he says. “Anyone could decide to form a collection very close to mine with most of the same things – and I like that. It’s anti-exclusive.”

bringing-back-postcards-1

A postcard writing Rube Goldberg machine in a suitcase
As the sun regrettably sets on the art of letter writing, the inventive folks at design studio HEYHEYHEY have pieced together a clever contraption that promises to keep the art of travel postcards a thing of the present. Kind of. Melvin the Traveling Mini Machine is an elaborate Rube Goldberg machine that fits in a pair of suitcases that executes the simple task of “writing” and stamping a postcard of your choice, that is, if the absurdly elaborate sequence of steps goes off without a hitch.

bringing-back-postcards-2

Two women leading parallel lives are getting to know each other through data
Giorgia Lupi, who lives in New York, and Stefanie Posavec, who lives in London, are engaged in a long-distance, postcard-based data exchange in order to get to know each other better: “Dear Data.” They’ve only met in person twice, and they’re both interested in data, so they’re sending each other postcard drawings of data about their day-to-day lives.

Four Corners Books announces its next publication
Four Corners Books has just announced the next book in its Irregulars series titled, Leeds Postcards, a celebration of the independent postcard press. For the past four decades, independent postcard press Leeds Postcards has been making oppositional, inspiring images; activism by design. The cards are not of Leeds; the name represents a defiant rejection of the hegemony of London. The images cover a fascinating range of domestic and international politics, causes and campaigns, creating, in their own unique and graphically inventive way a record of the struggles as well as the progressive political triumphs from 1979 to the present day.

GDPR is still a thing, right?

Some recent data protection stories that have caught my eye.

French data watchdog dishes out largest GDPR fine yet: Google ordered to hand over €50m
The French agency, CNIL, ruled today that the search giant had offered users inadequate information, spreading it across multiple pages, and had failed to gain valid consent for ads personalisation. […] The CNIL concluded that Google had breached the General Data Protection Regulation in two ways: by failing to meet transparency and information requirements, and failing to obtain a legal basis for processing.

Amazon, Apple and Google face data complaints
General Data Protection Regulation (GDPR) rules say EU customers have the right to access a copy of the personal data companies hold about them. However, privacy group noyb said it found that most of the big streaming companies did not fully comply. It has filed formal complaints, which if upheld could result in large fines.

Google accused of GDPR privacy violations by seven countries
Consumer groups across seven European countries have filed GDPR complaints against Google’s location tracking (via Reuters). The European Consumer Organisation (BEUC), of which each of the groups are a member, claims that Google’s “deceptive practices” around location tracking don’t give users a real choice about whether to enable it, and that Google doesn’t properly inform them about what this tracking entails. If upheld, the complaints could mean a hefty fine for the search giant.

The NOYB organisation gets mentioned a number of times there.

Max Schrems: The privacy bubble needs to start ‘getting sh*t done’
After years locked in numerous long, drawn-out and often bitter legal battles, Schrems decided to launch a nonprofit aiming to help people bring their own consumer privacy cases to court.

The plan is for NOYB (None Of Your Business) to take advantage of the incoming European Union General Data Protection Regulation, which offers more options for collective redress across the bloc, and harness the momentum Schrems has built up with various high-profile court cases.

Seems to be working. (Via)

Charting Van Gogh’s shifting colours

Taking a break from their AI coverage, Artnome take us on an interesting journey through Van Gogh’s shifting colour schemes, busting a couple of myths as they go.

New data shows why Van Gogh changed his color palette
We were not convinced by the medical reasoning behind the shift in Van Gogh’s color palette and we could not think of any French Impressionists that painted with colors nearly as bold as Van Gogh, so we decided to take a look at some other possibilities.

Van Gogh was a restless soul and moved around quite a bit. He also spent a lot of time painting outdoors, especially in his later years. As someone who famously struggled with mood swings, we thought location, and more importantly, weather patterns may have impacted his use of color.

To test this, we created composite images averaging every painting Van Gogh created from each of the major locations he worked from and compared them to weather patterns from those regions. We think the results are quite remarkable.

Using data and technology like this is a fascinating way to learn more about works of art and the hidden narratives behind them.

“Hello? Is this thing on?”

I certainly enjoy reading about these voice assistants more than I do using them.

You bought smart speakers over the holidays. Now what are Amazon and Google doing with your data?
Ultimately, the choice to keep a smart speaker around comes down to what you’re getting out of the product. For some people with physical disabilities or intellectual differences, smart speakers can make household tasks easier or provide an engaging presence in daily life. For tech junkies like my friend, the sheer joy of commanding a smart home network might be enough. For Hoffman-Andrews, though, the benefits of a speaker don’t outweigh the costs. He bought a couple of products for testing, but he admits he couldn’t actually bring himself to set them up. Being able to ask a speaker to dim the lights or play a weather forecast just didn’t seem like a good enough tradeoff for giving companies access to his home.

“Is it normal to have cameras and microphones pointed at you and your guests? Currently the answer is mostly no,” he says. “These devices aim to change the answer to yes.”

The best data visualisation work of 2018

Another end of year roundup, this time looking at data visualisation design.

Information is Beautiful Awards 2018: The Winners
Let’s raise a glass to dataviz that pushes boundaries, illuminates truth, and celebrates beauty. Thank you to everyone who joined us on the Information is Beautiful Awards journey this year – now see which entries took home trophies at tonight’s spectacular ceremony.

There is so much to pour over, here. Two that stood out particularly for me was this visual representation of a Beethoven string quartet and this unusual view of our lively planet.

Dynamic Planet Interactive Scientific Poster
The Interactive Scientific Poster „Dynamic Planet” was designed and developed for the exhibition „Focus Earth” of the GFZ German Research Centre for Geosciences in Potsdam. One of the main advantages of a digital poster is that it can display dynamic content. This is at the same time the essential statement of the scientific poster “Dynamic Planet”: our earth never stands still, is permanently shaken by earthquakes. These tensions are measured by three measuring points of the GFZ and their data is visualized in real-time in an interactive poster in the exhibition context. The viewer is given a direct impression, he can be a “witness” to current measurement and research.

data-visualisation-2018

This video demonstrates how people can interact with the poster, to navigate the large amount of data presented in an intuitive, visual manner.

Dynamic Planet – Scientific Poster
The challenge for the interface design was to ensure a clear overview, despite of the massively many events. The solution consists in an interactive graphical representation of the events by filtering the earthquakes by eg. magnitude and depth. A special visual feature of the scientific poster “Dynamic Planet” is the representation of the earthquakes depth in a transparent, rotating globe.