Weapons of Math Destruction Part 4

These chapters (actually they were last week’s) cover employment . Here’s Bryan’s prompt.

On the hiring side, I’m not sure whether algorithmic arbitrariness or human arbitrariness is worse.  I have a sense that, distinct from the expected biases (ethnicity, gender, geography/wealth)  algorithms might bias for similarity.  That is, they bias against candidates who have the larger skills to do the job, but whose previous job titles or majors aren’t a close word for word match for a job description.  Of course humans might be just as likely to have that bias, but a human, if they wanted to think “outside the box”  could at least be metacognitively aware of it.

I found the next chapter “Sweating Bullets”  more alarming. The core of the problem is that outside of widget production for a factory worker or sales volume, the link between what an individual worker does and an institutional KPI is often tenuous.  My instinct is that bad algorithms full of second or third order proxies make this much worse that a human based system with safeguards (such as something like 360 evaluation)

Did anyone else find the sociometric badge used in the call center (132)  seriously creepy?

As to one of Bryan’s questions, about whether boycotts can provide a meaningful check on this sort of thing, it seems to me it might work in the public sector where transparency can be enforced via FOIA, but I have little hope for the private sphere.  Boycotts sound good, but are rarely well enough organized or maintained to provoke real change.

Notes and Quotes

“…we’ve seen time and again that mathematical models can sift through data to locate people who are likely to face great challenges, whether from crime, poverty, or education. It’s up to society whether to use that intelligence to reject and punish them — or to reach out to them with the resources they need.” (118)

“The root of the trouble, as with so many other WMD’s, is the modeler’s choice of objectives. The model is optimized for efficiency and profitability, not for justice or the good of the ‘team.’  This is, of course, the nature of capitalism.” (129-130)

I was struck the other day by how similar Cory Doctorow’s whuffie system (from Down and Out in the Magic Kingdom), the rating system in the Black Mirror episode “Nosedive” and the Chinese social credit system I described in last week’s post are.

Weapons of Math Destruction Part 3

Here’s last week’s prompt for the Weapons of Math Destruction Book Club, I have chosen to ignore the provided questions, however. (Sorry, Bryan)

My big takeaway from these chapters is the importance of the decisions that are made about how to use data.  Both predatory recruiting and nuisance policing seem to start with explicitly harmful (the former) or flawed (the latter) justifications.  This makes the issue one of big data making it easier for people to do bad things.

The description of how the Chicago predictive policing initiative included social network analysis reminded me of the Social Credit system China is developing. (See this article from the Independent or this one from the Financial Times [warning:paywall]) Incidentally, the Independent article has a video and I was shown a pre-roll Lexus ad that was in Mandarin.

Unlike the Chicago system, where one’s core is known presumably only to police, the Chinese system, which includes in the “social credit score” algorithm your activity on social media and the scores of your friends, makes those scores public, encouraging you either  to lean on your friends with low scores in an effort to improve their behavior or to shun them. Both approaches will improve the social component of your score. I wonder to what extent social credit scores are used in the Western world and we just don’t know about it yet.

 

 

NOTES AND QUOTES*

(96) Justice cannot just be something that one part of society inflicts upon the other.
(102) Part of the analysis that led police to McDaniel involved his social network.

 

 

*Yes, I’m aware that it should probably be Notes ans Quotations, but I will sacrifice grammatical accuracy for rhyme scheme

Weapons of Math Destruction Part 2

I’m moving on to Chapters 2 and 3 of Weapons of Math Destruction .  In this week’s prompt, Bryan asks:

  • If creating or running a WMD is so profitable, how can we push back against them?

By making them less profitable.  The only way I see to do this is to require some human intervention in the decision processes these algorithms facilitate.  Making a person responsible for verifying the algorithmic outputs would at least improve accountability.  In the event of egregious harms, the person who signed off on the algorithmic output could be held to account for his or her decision.

  • Do you find other university ranking schemes to be preferable to the US News one, either personally or within this book’s argument?

I don’t know enough about them to say.

  • At one point the author suggests that gaming the US News ranking might not be bad for a university, as “most of the proxies… reflect a school’s overall quality to some degree” (58).  Do you agree?

This doesn’t matter.  Even if the proxies are good proxies, the fact that it’s a ranking system creates the arms race condition which forces colleges to game the system aggressively by doing things like rejecting qualified applicants who are unlikely to enroll.  O’Neil discusses this decline of the safety school.  The root of the problem is the role of reputation in the whole system.


Not surprisingly the discussion of college rankings in Chapter 3 resonated strongly with me for two reasons:

I applied to colleges just at the end of Ivy League collusion on financial aid offers.  I wonder how much the early effects of the US News rankings might have affected me as an Ivy League applicant.

As the parent of teenagers, I had only a vague sense of how the admissions process has changed since I was a college applicant. I worry for my children.

 

Note:

“However, when you create a model from proxies, it is far simpler for people to game it.” (55)

 

Weapons of Math Destruction Part 1

Bryan Alexander is facilitating an online book club reading of Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.  I am about two weeks behind (typical), so I will focus on just a couple of his questions for Part 1.

A. “What would it take for an education algorithm to meet all of O’Neil’s criteria for not doing damage?”

The big problem with  almost all educational measurement is the use of clumsy at best proxies (O’Neil uses the term) for the learning  we want to measure.  Since the fundamental output metric is a test with all the possibilities for manipulation that suggests, when we then try to measure what input changes improve that output we are are at least two levels of abstraction removed. Until we can measure educational outcomes some way other than by means of a crude manipulable proxy, I’m not sure we can fix this.

B. “What are the best ways to address the problem of “false positives”, of exceptionally bad results, of anomalies?”

I think the best way to solve this problem is to place some limits (preferably not determined by an algorithm) on the kinds of decisions we allow algorithms to make without human input.  The potential harm of a bad book recommendation from Amazon is much lower.  That probably means  thoughtful review of every adverse algorithmic recommendation by at least one live human being. Of course, thus undermines the efficiency and scale that algorithms are designed to create.  An important step is to acknowledge that algorithms are not neutral even if they manage not to be arbitrary.  They encoded the assumptions and biases of their creators, and acknowledging those assumptions and biases is a key part of the design process.

The DC schools example draws attention to the importance of checking for flawed input data.  After all, the algorithm is only as accurate as the data you feed it.

Notes: O’Neil’s three criteria for a Weapon of Math Destruction are “opacity, scale, and damage.” She uses the initialism WMD.  I wish she had come up with something else, because of the namespace confusion with chemical, biological and nuclear weapons.

Opacity makes me think of Frank Pasquale’s The Black Box Society, which I haven’t read yet.  The synopses of Pasquale’s book make me wonder how his and O’Neil’s work intersect.

 

 

 

Tooting Alone

This month, the early adopters are all on mastodon. Mastodon is actually a server implementation for OStatus (which used to be StatusNet, which was originally on identi.ca) the TL;DR of Ostatus is “like twitter, but federated”. As of this morning there are almost 900 active instances. Since the software is open, different instance administrators can set their own policies and users can find an instance whose culture agrees with them.

Mastodon also has an option to operate a single user instance, and this is where it gets less clear. Mastodon is designed to show three different timelines, the users personal timeline, a public timeline for the local server, and a federated timeline. On a single user server the local timeline will show the “toots” (Yes, that’s what they call a status post) of the instance’s one user, and the federated timeline will look very similar to the single user’s personal timeline. In managing your own presence on the network, you simultaneously isolate yourself from it. It’s possible however that this won’t end up mattering very much. I can’t remember the last time I looked at the Twitter public timeline. If OStatus ends up working the same way, it won’t matter how many people are on your instance, because you will interact with the network through the people you follow, even if they are on many different instances. While the local timeline won’t show much , the federated timeline, which is sort of a second degree network (see https://cybre.space/users/nightpool/updates/13933 ) looks as if it may end up, on a single user server, as a very personalized feed.

This ties in to the IndieWeb movement with its idea of POSSE (Publish own site, syndicate elsewhere) and there are already connectors to publish from tools like withknown to mastodon. Withknown is great for publishing, but not very good for aggregating. There is always RSS, and in fact mastodon autogenerates atom feeds per user (site.tld/users/username.atom). This leaves you using one application to read and another one to reply. I really wish something like Mark Pesce’s Plexus (https://github.com/mpesce/Plexus) was still active. How hard would it be to build a personal dashboard that would bring together RSS reading / OStatus / blogs /etc.?

Social Media and Tool Creep

Last week, Mike Caulfield lamented that social media is poorly suited to enhancing human potential. If you think about it, this shouldn’t surprise one too much, since it wasn’t designed for that.  Facebook was, after all, first and foremost a social tool, a virtual version of the paper books new college students resorted to in ages past to figure out who that cute guy/girl in your English class was.

For the task for which they was originally designed, fostering social connections between people, Facebook, Twitter and other social platforms work well, but then something happened.  As social platforms moved to the center of our online lives we wanted them to be the hubs not just of our social interactions, but of our information gathering.  This dovetailed nicely with the platform creators quest to grab, quantify and monetize more and more of our attention, but, as Mike points out, was not necessarily good for us.

D’Arcy Norman quoted an old post that touched on the same issue.  In 2008  he wrote about what he recently dubbed real-time toll.

Every time I read an update by someone that I care about, I think about that person – if only for a second – and my sense of connection is strengthened.

But, I fear that the strengthened social connections are not worth the cost borne in superficial thinking.

This led me to a little experiment. I looked at my Facebook activity feed for the almost completed month. I’ve only interacted with about 75 entities, and two thirds of those are people in the county I live in.  This comes with the usual caveat that it includes outbound plus inbound tags but not inbound likes and reactions.

Maybe the key to managing D’Arcy’s real time toll is to only follow people you care about enough that whatever superficial thinking it causes is worth it.

I’m going to presuppose that social networking sites are not very good tools to expand human potential.  The ratio of signal to the noise of social interaction is just too low.  What would such a tool look like?  Is a good list of RSS feeds adequate, or is something like fedwiki, wikity, or a choral explanations platform necessary?  If you end up with something that isn’t extremely decentralized, how do you generate beneficial network effects while keeping the signal to noise ratio high enough to generate value?

Verifying Academic Credentials with Blockchain

This morning, a college classmate posted a link to this Campus Technology article on blockchain based transcripts.  It turns out the University of Nicosia  is already doing this. Campus Technology used the d- word , disrupt, to describe the potential of this approach.  On the plus side:

  • this would allow verification of credentials without contact with the issuing institution.  That would seem to save lots of time and trouble in registrars offices everywhere.
  • the permanence of the ledger would mean that it wouldn’t matter if a credential issuing entity ceased to exist
  • The lack of having to produce paper trails might make traditional institutions more willing to offer micro credentials

Pitfalls include:

  • Security – I’m a blockchain novice, but my understanding is that the ledger is quite secure because so many users are verifying it.  That said, even more important than actual security is the perception thereof.  It may take a long time before credential audiences (education institutions, employers, etc.) trust blockchain credentials.
  • Privacy – Blockchain records are permanent and public.  How do you  ensure that only authorized viewers can see the details of a credential (courses and grades)?  What if you don’t want to publicize your attendance at a particular institution?

 

Coding and Literacies

This morning my feed is full of discussion of the coding for all movement.  Anya Kamenetz asks how long “I’m not a coder” will be a socially acceptable thing to say.  Some, including The Atlantic’s Melinda Anderson are more skeptical.  I’m not sure this is the right question, any more than teaching everyone to repair cars is necessary.  A more important issue is fundamental understanding of how systems work.

Let’s go back to cars.  I don’t know enough to repair my own car. I do on a basic level understand how cars work.  Refined petroleum is ignited by a spark from a battery in a closed cylinder, the resulting explosion moves a piston while creating exhaust gases, the moving pistons turn a drive shaft.  This commonly shared understanding of how cars work and that even providing a modest 12V requires a fairly large battery means that it is commonly understood that creating an inexpensive zero emission vehicle is a hard problem.

Contrast this with the collective understanding of digital encryption, given FBI v Apple and the preceding discussions of encryption backdoors.  I have yet to find an expert on how encryption works who believes that a mechanism which would allow law enforcement to bypass encryption without allowing hostile governments and criminal actors to do the same is technically possible.  See this summary for one example.  However, those with less technical knowledge don’t seem to share this belief.

Perhaps the key is not being able to code per se, but having enough fundamental knowledge of how computers work in order to have a shared understanding of what, for a computer, is possible or impossible, easy or difficult. The broader question when designing education is, “Which systems are important enough that we need  a shared understanding of their fundamental principles in order for society to function well?”

Blogs are blogs and wikis are wikis and never the twain….

Mike Caulfield, whose latest project, Wikity, brings to WordPress some features of federated wiki, asks whether an architecture that would allow data to flow seamlessly between blogs and wikis is a desirable thing.  In a comment, Kartik Agaram suggests that tagging makes blogs behave in a more wiki-like way.

To unpack this, I found it helpful to think all the way back to physical libraries. The whole notion of card catalogs and call numbers is a system designed to make physical objects findable. No matter how many cards referred to an item, the call number (a primary key, as it were) pointed to one spot on a shelf.  There has been a tendency to think of tagging as being fundamentally different because the artifacts are digital, but as Mike points out, the web is still location based, even if the locations are virtual.  Tagging merely allows, to extend the card catalog analogy, there to be a theoretically infinite number of “subject” cards for any given entity or entities under any given subject.

Given that the blog is clearly one person’s writing and thought, it makes more sense for it to have a single canonical address.  Wiki is more reference like and seems to lend itself better to Mike’s notion of connected copies, since the question of authorship is less important.

Now on to Mike’s actual question.  How valuable is it to be able to seamlessly move data across this divide?  I think the answer depends on how important you think the attribution chain is.  If it’s not important at all, just cut and paste.  If it is important, is it equally important in both contexts?

For the blog, some sort of attribution clarifies what is the author’s own thought versus what came from somewhere else.  However, when that somewhere else is a wiki, you deal with a source that is designed not to be static.  All of the web does that, in fact, which is why we have accessed on fields in web citations and everyone should love the Internet Archive Wayback Machine.  The vary malleability of a wiki page may lessen its value as a source. Would a wiki to blog bridge, like a fedwiki fork, pull the entire history of a wiki document up to the point of citation?  It’s with connected copies that this sort of link makes more sense. Even if the copy you originally cited has disappeared, you might find another.

Going the other direction, one expects a blog post, with it’s time and date stamp, to be a fixed oeuvre, so it makes more sense as a source or reference for a wiki document. It’s usually static nature also makes this process easier.

Having thought “aloud” through the use cases, I’m not in desperate need of a bridge. If reference by content grows in importance, it might make more sense.

Musical Theater and Transculturation

Well, I finally did it. I broke down and started listening to the Hamilton cast album. Hamilton is, for those who don’t know, the (now grammy winning) hottest thing on Broadway, a biographical musical about the founding father, duel victim and face of the ten dollar bill, which stars show creator and MacArthur Fellow Lin-Manuel Miranda.

From the very opening line, it’s clear that this is not a period piece. Hip-hop and rap influences are immediately apparent, and I found that just a bit off putting at first hearing.  Then I thought about why I found it off putting.

Douglas Hofstadter uses the word transculturation to refer to the process of replacing cultural references when translating a text to a new language.  That’s sort of what’s happening here.  I’m sure that the founding fathers didn’t rap.  If you watched the performance of the show’s opening number on the recent Grammy telecast, there was an interesting juxtaposition.  Although the musical style is contemporary, the costumes are period, so you see the eighteenth century and hear something much more modern.

Of course Hamilton is far from the first show to do such a thing.  In West Side Story, Bernstein completely transculturated Romeo and Juliet in Verona to Tony and Maria in New York City.  Shows like Candide and A Little Night Music juxtapose modern music with older settings.  There’s (so far, I’m only a few songs in) one moment in Hamilton that evokes actual 18th century musical style, Samuel Seabury’s1 half of  “Farmer Refuted”.  I wonder if Miranda is contrasting Loyalists and Patriots by having the former sing music that sounds eighteenth century and European, while the latter sound hip, modern, and American.

This sort of approach has some benefits. A noticeable spike in Google search volume for the term “Alexander Hamilton” followed the Grammy performance. On the other hand, it doesn’t exactly encourage one to consider an event in its own historical and cultural context. What do we gain and lose when we retell an historical story through a modern cultural lens?

 

1Students of religious history will recognize the name. Seabury was later the first Anglican bishop in the United States.