TheoryFest: Avi Wigderson's plenary talk.

The plenary talk on Tuesday morning was by Avi Wigderson, on the nature of TCS.

Now if you haven't attended an Avi Wigderson plenary before, you should imagine yourself lying in a nice warm tub of water with soothing bath salts massaging your aching limbs. Avi's talks are balm for the poor persecuted theoretician who comes to STOC to remember that there people in the world who don't insist on demanding practical value for every darn theorem you prove.

His talk was not technical, and not even particularly informative for anyone who's been paying attention to TCS the last 40 years. What it was, was a love song to TCS, sung by a bard with +5 charisma who makes you feel like you have purpose and meaning in your life once again.

His slides will be up soon, and you can see them for yourself. But if there's anyone who can make the argument for the universality, depth and reach of TCS at its highest conceptual level, it's Avi. I'll link them when they're available and if you care at all about the field, you should take a look.

TheoryFest: Attack of the talks

Day 2 at TheoryFest, and people are still attending talks. Ok maybe I shouldn't be that surprised. But it's kind of nice to see anyway. The lounge area is deserted during the sessions and full during the breaks.

The TheoryFest organizing committee (as represented by Ryan Williams) has organized lunch time meetups for senior and junior people (where junior means grad students and postdocs, not assistant profs — sorry, assistant profs). The idea is that the senior people sign up and the junior people can email them to meet up. It's a nice idea, especially given the feeling that I've heard from people in the past that STOC is an unfriendly place for students, especially those not from high-powered theory places.

One interesting aspect of coming to STOC after a number of years is while I know many the older folk, I don't know most of the students, and so I float around relatively anonymous (it doesn't help that I haven't blogged much in recent months - I've been distracted, people!). It's almost like being at a brand new conference that I've never attended before.

TheoryFest: Short Takes

So far, the tutorials appear to have been well attended, The DL tutorial had a full house in a big room, but the other two tutorials did pretty well to. The plenary talks (the reason I'm here) start today and it will be interesting to see what kind of attendance we see.

The business meeting will reveal the official numbers: indications are that they will be quite good especially considering we're not in the US.

Montreal is a nice town. My AirBnB is right next to Chinatown and we're pretty much in the center of everything. The parts I've walked through have this kind of pacing with cul de sacs, pedestrian-only areas, main roads, and parks that feels like well-paced musical composition. 

TheoryFest I: Deep Learning

(ed: I'm guessing you never thought those words would appear together in the same phrase)

Ruslan Salakhutdinov gave a tutorial on deep learning today. Now deep learning is a tricky topic for theory (more on that below), but I thought he did a nice job in his two hours of

• explaining the basics of how a neural net works and how it's trained, without getting too far into engineering weeds, but also being able to explain important ideas like drop out, SGD, batch normalization and momentum. He skillfully avoided the alphabet soup of architectures in a way that didn't really affect one's understanding (I think). He didn't get too much into RNNs, but I think that was a conscious and fair choice. 
• Discussing the unsupervised element of DL - autoencoders, RBMs, DBMs, and GANs. Now I have a little bit of an advantage here because we're running a summer reading group on GANs, but I liked his framing here in terms of supervised and unsupervised, as well as the different kind of generative criteria (probabilistic/not, tractable/intractable, explicit/implicit) used to classify the different approaches. 

In a first at STOC (maybe?) the audience got to see a video of an RL system learning to play Doom. It was pretty neat.

Having said that, I'm not exactly the right audience for this kind of talk, since I'm decently familiar with deep learning. What surprised me though was that when I polled people during the breaks. most of the people who attended the tutorial felt the same way. And the common refrain was "We've heard so many faculty canddiates talk about deep learning that we know the basics now"!

So I almost wonder if Russ miscalbrated the level of the audience.

There was also some minor grumbling about the lack of clear open problems. I actually don't fault him for that. I think it might have been useful to expose core ideas for which answers don't exist, and some of these came out in the Q&A.

But let me make a more general observation. Deep learning is a tricky topic for theoreticians to negotiate, for a number of reasons.

• firstly, I don't think it's even useful to ask the most general form of "what does a neural net DO" questions. Neural nets are very very general (in particular a 2-level neural net can approximate any function). So asking general questions about them is like asking to characterize a Turing machine with no constraints. You can't say much beyond recursive and r.e. I think ther right questions are much more specific. 
• DL right now is very much an engineering discipline, which is to say that the practice of DL is focused on trying out engineering hacks that appear to get improvements. And these improvements are significant enough that it really doesn't matter why they work. In other words, DL doesn't need theory… at least now. 
• Even if you don't grant the previous two positions, there's another issue. Descriptions of DL systems feel a lot like experimental physics: "hey we did all of this and it worked this way. Now give us a theory". With the difference that there's no "there" there: there's no fixed Nature that we can design theoretical laws against. Only a gazillion-dimensional highly nonconvex landscape where we don't even try to find a provably high quality answer. 

So I think we're on our own if we want to (or care to) understand the computational power and expressivity of neural networks. It's very interesting, and we're seeing nice results begin to appear, but we should do it because there's interesting theory to be had here, rather than trying to hew to close to actual DL systems.

Off to TheoryFest 2017

I'm at the airport getting ready to board my flight for Montreal to attend TheoryFest 2017. And much to my amazement, I discover that STOC has its own event mobile app. Who knows, maybe this means that by next decade theory conferences will do double blind review? (ed: stop it, now that's crazy talk!)

Snark aside, I'm very excited to see how the new format for STOC works. This is an experiment, and like all experiments it will take some time to see how the idea plays out. But first impressions matter, and if this year's iteration goes well, that will be good news for the overall plan.

I'm of course excited for the plenary invited talks, but I'm also excited to see the tutorials and workshops. Even theory land is not safe from the invasion of the Huns deep learning and I'm sure there will be standing-room only space at Ruslan Salakhutdinov's tutorial on the foundations of deep learning.

My "coverage" will be a mix of tweeting and blogging. Stay tuned!

Easy memory aides

Certain memory aides are so ... well.. memorable that they stick in your mind exactly the way they should. Here are three that I've heard of:

• Keeping track of the Baltic states: I think I heard this from +Fernando Pereira - They are alphabetical in order from north to south. So it's Estonia, Latvia and then Lithuania. 
• Converting between miles and kilometers: This is a particularly geek-friendly mnemonic - If the value in miles is (close to) a Fibonacci number, the value in kilometers is (close to) the next Fibonacci number. This works because the golden ratio is very close to the miles-to-km conversion factor.
• Keeping track of type-I/type-II errors: I heard this one on twitter the other day, and it goes like this. In the story of the boy who cried wolf, the errors appear in order. 

Things I need memory aides for:

• Bernstein vs McDiarmid vs Azuma, or different Chernoff bounds in general. 
• Jensen's inequality, so I don't have to constantly draw the picture of the convex function to derive it.
• Submodularity vs supermodularity: I know you add the diagonals and compare them, but which way is which???

Others? 

TheoryFest at STOC 2017.

(ed: I can't believe it's been four months since my last post. Granted, I've been posting over at algorithmicfairness.wordpress.com, but still. Time to turn in my theory card...)

One of the things I've been doing is helping out with the STOC TheoryFest this summer. Specifically, I've been on the plenary talks committee that was tasked with identifying interesting papers from other parts of CS and beyond that might be interesting for a STOC audience to hear about. After many months of deliberations, we're finally done! Boaz has more details over at his blog.

Do take a look at the papers. Boaz has a great summary for each paper, as well as links to the sources. You'll see a pretty decent coverage of different topics in CS, including even some theory B !!

But more importantly, please do register for STOC! This year, the organizers are making a real effort to restructure the conference to make it more accessible with a diversity of events like the plenary talks, tutorials, and workshops, and there should be something for everyone. Early registration ends on May 21, so click that link now.

In this day and age, I don't need to emphasize the importance of standing up and being counted. If you've ever grumbled about the old and stultifying format of STOC, this is your chance to vote with your feet. A big success for TheoryFest 2017 will surely encourage more along these lines in future years.

A short course on FATML -- and a longer course on ethics in data science.

I'm at IIIT Allahabad teaching a short course on fairness, accountability and transparency in machine learning. This is part of the GIAN program sponsored by the Government of India to enable Indian researchers from outside the country to come back and share their knowledge. The clever pun here is that GIAN -- which stands for Global Initiative of Academic Networks -- also means 'knowledge' in Hindi (ज्ञान).
Anyway, I'm excited to be teaching a course on this topic, and that has forced me to organize my thoughts on the material as well (which has already led to some interesting insights about how the literature fits together). I'm writing out my lecture notes, and one of the energetic students at the course has kindly offered to help me clean them out, so I hope to have a set of notes in less than a year or so. 

Note that the syllabus on the linked page is evolving: it will change over the course of the next few days as I add more things in. 

I'm also excited to to be teaching a (brand-new) undergraduate course on ethics and data science next fall. I'll have more to say about this as I prepare material, but any suggestions/materials are welcome. 

Modeling thorny problems.

Today, I got into a nice twitter discussion with former colleague and NLPer extraordinaire (even when he hates on algorithms) Hal Daumé. Here's what he said:

one consistent msg I heard re ethics/policy today is that even though no one really knows the right answer, the question is how to move fwd

— Hal Daumé III (@haldaume3) October 19, 2016

akin to "perfect is the enemy of the good", but I think different because we don't even know how to measure "perfect" :)

— Hal Daumé III (@haldaume3) October 19, 2016

To which I replied:

@haldaume3 right. I think of this as a partial order rather than total. Measure better along chains and argue between maximal elements

— Suresh Venkat (@geomblog) October 19, 2016

It seemed to be worth fleshing this out a little.

In the work we've been doing in algorithmic fairness, one of the trickiest issues has been attempting to mathematically capture the different (and very slippery) notions of fairness, bias, and nondiscrimination that are used in common parlance and in the more technical literature on the subject. As Hal noted in his tweet above, these are very difficult problems for which no one really has the right answer.

How then does one make forward progress? In my view, framing is an essential part of the discussion, and one that we tend not to consider very explicitly in computer science. That is to say, deciding how to ask the question has much more impact on forward progress than the specific nature of the solutions.

This is not news if you look at politics, the law, policy discussions or the social sciences in general. But it matters a great deal as CS starts dealing with these issues because the choices we make in how to pose questions change the kinds of progress we hope to make. (while I realize that "questions matter, not solutions" is one of the oldest tropes in all of research, modeling issues like ethics and fairness requires a whole other dimension of framing -- see this nifty reframing of the editorial role of Facebook for example)

Specifically, I think of these problems as points in a partial order. It's relatively easy to talk about progress along a chain: that's what CS is really good at. But it's important to realize that we're living in a partial order, and that when we argue about different approaches, we might actually be arguing about fundamentally different and incomparable issues.

So then what happen when we reach different maximal elements (or imagine ideal maximal elements)? That's where the arguments center around beliefs, axioms and worldviews (not surprisingly, that's how we're thinking about fairness). But then the arguments are of a different nature altogether, and realizing that is very important. It's no longer an issue of what running time, what approximation ratio, what generalization bound you can get, but rather what aspects you're trying to capture and what you're leaving out.

So while it might seem that the problems we face in the larger realm of algorithms and ethics are irreducibly complex (they are) there's value in understanding and creating a partial order of formulations and making progress along different chains, while also arguing about the maximal elements.

Coda: 

One day, when I have nothing to do (ha!) I might attempt to write down some thoughts on how to model ill-structured problems mathematically. And it's not just about algorithmic fairness -- although that tests my skills royally. Any kind of mathematical modeling requires certain skills that we don't learn when we learn to solve crisply defined problems.

For example, one aphorism I found myself spouting some weeks ago was

Model narrowly; solve broadly. 

This was in the context of a discussion I was having with my undergraduate student about a problem we're trying to formalize. She had a nice model for it, but it was very general, and I worried that it was capturing too much and the essential nature of the problem wasn't coming through. Of course when it comes time to solve a problem though, using general hammers is a good place to start (and sometimes finish!).

Axiomatic perspective on fairness, and the power of discussion

Sorelle Friedler, Carlos Scheidegger and I just posted a new paper on the arxiv where we try to lay out a framework for talking about fairness in a more precise way. 

The blog post I link to says more about the paper itself. But I wanted to comment here on the tortuous process that led to this paper. In some form or another, we've been thinking about this problem for two years: how do we "mathematize" discussions of fairness and bias? 

It's been a long and difficult slog, more than any other topic in "CS meets X" that I've ever bumped into. The problem here is that the words are extremely slippery, and refract completely different meanings in different contexts. So coming up with notions that aren't overly complicated, and yet appear to capture the different ways in which people think about fairness was excruciatingly difficult. 

We had a basic framework in mind a while ago, but then we started "testing" it in the wild, on papers, and on people. The more conversations we had, the more we refined and adapted our ideas, to the point where the paper we have today owes deep debts to many people that we spoke with and who provided nontrivial conceptual challenges that we had to address. 

I still have no idea whether what we've written is any good. Time will tell. But it feels good to finally put out something, however half-baked. Because now hopefully the community can engage with it. 

Congrats to John Moeller, Ph.D

My student +John Moeller (moeller.fyi) just defended his Ph.D thesis today! and yes, there was a (rubber) snake-fighting element to the defense.

John's dissertation work is in machine learning, but his publications span a wider range. He started off with a rather hard problem: attempting to formulate a natural notion of range spaces in a negatively-curved space. And as if dealing with Riemannian geometry wasn't bad enough, he was also involved in finding approximate near neighbors in Bregman spaces. He's also been instrumental in my more recent work in algorithmic fairness.

But John's true interests lie in machine learning, specifically kernels. He came up with a nice geometric formulation of kernel learning by way of the multiplicative weight update method. He then took this formulation and extended it to localized kernel learning (where you don't need each kernel to work with all points - think of it like a soft clustering of kernels).

Most recently, he's also explored the interface between kernels and neural nets, as part of a larger effort to understand neural nets. This is also a way of doing kernel learning, in a "smoother" way via Bochner's theorem.

It's a great body of work that required mastery of a range of different mathematical constructs and algorithmic techniques.  Congratulations, John!

FOCS workshops and travel grants.

Some opportunities relating to the upcoming FOCS.

• Alex Andoni and Alex Madry are organizing the workshops/tutorials day at FOCS, and are looking for submissions. The deadline is August 31, so get those submissions ready!
• Aravind Srinivasan and Rafi Ostrovsky are managing travel and registration grants for students to go to FOCS. The deadline for applications is September 12. 

Of course FOCS itself has an early registration deadline of September 17, which is also the cutoff for hotel registrations. 

Pokégyms at Dagstuhl

Yes, you read that correctly. The whole of Dagstuhl is now a Pokégym and there are Pokémon wandering the streets of Wadern (conveniently close to the ice cream shop that has excellent ice cream!)

Given this latest advancement, I was reminded of Lance Fortnow's post about Dagstuhl from back in 2008 where he wistfully mourned the fact that Wifi now meant that people don't hang out together.

Times change. I am happy to note that everything else about Dagstuhl hasn't changed that much: we still have the book of handwritten abstracts for one thing.

Carl Zimmer's series on exploring his genome

If you haven't yet read Carl Zimmer's series of articles (one, two, three), you should go out and read it now!

Because after all, it's Carl Zimmer, one of the best science writers around, especially when it comes to biology.

But even more so because when you read the story of his personal quest to understand his genetic story in all its multifaceted glory, you understand the terrifying opportunities and dangers in the use of genetic information for predictive and diagnostic medicine. You also realize the intricate way that computation is woven into this discovery, and how sequences of seemingly arbitrary choices lead to actual conclusions about your genome that you now have to evaluate for risk and likelihood.

In a sense, this is the tale of the use of all computational approaches right now, whether it be in science, engineering, the social sciences, or yes, even algorithmic fairness. Zimmer uses the analogy with telescopes to describe his attempts to look at his genome, and this explanation is right on the money:

Early telescopes weren’t terribly accurate, either, and yet they still allowed astronomers to discover new planets, galaxies, and even the expansion of the universe. But if your life depended on your telescope — if, for example, you wanted to spot every asteroid heading straight for Earth — that kind of fuzziness wouldn’t be acceptable.

And this quote from Robert Green, one of the geneticists who was helping Zimmer map out his genome:

Ultimately, the more you know, the more frustrating an exercise it is. What seemed to be so technologically clear and deterministic, you realize is going through a variety of filters — some of which are judgments, some of which are flawed databases, some of which are assumptions about frequencies, to get to a best guess.

 In this is a message for all of us doing any kind of data mining.

NIPS Reviewing

This year, NIPS received over 2400 submissions. That's -- well --- a lot!

As a reviewer, I have been assigned 7 papers (note that this number will be utterly incomprehensible to theoryCS PC members who think that 30 papers is a refreshingly low load).

But none of that is as interesting as what NIPS is trying this year. From the PC Chairs:

New this year, we ask you to give multiple scores for technical quality, novelty, impact, clarity, etc. instead of a single global score. In the text boxes, please justify clearly all these scores: your explanations are essential to the ACs to render and substantiate their decision and to the authors to improve their papers.

Specifically, the categories are:

• Technical quality
• Novelty/originality
• Impact/usefulness
• Clarity and presentation

and there are also a few qualitative categories (including the actual report). Each of the numerical categories are on a 1-5 scale, with 3 being "good enough".

I've long felt that asking individual reviewers to make an accept/reject judgement is a little pointless because we lack the perspective to make what is really a zero-sum holistic judgement (at least outside the top few and the long tail). Introducing this multidimensional score might make things a little more interesting.

But I pity the fate of the poor area chairs :).

The Man Who Knew Infinity

I generally avoid movies about mathematicians, or mathematics.

I didn't watch Beautiful Mind, or even the Imitation game. Often, popular depiction of mathematics and mathematicians runs as far away from the actual mathematics as possible, and concocts all kinds of strange stories to make a human-relatable tale.

Not that there's anything wrong with that, but it defeats the point of talking about mathematics in the first place, by signalling it as something less interesting.

So I was very worried about going to see The Man Who Knew Infinity, about Srinivas Ramanujan and his collaboration with G. H. Hardy. In addition to all of the above, watching movies about India in the time of the British Raj still sets my teeth on edge.

To cut a long story short, I was happy to be proven completely wrong. TMWKI is a powerful and moving depiction of someone who actually deserves the title of genius. The movie focuses mostly on the time that Ramanujan spent at Cambridge during World War I working with Hardy. There are long conversations about the nature of intuition and proof that any mathematician will find exquisitely familar, and even an attempt to explain the partition function. The math on the boards is not hokey at all (I noticed that Manjul Bhargava was an advisor on the show).

You get a sense of a real tussle between minds: even though the actual discussions of math were only hinted at, the way Hardy and Ramaujan (and Littlewood) interact is very realistic. The larger context of the war, the insular environment of Cambridge, and the overt racism of the British during that period are all signalled without being overbearing, and the focus remains on the almost mystical nature of Ramanujan's insights and the struggle of a lifelong atheist who finally discovers something to believe in.

It took my breath away. Literally. Well done.

Google Recruiter Survey: Tell us what you think!

(ed: I can't believe it's been almost three months since I posted. Maintaining two blogs and twitter is more work than one might think)

Hello,
  Thank you for applying to recruit me to Google. I'm continuously working to provide a great experience to my recruiters throughout the hiring process, so I greatly value any feedback you’re willing to share about your experience—both what’s going well and what needs work.

Please share your feedback through my recruiter experience survey, which will be open from now through Monday, May 44. The survey should take less than 15 seconds to complete, and you can skip over any questions you prefer not to answer. Please keep in mind that your responses are not confidential, and will be used for humor improvements—not in decisions as to who I choose to allow to recruit me.

I absolutely do not love chatting with recruiters, though I sometimes receive more emails than I can respond to and have to prioritize questions regarding technical difficulties. Below are a few of my most frequently asked questions (FAQs) that may provide the answer you need.

Thank you for your time and have a great day,

Suresh, Suresh Venkatasubramanian Recruitment Experience Team.

FAQs
I never received feedback on my recruitment email - can you give me feedback?

I'm pretty limited on what I have access to within your recruiter profile (to ensure that my brain doesn't melt). I don't particularly mind you feeling confused about the outcome. Therefore, I suggest reaching out to your friends and family  if you have specific questions regarding your attempt to  recruit me at Google.


I’d like to provide more input on your process - should I email that over?
Please use the open ended comments at the end of the survey to leave any anecdotal feedback or additional thoughts. It's in the handy box titled /dev/null. This ensures your thoughts are not saved as I ignore aggregate feedback to share with my internet following.

 

Time to cluster !

After many blog posts, much discussion, and some contract back and forth, I'm happy to announce that +Sergei Vassilvitskii and I have signed a contract with Cambridge University Press to write a book on clustering.

"What's that?", you say. "You mean there's more than one lecture's worth of material on clustering?". 

In fact there is, and we hope to produce a good many lectures' worth of it.

Information on the book will be available at http://clustering.cc. It currently forwards to my blog page collecting some of the posts we wrote, but it will eventually have its own content.

Now the painful part starts.

Cartograms only exist in years divisible by 4...

Every four years, America suddenly discovers that it likes football soccer. Every four years also, America discovers the existence of the cartogram. The last time I ventured into this territory (here's the link, but link rot has claimed the images), I was accused of being a liberal shill. And here's more on cartograms and the 2004 election.

Teaching a process versus transmitting knowledge

Ta-Nehisi Coates is forced to state who he's voting for in the next election. He struggles with the act of providing an answer, because

Despite my very obvious political biases, I’ve never felt it was really my job to get people to agree with me. My first duty, as a writer, is to myself. In that sense I simply hope to ask all the questions that keep me up at night. My second duty is to my readers. In that sense, I hope to make readers understand why those questions are critical. I don’t so much hope that any reader “agrees” with me, as I hope to haunt them, to trouble their sense of how things actually are.

In the last few years, I've had to think a lot about what it means to teach a core class (algorithms, computational geometry) for which most material is already out there on the Web. I think of my role more as an explorer's guide through the material. You can always visit the material by yourself, but a good tour guide can tell you what's significant, what's not, how to understand the context, and what is the most suitable path (not too steep, not too shallow) through the topic.

That's all well and good for teaching content. But when it comes to teaching process, for example with my Reading With Purpose seminar, I have to walk a finer line. I've spent a lot of time with my own thoughts trying to deconstruct how I read papers, and what parts of that process are good and useful to convey, and what parts are just random choices.

I want to make sure my students are "haunted and troubled" by the material they read. I want them to learn how to question, be critical, and find their own way to interrogate the papers. I want them to do the same critical deconstruction of their own process as well.

On the other hand, the "stand-back-and-be-socratic" mode is very hard to execute without it seeming like I'm playing an empty game that I have no stakes in, and so I occasionally have to share my "answers". I fear that my answers, coming from a place of authority, will push out their less well-formed but equally valid ways of approaching the material.

I deal with this right now by constantly trying to emphasize why different approaches are ok, and try to foster lots of class discussion, but it's a struggle.

Note: I'm certain that this is a solved problem in the world of education, so if anyone has useful tips I'm eager to get pointers, suggestions, etc.