Algorithms are increasingly assisting or replacing people in making important decisions. Today, algorithms help decide who gets hired, how much to charge for insurance, who gets approved for a mortgage or a credit card. They also inform choices about sentencing, parole, and bail. We tend to hear about these algorithms when they mess up -- when they offer women less credit than men, or make it harder for people with mental health status to get jobs, or treat black defendants more harshly than white ones.
Whether made by people or algorithms, these are hard decisions. Sometimes they will be wrong. But there is no excuse for an algorithm to be racist, sexist, ageist, ableist, or otherwise discriminatory.
What We Do
ORCAA is a consultancy that helps companies and organizations manage and audit algorithmic risks. When we consider an algorithm we ask two questions:
What does it mean for this algorithm to work?
How could this algorithm fail, and for whom?
Often we ask these questions directly with companies and other organizations, focusing on algorithms they are using. We also ask them with regulators and lawmakers in the course of developing standards for algorithmic auditing, including translating existing fairness laws into rules for algorithm builders. No matter the partner, our approach is inclusive: we aim to incorporate and address concerns from all the stakeholders in an algorithm, not just those who built or deployed it.
Services
Algorithmic audit
The audit is a deep dive investigation into an algorithm in context. The goal is to identify risks related to the use of the algorithm, with an emphasis on fairness, bias, and discrimination risks. It comprises 3 phases over 2-4 months:
Fix the Context. Define a specific use case to audit and gather background information about the algorithm.
Build the Ethical Matrix. Interview representatives of stakeholder groups - both internal and external, technical and nontechnical - to elicit their concerns. Map these concerns using our Ethical Matrix framework.
Prioritization and Guidance. Identify the most urgent concerns and provide recommendations about how to address each one. These usually include measurement (e.g., is this potential concern actually happening, and how much?) and mitigation (e.g., if we verify this is happening, how do we fix it?).
Common extensions to the audit engagement include:
Implementation assistance. Help carry out analyses, measurements, or other of the recommendations delivered, or review the client’s own implementation to assess whether they addressed the corresponding stakeholder concern.
Public communications. Help clients talk publicly about the audit, including the process, its findings, and steps the client has taken - or has committed to take - to address the concerns identified.
Fairness testing with Pilot
We offer Pilot, our patent pending (filing no: 17/804,018) cloud-based software platform that performs disparate impact style analyses on a client’s own data to investigate whether a given algorithm produces different outcomes across race/ethnicity or gender groups. Pilot can perform these analyses even if the client does not have gender or race/ethnicity data on its customers, by using privacy preserving inference techniques. Since it presupposes a focus on race and gender, Pilot’s analysis is narrower than a full algorithmic audit; but it may be appropriate for clients navigating compliance in regulated industries.
The process takes up to six weeks, depending mainly on the time required to pull and upload data from the client’s system. It comprises three phases:
Preparation. Learn about the algorithm and context, and provide instructions for the data the client will need to supply for analysis.
Data pull and upload. Client gathers the requisite data from its systems, while we provide support as needed. When ready, client uploads data to Pilot.
Analysis and presentation of results. Analysis is performed on Pilot, and we deliver test results. Note that phases 2 and 3 can be repeated for subsequent rounds of testing.
Early warning system
Do you worry that your organization’s algorithms in development or in production are problematic? We can tailor an "early warning system" that gives you advance warning for such problems, raising questions and ethical or legal issues to the correct committee.
Vendor vetting + procurement assistance
We help organizations perform stronger due diligence when procuring AI or predictive technologies from third parties. For instance, we can raise potential issues in advance, prepare questions and requests for vendors, and review materials provided.
Workshops + education
We offer workshops that give participants hands-on experience with our auditing framework and process. We also give talks and trainings on algorithmic auditing and fairness.
Expert witness work
We assist public agencies and law firms in legal actions related to algorithmic discrimination and harm.
Bespoke consulting
We help organizations prepare for the age of algorithms by
Creating strategies and processes to operationalize fairness as they develop and/or incorporate algorithmic tools in their operations
Working with industry regulators to translate fairness laws and rules into specific standards for algorithm builders
Industries + Clients
Our partners range from silicon valley to the DMV, including private and public companies of all sizes and public agencies, in the US and internationally. We have a special interest in algorithms being used for credit, insurance, education, and hiring – because we think algorithms in these sectors are under relatively little scrutiny, but have major impacts on people’s lives.
Past and current clients include:
Explainable Fairness
When is an algorithm fair?
We propose the Explainable Fairness framework: a process to define and measure when an algorithm is complying with an anti-discrimination law. The central question is, Does the algorithm treat certain groups of people differently than others? The framework has three steps: choose a protected class, choose an outcome of interest, and measure and explain differences.
Example from hiring algorithms
Step 1: Choose a protected group. Fair hiring rules prohibit discrimination by employers according to gender, race, national origin, and disability, among other protected classes. So all these could be considered specific groups for whom fairness needs to be verified.
Step 2: Choose an outcome. In hiring, being offered a job is an obvious topline outcome. Other outcomes could also be considered employment decisions: for instance, whether a candidate gets screened out at the resume stage, or whether they are invited to interview, or who applied in the first place, which might reflect bias in recruitment.
Step 3: Measure and Explain Loop. Measure the outcomes of interest for different categories of the protected class. For example, are fewer women getting interviews? If so, is there a legitimate factor that explains that difference? For example, are men who apply more likely to have a relevant credential or more years of experience? If so, account for those legitimate factors and remeasure the outcomes. If you end up with unexplained large differences, you have a problem.
The process can be applied more generally, and looks like this:
Bias Audits for NYC Local Law 144
The new NYC law Int 1894-2020 (“Local Law 144”) requires annual bias audits of all automatic hiring decision tools used to hire candidates in or from NYC.
We offer a Bias Audit service to help companies comply with this law. We use our Pilot analysis platform to conduct disparate-impact-style and other analyses on real-world data arising from a specific use of a given decision tool. To show what we mean, here is a mock Bias Audit report for a fictitious company NewCo, which is using fictitious ToolX in its hiring process.
We can conduct a Bias Audit whether you are a vendor building hiring tools to be used by others, or a company using an AI tool in your own hiring process. If you already have race/ethnicity or gender information about candidates, we can use it in the audit; if you do not, we offer inference methods to model this information.
Please contact us to learn more about getting a Bias Audit.
Principles
Context Matters
An algorithm isn’t good or bad per se – it is just a tool. Who is using it, and to what end? What are the consequences? We go beyond traditional accuracy metrics to ask how the algorithm is used to make decisions, and who is affected.
Putting the science into data science
Graphs and statistical tests can be useful – if they answer a meaningful question. We translate specific ideas and concerns about fairness into statements we can test empirically. Then we design experiments to say whether those standards are being met.
AI Ethics cannot be automated
There cannot be a universal algorithmic auditing checklist or a fully automated review process – context is too important. An audit for a specific purpose may be repeatable, but human discussion and judgment will always be essential. We will tailor a tool to your needs, not repurpose what worked in another setting.
Contact
We are ORCAA
Cathy O’Neil
CEO
Cathy has been an independent data science consultant since 2012 and has worked for clients including the Illinois Attorney General’s Office and Consumer Reports. She wrote the book Doing Data Science in 2013 and Weapons of Math Destruction: How Big Data Increases Inequality And Threatens Democracy, released in September 2016.
Tom Adams
COO and General Counsel
Thomas Adams has over twenty-five years of business and legal experience. He has represented banks, companies and individuals on corporate, securities and business law matters. He also provided strategic advice, litigation support and expert witness testimony on issues relating to the financial crisis. Mr. Adams is an expert in creating solutions and solving problems for complex financial and corporate transactions and has provided strategic advice and analysis to banks, insurance companies, private equity companies, hedge funds and a variety of other companies. He graduated from Fordham Law School in 1989 and Colgate University in 1986. He is admitted to practice in New York.
Jacob Appel
Chief Strategist
Jake is ORCAA’s Chief Strategist. He conducts algorithmic audits, and specializes in designing tests and analyses to assess the performance of algorithms and their impacts on stakeholders. Before joining ORCAA he worked with the Behavioral Insights Team, where he advised state and local governments on incorporating behavioral science “nudges” into citizen-facing policies and programs, and testing them with randomized experiments. Jake received a BS in mathematics from Columbia University and an MPA from Princeton School of Public and International Affairs. He coauthored two books: More Than Good Intentions: How a new economics is helping to solve global poverty, and Failing in the Field: What we can learn when field research goes wrong.
Meredith Broussard
Affiliate
Data journalist Meredith Broussard is an associate professor at the Arthur L. Carter Journalism Institute of New York University, research director at the NYU Alliance for Public Interest Technology, and the author of “Artificial Unintelligence: How Computers Misunderstand the World.” Her academic research focuses on artificial intelligence in investigative reporting and ethical AI, with a particular interest in using data analysis for social good. She appeared in the 2020 documentary Coded Bias, an official selection of the Sundance Film Festival that was nominated for an NAACP Image Award. She is an affiliate faculty member at the Moore Sloan Data Science Environment at the NYU Center for Data Science, a 2019 Reynolds Journalism Institute Fellow, and her work has been supported by New America, the Institute of Museum & Library Services, and the Tow Center at Columbia Journalism School. A former features editor at the Philadelphia Inquirer, she has also worked as a software developer at AT&T Bell Labs and the MIT Media Lab. Her features and essays have appeared in The Atlantic, The New York Times, Slate, and other outlets.
Şerife (Sherry) Wong
Affiliate
Şerife (Sherry) Wong is an artist and founder of Icarus Salon, an art and research organization exploring the societal implications of emerging technology. She is a researcher at the Berggruen Institute where she focuses on the data economy for the Transformations of the Human program, serves on the board of directors for Digital Peace Now, and is a member of Tech Inquiry. She has been a resident on artificial intelligence at the Rockefeller Foundation Bellagio Center, a jury member at Ars Electronica for the European Commission, and frequently collaborates on AI governance projects with the Center for Advanced Study in the Behavioral Sciences at Stanford. Previously, she created the Impact Residency at Autodesk’s Pier 9 Technology Center where she worked with over 100 leading creative technologists exploring the future of robotics, AR/VR, engineering, computer-aided machining, and machine learning for product development, and worked at the Electronic Frontier Foundation.
Betty O’Neil
Affiliate
Betty O’Neil (really Elizabeth) is a computer scientist specializing in database internals, and is also interested in how computers can be used to make the world a better place. Like her daughter Cathy, she earned a PhD in Mathematics (Applied in her case) at Harvard to get started. She was a professor at the University of Massachusetts Boston for many years, and now is joining ORCAA’s efforts in using data science in socially responsible ways. She is a co-author of a graduate database textbook. As a database internals expert, she has helped implement several important databases, including Microsoft SQL Server (two patents in 2001), and more recently, Stonebraker’s Vertica and VoltDB. She is a lifelong nerd and can program anything.
Deborah Raji
Affiliate
Deborah Raji is an affiliate at ORCAA. She has worked closely with the Algorithmic Justice League initiative, founded by Joy Buolamwini of the MIT Media Lab, on several award-winning projects to highlight cases of bias in facial recognition. She was a mentee in Google AI’s flagship research mentorship cohort, working with their Ethical AI team on various projects to operationalize ethical considerations in ML practice, including the Model Cards documentation project, and SMACTR internal auditing framework. She was also recently a research fellow at the Partnership on AI, working on formalizing documentation practice in Machine Learning through their ABOUT ML initiative, as well as pushing forward benchmarking and model evaluation norms. She is a Mozilla fellow and was recently named as one of MIT Tech Review’s 35 Innovators Under 35. She is currently pursuing a Ph.D in Computer Science at UC Berkeley.
Anna Zink
Affiliate
Anna Zink is a principal researcher at Chicago Booth's Center for Applied AI where she works on their algorithmic bias initiative. Her research is focused on algorithmic fairness applications in health care, including the evaluation of risk adjustment formulas used for health plan payments. Before receiving her PhD in Health Policy from Harvard University, she worked as a data analyst at Acumen, LLC. where, among a small team of analysts, she partnered with the Department of Justice on cases of Medicare fraud, waste, and abuse and helped develop fraud surveillance methods.
Emma Pierson
Affiliate
Emma Pierson is an assistant professor of computer science at the Jacobs Technion-Cornell Institute at Cornell Tech and the Technion, and a computer science field member at Cornell University. She holds a secondary joint appointment as an Assistant Professor of Population Health Sciences at Weill Cornell Medical College. She develops data science and machine learning methods to study inequality and healthcare. Her work has been recognized by best paper, poster, and talk awards, an NSF CAREER award, a Rhodes Scholarship, Hertz Fellowship, Rising Star in EECS, MIT Technology Review 35 Innovators Under 35, and Forbes 30 Under 30 in Science. Her research has been published at venues including ICML, KDD, WWW, Nature, and Nature Medicine, and she has also written for The New York Times, FiveThirtyEight, Wired, and various other publications.