Services

Algorithmic Audit

A comprehensive assessment of risks associated with a specific use case of an algorithmic system. We audit systems of all kinds, including generative AI, automated decision systems, predictive models, and facial recognition.

  • Uses our Ethical Matrix framework

  • Identifies high-priority issues around fairness and performance

  • Delivers recommendations for measuring and mitigating risks

  • Outputs / Deliverables: Algorithmic Audit Report

Pilot: Quantitative testing for regulatory compliance and more

Analysis to measure bias in algorithmic systems, using data from live deployments or test data.

  • Proprietary, patent-pending cloud based analysis platform

  • Incorporates inference methodologies, so we can measure gender and race/ethnicity bias even if you do not possess this data

  • Double-firewall privacy protection: we never see personally-identifying information; client never sees individual inferences

  • Outputs / Deliverables: Bias Audit Report (e.g. for NYC Local Law 144); custom analysis reports

AI Governance + Risk Management Consultation

We help build core infrastructure to help you use AI responsibly.

  • Develop an organization-level approach to measuring and managing the risks of deployed AI systems, especially with regard to bias/discrimination

  • Review and recommend governance policies, processes, and structures 

  • Assistance with procurement of AI technologies from vendors and other 3rd parties

  • Outputs / Deliverables: Documentation of risk management structures; diligence reports on prospective vendors

Cockpit Design

Running a deployed AI system is like piloting a plane: to fly safely, you need critical information in real time. To make an effective cockpit, you must understand what can go wrong in flight, and include dials/gauges that monitor for those risks. We help build cockpits for your AI systems.

  • Identify and/or construct metrics that address key risks

  • Calibrate thresholds and develop mitigation tactics 

  • Can work with existing analytics platforms or assist with your custom implementation

Education + Training

We teach people how to think critically about AI and algorithmic systems, how to identify blind spots, and about the craft of auditing.

  • Workshops

  • Bespoke training for audit and risk management teams

ORCAA is a consulting company that helps companies and organizations manage and audit their algorithmic risks.