Despite advances in AI, machines still have limitations in accomplishing tasks that come naturally to humans. When AI systems are fielded in the open world, these limitations cause concerns around reliability, biases and trust. In this talk, I will argue that hybrid systems that combine the strengths of machine and human intelligence is key to overcoming the limitations of AI algorithms and developing reliable systems. In the first part of the talk, I will present techniques that can guide human labeling efforts for efficient discovery of blind spots of machine learned classifiers.
