Chermaine Leysner was one of the tens of thousands of Dutch citizens whose life was ruined by the “toeslagenaffaire”–the child care benefits scandal. One day, in 2012, she received a letter from the Dutch tax authority demanding she pay over €100,000 (~$105,000 USD) for child care allowance dating back to 2008. A student and a mother of three children at the time, Leysner spiraled into depression and burnout–caused by the stress of the tax bill.

“I thought, ‘Don’t worry, this is a big mistake.’ But it wasn’t a mistake. It was the start of something big,” Leysner said.”

What was the issue?

Dutch tax authorities had implemented a self-learning algorithm to generate risk profiles to identify child care benefits fraud. Authorities relied too heavily on the system’s risk indicators and punished families over a mere suspicion of fraud.

As a result:

• Tens of thousands of families, particularly those that ethnic minorities or with lower incomes, were pushed in poverty
• Victims committed suicide
• More than a thousand children were taken into foster care

Taking a closer look at the algorithm

Dutch tax authorities developed the criteria for the risk profile factors. These indicators include:

In addition to these criteria, authorities also devised a secret blacklist that tracked credible and substantiated signs of fraud. Citizens had no ability to understand why they were on the list or defend themselves.

Furthermore, a report from the Dutch Parliament found several institutional biases of the tax agency and that tax authorities hid information.

“There was a total lack of checks and balances within every organization of making sure people realize what was going on,” said Pieter Omtzigt, an independent member of the Dutch parliament who helped uncover the child care benefits scandal.

What does this mean for the future?

Omtzigt said he worries that the Dutch government hasn’t “taken even vaguely enough preventive measures” to prevent a future scandal.

As governments around the world increasingly turn to algorithms and AI-automated systems, the Dutch scandal, Toeslagenaffaire, demonstrates how, without the proper measures and safeguards, automated systems can have devastating consequences.

The European Union plans to introduce the AI Act, an ambitious and encompassing law that seeks to restrict the use of “high risk” AI systems and ban certain “unacceptable” uses. To learn more about the AI Act, here’s a guide from MIT Tech Review.

Link to original article published 10/12/2022