We've used what we call the "Kobayashi Maru" coding exercise with developers.
It was code deliberately filled with various errors, from JavaDoc with @param names not matching argument names, to generic issues, to more subtle issues. One of the last ones involved threading and concurrent access. It also had a test suite to help candidates gauge their progress for programmatic issues, not so helpful for code style issues (that JavaDoc comment).
Basically, we wanted to see the dev a) look at the reds and yellows in their IDE, b) run the tests, and then c) prioritise the issues. And d) say "these tests are wrong"
It was a highly effective filter for people with "qualifications" from dubious private training institutes (one candidate fixated on the red Javadoc for a mismatched parameter name, we hinted that he check the arguments in the method, he muttered "method, method, method..." to himself while scrolling down the code looking for the actual word 'method' - spoiler, he didn't find it, and he didn't get the job), but we left it behind when we found it was causing candidates genuine anxiety when they couldn't get all the tests to pass - which was intentional - anyone who figured out that the tests were deliberately wrong was an instant hire.
But we felt it was unfair, it's very very hard in a job interview to say "hey, I think your tests are wrong" so we dropped it.
These days, we use a simple filter - write code in your preferred language to find the element in the list that only has one occurrence. It's surprisingly effective as a first pass. People with years of Scala on their resume from writing Spark jobs, who choose Scala, but who can't quite get the hang of how you define an array of X in Scala... when you find yourself telling them to use square brackets for generics instead of angle brackets, you learn a lot about their actual capabilities vs. their projected.
Maybe if prefaced with something like "anything in this project, including tests may be buggy, your job is to make everything work as intended", it would be less of a test of "guts to speak up on interview" and more just technical challenge. Of course that guts may be a value in itself..
The tests are wrong 70% of the time. Either a silly mistake, the design wasn't conceptualized properly (or was changed), or just plain bad code.
If you're tackling your suite one test at a time, and committing code that solves just that test, seeing it fail and then pass, and thinking about why, everything will be fine. If you batch things, or don't debug success, you'll ultimately be buggy.
It was code deliberately filled with various errors, from JavaDoc with @param names not matching argument names, to generic issues, to more subtle issues. One of the last ones involved threading and concurrent access. It also had a test suite to help candidates gauge their progress for programmatic issues, not so helpful for code style issues (that JavaDoc comment).
Basically, we wanted to see the dev a) look at the reds and yellows in their IDE, b) run the tests, and then c) prioritise the issues. And d) say "these tests are wrong"
It was a highly effective filter for people with "qualifications" from dubious private training institutes (one candidate fixated on the red Javadoc for a mismatched parameter name, we hinted that he check the arguments in the method, he muttered "method, method, method..." to himself while scrolling down the code looking for the actual word 'method' - spoiler, he didn't find it, and he didn't get the job), but we left it behind when we found it was causing candidates genuine anxiety when they couldn't get all the tests to pass - which was intentional - anyone who figured out that the tests were deliberately wrong was an instant hire.
But we felt it was unfair, it's very very hard in a job interview to say "hey, I think your tests are wrong" so we dropped it.
These days, we use a simple filter - write code in your preferred language to find the element in the list that only has one occurrence. It's surprisingly effective as a first pass. People with years of Scala on their resume from writing Spark jobs, who choose Scala, but who can't quite get the hang of how you define an array of X in Scala... when you find yourself telling them to use square brackets for generics instead of angle brackets, you learn a lot about their actual capabilities vs. their projected.