Mark Holthe 0:00
Welcome, everyone to another episode of the Canadian immigration podcast. This is a special episode. We're not talking today about a routine policy update, which is usually what we do. We're going to talk about what happens when artificial intelligence is inserted into immigration processing and the machine gets it wrong. Imagine submitting your permanent resin application, including your actual job duties, your employer documents, your work history, and then receiving a refusal based on a job description that is not yours. This isn't a small typo. It's not a minor understanding or misunderstanding. It's duties from an entirely different universe. And so today, joining me is immigration lawyer Luca vukolic, who practices out of St Catharines, Ontario, and the lawyer that represented the applicant in this bizarre Express Entry refusal. How you doing? Luca,
Luca 1:03
I'm doing well, Mark and yourself?
Mark Holthe 1:06
Great. Yeah, this is one of the most bizarre. And you know what? It's not surprising. Luca, it's not I think when we know what AI does in our day to day lives, as we experiment with it, we know that it has a tendency to hallucinate, and immigration has spent a lot of time trying to dissuade any of our concerns about AI kind of going sideways, but this is a perfect example of where their little system behind this black box is maybe not so cut and dry as they envisioned. So maybe we can walk through this, you know what actually happened in this case, a little bit. So who's the applicant? And I guess I should qualify this with the fact that it all stems from a Toronto Star article that was posted, you know, after this happened and and so you guys can go back and check that one out. But so a lot of this is already in the public domain, if you will. So yeah, share a little bit of insight on the applicant.
Luca 2:07
Certainly mark. So the applicant, my client, her name is Kemi, she's a French citizen, and she's a McMasters research associate, post graduate. She's a PhD from Sorbonne in Paris. Has been in Canada for several years now, three or four, and she's been a biology lab researcher studying respiratory illnesses using mice and various techniques to research How lung diseases can be transmitted and how they develop. Gotcha and she, along with her, her family, her husband and children, they were invited for the PR through the Express Entry program last year, I helped them with the profile. They got the invitation very quickly, since she has a very high score, being bilingual, having French and Canadian work experience, being very young, highly educated. Overall, she was a prime candidate and easily got invited.
Mark Holthe 3:14
Yeah, she's a slam dunk, right? Like, if we were to analyze getting itas, yeah, she's a slam dunk, you bet?
Luca 3:23
Yes. So in the high 570s and probably now 580s a year later. And so we submit last February, and for a while we don't hear very much. The usual biometrics medical checks are done. You get one request for an updated over the summer, an updated employment letter from McMaster. We reply to that, along with a few documents needed for her husband, and then we don't hear from them again for six or more months after that. So early September, we submit those extra documents, and we just start thinking, hold on, something strange is going on. January or February come, and still nothing. So it's already a little delayed, and so we're wondering what's happening.
Luca 4:11
And I suggest, and she agrees to send in a Privacy Act request to get a status update.
Luca 4:19
Two days later, she gets a refusal letter.
Luca 4:23
And the refusal letter is notable in a few different ways. I'd never seen anything quite like it before or since.
Luca 4:34
It's refusing her because it claims her job duties on the knock, namely, a
Luca 4:40
researcher, a professor and lecturer. That's the knot we chose
Luca 4:45
doesn't correspond with what she's actually doing, which we consider very strange, because the refusal letter gives a completely alternate job description involving robotics, mechanical devices. Uh, control units. Nothing to do with working in a biology lab.
Mark Holthe 5:05
We actually have a little copy of that. Why don't we pull it up now I'm just gonna have to pull you on yes screen, but let's show the viewers what we've got here. So I'm just gonna flip this around see if it wants to work here. Give me one second. I'm gonna add you in here, Luca, there we go. Okay, so here is the actual redacted letter that was that was issued February the 25th and this is what Luke is talking about here. After reviewing the employment letter, I found that your job duties were mainly technical and operational in nature and not related to the actual duties that you've listed in your knock essentially in your in your application. And you can see here, these are the duties that Luke is talking about, wiring and assembling control circuits, building control and robot robot panels to ESA standards, programming and troubleshooting PLCs interesting, integrating robotic equipment. So this is truly a different universe. That's kind of what I was alluding to earlier. And yeah, it's quite crazy. And one other thing I want to point out too Luca is the fact that within this process, she's a French candidate. So you said something that I think a lot of lawyers who do a lot of express entry will be very surprised at you indicated that you got a procedural fairness letter to update her employer letter. Well, all of my English candidates just get their applications refused having because they didn't prove their work history, but we know that they have a softer touch with French applicants, and so even in those circumstances, very
Luca 6:40
it was a very minor issue. There was no date on the top of the employer letter from February, but we just got a new one with the date in August, and that was it, yeah, and that would have resulted in a refusal on many previous English applicants. Luca, it's crazy, the standard so to have this happen to someone that was drawn under one of the Francophone draws, the French language draws, is crazy.
Luca 7:05
I believe it was CEC actually for her, yeah, she's straight up CEC because her points were so
Mark Holthe 7:11
even then, yeah, yeah, crazy.
Mark Holthe 7:16
So, yeah, so continue on. So you get this so,
Luca 7:20
yeah, so we get this letter, and I'm like, What's going on as I read through it, you can put it back on your screen, because by the time I get to the bottom of the page, something in you completely left out, which I'd never seen before, namely that generative AI was used in the creation of this processing of the application. Notice it says processing of application, not just writing the letter right. Or
Luca 7:50
if processing is a very broad term, then they say they're not used to make a recommended decision, right, just using the processing, whatever that means. So that immediately leapt out to me. How could this AI, whatever it was used? Did it have some factored effect on the decision? Did this visa officer just not look at her employment letter, and he had some system look at it for them,
Luca 8:19
or if they didn't, and the AI was used in some other fashion. It's very ambiguous. How was this used? And same on the bottom page as well, the same footnote, and similarly, it
Luca 8:34
just came out of nowhere, right? Yeah.
Luca 8:38
So obviously the decision here is they decide to immediately close it. I, of course, tell her, she's a little shocked, but we submit a reconsideration request first thing in the morning the next day,
Luca 8:52
I show exactly the employment letter I submitted the screenshots from my system showing the contents of the letter in the preview pane and the file name,
Luca 9:06
and showing, of course, that the file name that they uploaded, which of course, I could still access, is identical, so we know they got the right employment letter from McMaster. It was not some mistake on my end, and I have no clients who work in robotics anyway, so it wasn't me, it was all them.
Luca 9:27
We send in the reconsideration request, and as is prudent, I also
Luca 9:34
start the process of filing in the federal court, where
Luca 9:38
I more or less repeat everything but add in, of course, a great deal of detail about procedural fairness and the unreasonableness of this decision, the ambiguity of how the AI was used if it was
Luca 9:56
and I do tell some other lawyers I know.
Luca 10:00
Them, and they were the ones who put me in touch with the reporter at the Toronto Star.
Mark Holthe 10:08
So that's an interesting how it's all plays out. So you know, if we were to fast forward, I think most of us will realize exactly where this is going to go and where it did. So obviously, IRCC recognized pretty quick. The mistake this is somewhat embarrassing, especially when you have a Toronto Star article, so they're highly motivated to get it resolved. So within a short period of time, things moved forward. Positive
Luca 10:36
reconsideration went forward. The file was reopened on our end, and they started making requests for documents that will be needed for a landing, like an updated travel document from the husband
Luca 10:51
the rprf that they've been refunded that fee, they asked for it back again, and since then, they got their
Luca 11:02
confirmation email to register for the PR landing inland system, and so they've done that, and they registered that just yesterday, and hopefully the PR card shouldn't be in the mail in the next couple of weeks.
Luca 11:21
We also went through the court case, or I did just last week after, after it became obvious that yes, they were going to be approved. After all,
Mark Holthe 11:31
that's great. And when you think about this whole situation, you know, the real issue isn't, you know, simply that AI hallucinated. The issue is that the hallucination appears to have somehow survived this human review that they've indicated. And as we go forward, you know, as immigration lawyers, we we know what the government has has put out in their public facing explanations as to their AI strategy. You can go online, you can look at it, you know. And if I review a few, a few aspects of it. This is where I think us as lawyers need to really watch carefully. And I know some of our colleagues are
Mark Holthe 12:07
have really dive deep into this, and have been asking, you know, the right questions, and have been really pressuring the government, because, as the title says, for this special episode, it really is a black box and IRCC in their AI strategy. They say that it will be used to improve client service. Well, I don't know about that. Maybe it'll help with their efficiency, which is the next one, operational efficiency, program integrity. I don't know how it's going to improve program integrity and protection against fraud and cyber threats. Well, we've seen the fraud angle because their little AI is scrubbing through everything to look for any inconsistencies that they then will determine, wow, you used a different title when you applied for your study permit than you did in your Express Entry application. So you must be misrepresenting and Haha, we're catching fraud, right, these innocent mistakes. But in some cases, they see patterns. They see patterns of the same exact explanations on a variety of different immigration streams, where they start to question, wait a minute, how can these, you know, 20 people, all have the same story, you know, and so they are using it to to, you know, protect against that. And I have no reason to doubt, you know, that AI is helping with cyber threats. Heck, the the site, the folks that are that are engaging in the the
Mark Holthe 13:23
cyber activities, I guess, if you will, are using AI all over the place. So you fight AI with AI. Also in this strategy, they state that that AI principles include being human, centered and accountable, transparent and explainable, fair and equitable. I love these words, secure and privacy, protecting and valid and reliable. Okay, well, this calls all of this a little bit into question.
Mark Holthe 13:49
Yeah, and it says that it describes everyday uses, such as the following, creating summaries and producing documents and program uses such as identifying anomalies. So that's what we're talking about, scrubbing through the other applications, matching data, yep, making assessments, oh yeah, recommending options
Mark Holthe 14:12
and flagging straightforward low risk files for expedited officer decisions, which we see a lot within the temporary visitor visas and even sometimes study permit process. And then my favorite is that IRCC public position is that tools do not refuse or recommend refusing applications. So on paper, IRC sees AI strategy uses all the right words, my friend, accountability, transparency, fairness, reliability, but unfortunately, in the case of your client, it forces us to ask whether those principles are actually being delivered at the file level. So you know this whole concept of human in the loop, well, is it even enough? And like, what do we know? Luca about this? We don't really know anything. Now. Did they tell you?
Mark Holthe 15:00
Anything? Did they give you any responses when you you know as you went forward? So
Luca 15:07
here's the funny thing.
Luca 15:09
Here's the funny thing. There was the eight
Luca 15:14
tip requests that we'd already submitted before, and that was going through while we did the reconsideration, went to the court, and within two weeks, so unusually short time, we get the request for the notes back. And so this is a prime opportunity for me to check, was the AI being used in this fashion, was there some trace of it in the notes? I have, of course, looked at hundreds and hundreds and hundreds of notes over the years. And what struck me with these is, no, there didn't seem to be anything out of the ordinary. There were just slight differences in wording of the refusal letter, in the notes versus in the actual letter that my client got.
Luca 15:55
And it was slight bits of grammar, but it was the same, the same overall pattern, namely that it was a completely different Knock Code.
Luca 16:05
And
Luca 16:07
as this got out into the press, this was also helpful, because I did speak with someone from radio Canada, another reporter, and she got more of a direct line to IRCC, and from what they told her, translating from French, the only mistake was just by a human being who put in the wrong description of the job for the Knock Code that she was refused for. And so it was a human just writing out the wrong description. And then the AI just fixed up their letter.
Luca 16:41
So they are still denying that AI was used in any way to look at
Luca 16:46
the notes, and it was just a human error, and she was going to be refused anyway. Of course, now we know she's been approved,
Mark Holthe 16:55
so obviously something isn't matching there. What it is, I think we can only speculate. Yeah, it's not, it's not possible in any way to to believe that kind of an explanation. You know, some of the key questions that I have, and I'd love for IRCC to be able to respond to this. But you know, what does verified by an officer actually mean? You know, are, did the officer review the original documents at all, did the officer compare the generated summary against the actual evidence in the file? Like, I don't know how that would happen when you have job descriptions that are in completely different universes. And you know, was it copied from someone else's file? Like, where did those duties come from? Was it someone others, you know, another person's knock and the AI just got them mixed up, because it's all working from the same, you know, the same chat stream or whatever, within AI, and if the officer did verify the generated content, considering,
Luca 17:57
I was considering that perhaps someone else at McMaster somehow got mixed up with it. But I still don't see how that could be, because it's a big university, and you know, there's other areas of work. It really could be any number of them, honestly. And the fact is, it's just hard to say without knowing. I didn't know this from the notes that there was the
Luca 18:24
preview other offices, and like Vancouver and you know, Mississauga, a few other offices were involved, like the request back in last summer was made by some other office, and then they got the notes before they got the new documents, but then this one in February was apparently the Vancouver office that refused it,
Luca 18:49
despite the fact that the the letter itself says it's from their Ontario office. So there's a bit of a mismatch there too.
Mark Holthe 19:00
It's really hard. I would just speculate on what happened here. Someone made a mistake in February, and they refused her when they shouldn't have. Yeah, that, and it is pretty clear, right, you know? But I just, I just wonder, you know, this whole concept of being verified by an officer, if the officer verified this generated content. So yeah, they use AI to help generate documents which clearly would have it. As it says in the refusal letter, it was generated generative AI that that produced it. Well, okay, if How would a mismatch this, this obvious. How would it survive? And so the question isn't whether or not a human click the final refusal button, which is what IRCC tells us all the time. They don't make, you know, AI doesn't refuse. It's the officer who clicks it. The question is whether the human meaningfully reviewed the actual evidence. And we'll just have to let this, you know, be one example of where, you know, IRCC has got some.
Mark Holthe 20:00
Some questions to answer. I remember once Luca I had they came back to me and said, some of my clients passports were lost or they weren't submitted. They said, You should check your office see if the passports are there. I knew they weren't there. We submitted them, but that was in the day of paper processing, and so clearly what happened was the passports got put in someone else's application and sent to a different client for passport request. And in that world of paper, I could see you've got multiple files on your desk. I, you know, my early days of practice were with stacks of paper, and our immigration files were, you know, were thick, they were deep. And, you know, it's possible that papers kind of slide over as you're moving things around, and then you get things mixed up. But it's pretty hard to do that in a digital world, at least through human error. And so I think there's, I think personally, that you should, you should pursue some form of a privacy
Mark Holthe 20:56
a privacy request, to figure out if her data was actually mixed up or exposed to another, another client, and I would dig a little bit deeper and get a better explanation than whatever they gave to, you know, to the other reporter. I think your clients entitled to it, especially in these types of situations when you know, and they they're great at quickly fixing things that's that's awesome, and I'm so happy for your clients that you know the result that should have happened in the beginning is now nearing conclusion, with them obtaining their permanent residence. You know, the good Lord knows we need, you know, people that are involved in respiratory you know, medicine and our country needs individuals like her. So it's just so hard how, you know, trying to merge this, this world, that we don't know anything about this black box. So, yeah, so you know, aside, leaving aside the administrative fairness, things that we could have a whole podcast on, it's, it's quite interesting to see how this all played out. So do you have any, any thoughts or ideas or directions that you can give other council, if you know, if you know when through the course of your experience in this crazy file,
Luca 22:08
well, the first is to move quick,
Luca 22:12
prudently, but quick send in the restoration request in the web form. You have the proof of what you submitted and when you submitted it, you have the screenshots. You lay it out in your letter of submission, and bullet point by bullet point, here is the evidence that there was no fault on our end.
Luca 22:36
You, of course, preserve the right to go to the court. There's that 15 day limit for the federal court. And my client did agree to that. You know, she knew there would be a little bit of an extra fee to do that, but she was completely in agreement with them, of course, for the reconsideration request, of course, that was just included with my old retainer. I wasn't going to charge her for something this obvious, but for the court case, in the court filing, there was a little bit more
Luca 23:05
involved. The other thing would be,
Luca 23:07
you know, tell your client, of course, and also, to the extent you can talk to other counsel, they're the ones who put me in touch with the reporter. Some of them have made this AI Working Group in response to the government's use of this, as well as their own
Luca 23:27
press releases about how they're using it. And they were the ones who put me in touch with the reporter, and they gave me some tips, including on you know, how to write out the federal court filing, the A application for leave and do this review, which I'd only ever done one before, right in a completely on the related matter involving a study permit. And so I was very much not experienced in the litigation side, so that was key for me to have that support. And of course, once you are talking to the reporter, obviously you tell them to what extent you can you inform your client. Here's the pros and cons of going to the press. It can, of course, bring great deal of good publicity, but you you know you don't need to tell them anything that you don't want you know. Keep your keep your confidentiality to what extent you feel comfortable. That's why I told them. And in terms of my office, I'm just, you know, carefully looking at every other letter I get so far, I haven't seen any other footnotes about generative AI, thankfully, it's because there haven't been any. I haven't had refusals since then, so that's the good news. But even in routine requests, I'm hoping that it won't show up.
Luca 24:46
Biometrics, medicals, those are all very routine. They've been done for decades now without any major change, and I don't see any change for them, procedural fairness, letters and ambiguity.
Luca 25:00
And job duties, that's where I think I see it being the temptation to use it because more difficult files, more complex files that take much more of the officers time at ircdc, those are the ones that they are obviously going to look at to see if they can fast track in some sense, and speed up what would be a slow process. And so that's where I would be most directly concerned. Or, for example,
Luca 25:33
students who had their former students who are applying for PR way back in their study permit and PGWP, they may not have mentioned some work experience, and now they're claiming points for that work experience, right? That would be another situation where
Luca 25:53
you have to tread carefully. And you know, they might set up some system to just automatically flag any discrepancy in employment, right? Anything involving the
Luca 26:06
nuance or human judgment, that's where I would I would be looking at it less so, like language test scores, right? That's just numbers, right? You just input the number and it's either above or below a certain pressure.
Mark Holthe 26:21
Yeah, that's perfect. You know, as I circle back a little bit, just on this whole world of AI utilization, and I've got lots of questions, and I know that, like I said, Our colleagues are digging and trying to get a little bit more transparency
Mark Holthe 26:38
in IRCC zone their own public facing documents, they say that the rules used by certain tools in how they use AI are kept internal to prevent fraud and protect program integrity, while also saying that these tools are reviewed routinely. So my view is, well, I recommend that you IRCC review these particular rules and how this this refusal arose, you know, if generative AI,
Mark Holthe 27:09
if it prepared or somehow factually, you know, affected the summary that that was generated, at least in this refusal letter, should the applicant be told exactly what was generated, you know, should AI and these, you know, summaries, these AI generated summaries. Should they be part of the GCMS notes, you know, should we be should Council be able to to request the the prompts and the model outlines and the officer verification steps, like, I think, all of that transparency, if you really want to be transparent, and they kind of hide behind this program integrity thing, which has always been the case, you know, but if it's buried in some black box or semi transparent workflow, well it kind of makes it difficult for proper and this is where the administrative law comes in, you know, knowing the case to meet and being able to properly assess or address when things like this happen. And I'm so grateful that you came on Luca, because, you know, it's about awareness, too, and sometimes this council, we're not aware of what's going on until we hear of something bad that's happened to one of our colleagues and their clients. And so I, you know, as I was thinking about this topic, I wanted to leave our viewers with just a few little tips, just like you did, and I echo some of the ones that you did, but just takeaways. You know what we as practitioners should do now? And you know one of the things that we can do, although we can never make it AI resistant, right? You can't, you can't prepare a file in such a way that you're going to avoid what happened here. They're just fabricating duties that you never included. Well, that's a whole different universe, like I continue to say, but using clear labels and concise summaries and structured evidence. You know, when you're when you're submitting these applications that are so regimented, there still is room for us to put things in a way that makes it as easy as possible for the officers to understand what's happening. You know, I have something that I call and I put in my DIY course, and I use with our clients, called My knock selection tool, which is essentially, it's a, it's it's a duty matching table, essentially that maps the actual duties with those in the knock. So if an officer is actually looking at an application, we're making the job as easy as possible for them, because we're pulling out the duties from the reference letter itself and lining them with the not code. And so if they're saying, Oh, your duties don't match with these, you know, they can say, wait a minute, none of the duties I have in my refusal letter here match with this little summary. And then you mentioned the importance of not submitting vague employer letters. But boy, they are Cru they like obviously, with anyone who speaks French, apparently, because I've had people come to me after they've had applications refused because there wasn't a dated letter, they didn't give them a chance for to update the letter with the employer. Of course, we keep copies of everything, so we make sure that you know that whatever we've submitted, we've got a reference point to identify when.
Mark Holthe 30:00
There's something like this that happens that's completely out to lunch. And so we know with Express Entry, and I think many of our like, there's a little, a little URL that you can use to hack in and take a look at the eapr, so you can actually look at the information. It always seems to change. It's changed twice in the last 10 years, but, you know, but even making sure that we keep copies and retain copies of everything that's happened. Because once they're uploaded, you can't look at them. And when you get a refusal letter like this, you if you're going to go for a leave application, you've got a very limited period of time, 15 days, not enough time to request, you know, an access to information and get the actual documents. And then, of course, you know, you've did exactly what, what most of us would do, which is request the an ATIP to obtain the GCMS notes to figure out if there were, you know, if the officer has got things in there that's just out to lunch and like you identified before, just acting quickly, because there's only so much time. So I guess ultimately, you can't assume that, because the refusal letter says an officer verified the content, the decision is automatically reasonable. So so there we go. I'll forego telling IRCC what they need to do now, although I had some tips for them too, I'll save that maybe for my presentation at the Canadian Bar Association's national conference in Ottawa, here at the end of May, I'll be speaking there with one of the head officials over express entry, and we touched on this a little bit in our calls preparing for the for the presentation. But obviously he did not have any information or wish to to discuss this in any degree of detail, but it's a reality, and I know these you know, the use of AI will definitely come up at our conference. So, so there we go.
Mark Holthe 31:45
Yes, it'll be fun. All right, Luca, thank you so much for joining me today. This was a great episode, and for those that have tuned in, we hope that this was beneficial, and it shares a little bit of light on what's happening within Express Entry processing, and we will see Luca what the world looks like as they move away from at least our current style of PR processing, merge application, you know, these, these programs together, and we'll have to see how this all plays out as they as they roll out the new method for for basically assessing economic permanent residence for Canada. So thanks for joining and take care and we'll we'll see you again soon.
Luca 32:25
Thank you, Mark.
Transcribed by https://otter.ai
