EFF sues Proctorio on behalf of student falsely DMCA'd • Hacker News Board

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#451

Post by mkl » Thu, Apr 22, 2021, 9:52 PM UTC

cycomanic wrote: post #120

mkl wrote: post #51

Earlier quoted context omitted.

Copy-pasting from my other comment: > I see it as a currently-necessary annoyance, as the least bad option. The alternatives have greater deficiencies: human invigilators using Zoom etc. don't scale (institution experience); oral exams don't scale (my experience moderating such assessments); no invigilation leads to cheating (by few students but enough to be a real problem, especially for professional qualifications…

I find your statements in different parts of the thread quite contradictory. On one hand you say that things don't scale because of number of students, but then a bit down thread you mention it's not worth making a new course for 10 students (out of 640), but for 10 students in person proctoring is clearly not a problem. You also mention the issue of cheating for professional qualifications, but again, that's a tiny…

Well I'm talking about decisions about different things by different people 1) running exams (institution level - run centrally), 2) creating bespoke purpose-designed online courses (mostly department level). The decision about exam invigilation was made at the institutional level, where 10 is a small number of online students for a course, and they have dozens of courses to worry about. For most of the institution's courses, other solutions don't scale, and for the institution as a whole they definitely don't. The course with 10 online students may only run online this year, and it takes hundreds of additional hours to make a purpose-designed online course (we're doing that for other courses where the material will be reused).

The oral exams I moderated didn't seem like they could scale past 20 students. The reason our online students are online is almost always because they are overseas, and the majority of them have limited English. In that context, orals are discriminatory, stressful, and very slow.

I'm in NZ. Our universities have mostly been operating in person as normal (i.e. costing just as much) with far fewer international students (who bring in money), except in addition we have comparatively small numbers of resource-intensive online students as well, and are attempting to bring parts of entire degree programs online for the first time.

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#452

Post by ska » Thu, Apr 22, 2021, 10:00 PM UTC

twic wrote: post #449

ska wrote: post #443

Earlier quoted context omitted.

> America is leaking again. Ok, from a terminology point of view that's fair - it isn't the same everywhere by any means. However, most if not all of the graduate programs I know if internationally have something roughly equivalent, whether they are comps or prelims or qualifying or whatever. The basic idea is that a department (and university, generally) has an interest in maintaining the quality of their programs,…

No UK PhD programme has exams like this, that i have heard of.

Oh, good point, the UK programs I know of are notably ligher in this regard (and PhD shorter) though they do generally require a 1st class honors (honours, i guess!) degree in subject, which includes "tripos" which is roughly equivalent. So in some ways a higher bar for undergrad matched with a lower bar for grad.

At least that's the theory - I don't know if in practice it holds up; most of the grad students and later I knew from that system came from oxbridge which has a number of quirks.

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#453

Post by g_p » Thu, Apr 22, 2021, 10:03 PM UTC

Have been overseeing some exams being run at a well regarded university. No remote proctoring software (i.e. spyware) is in use. Students are taking their exams in the web browser of their choosing, on the platform of their choosing. They aren't sat dialled into a video call or anything else (but they have a link for a backup one in case they have issues or need clarifications on a typo etc.)

It's straightforward - a well-designed examination should allow for adequate distinction between students, allowing everyone (who studied the course and learned) to show basic knowledge, and those who have more advanced understanding to demonstrate this.

In designing assessments, questions were peer-reviewed to ensure they are not "easily googleable". They were designed to focus not on asking "what is X?" but on "tell us a way that Y could be achieved" or "give an example of how you would do Z". These questions are pretty hard to google, and time pressure makes it harder still.

Focusing on understanding, and applying knowledge really seems to be the right way to design an exam. I can say from what I've seen and heard from colleagues so far, this approach is giving equal (if not more) differentiation among students than usual - the good students still perform well, and the poor students still perform poorly. And personally (from experience), I find it a more enjoyable to sit an exam that asks you to answer meaningful questions, than one that simply expects you to memorise and recite facts back. We're not bothered if you memorise the name - just describe how you'd solve the problem.

From the number of people doing poorly even on fairly straightforward questions, I'm not hugely concerned that giving the exam online made any significant difference. A non-trivial number of students didn't even complete a mandatory question (which was clearly marked), so I assume they found it suitably challenging, even with access to the world's knowledge at their fingertips. I'd say that's a good exam.

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#454

Post by filoleg » Thu, Apr 22, 2021, 10:25 PM UTC

steelframe wrote: post #440

filoleg wrote: post #328

Earlier quoted context omitted.

From my personal experience (and that of many friends of mine) who interviewed at Google, you won't be asked about memset or any C-related question unless you stated you were proficient in C and picked C as your interviewing language. Knowing this context might actually explain the situation that the parent comment describes a bit better. It is absolutely possible, as you described, to be perfectly capable within you…

> you won't be asked about memset or any C-related question unless you stated you were proficient in C and picked C as your interviewing language I might not ask about C-specific context, but if you don't know how to write a value to a sequence of bytes, you have no business being a SWE at Google.

>I might not ask about C-specific context, but if you don't know how to write a value to a sequence of bytes, you have no business being a SWE at Google

I know quite a few engineers who passed Google interviews and work there, and they wouldn't know how to do this off the top of their head. I also interviewed with Google a few times, and not once was I asked a question like this.

Also seems weird you would say that they have "no business being an SWE at Google", given that this problem is pretty trivial, and any competent engineer would be able to figure it out after some quick googling. It isn't some difficult algorithmic problem, it is a very specific and small piece of trivia.

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#455

Post by kelnos » Thu, Apr 22, 2021, 10:33 PM UTC

fxtentacle wrote: post #5

In case you don't know who Proctorio is: (like me) Apparently it's a supervision software that students are forced to install on their private computer and (as expected) it'll do its worst to invade your privacy and flag "suspicious" things, based on which the university might punish you. "Suspicious" here means wearing glasses [3] or looking around in the room or blinking too much [4] or having eye and/or skin color…

From your Insider link:

> LSU student body president Stone Cox said that the fees, which could come out to $300, were prohibitive for students.

What the hell? Not only are universities mandating students infect their systems with malware, but they're making the students pay for the privilege? That's ridiculous.

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#456

Post by g_p » Thu, Apr 22, 2021, 10:47 PM UTC

ska wrote: post #260

samkater wrote: post #248

Earlier quoted context omitted.

I would add some nuance to this view. For nearly every undergraduate course I agree - the ability of knowing where to look for resources in a particular subject matter, synthesizing the available information to solve a reasonably novel problem, and presenting the information in a coherent way is a skill/art that should be one of the primary goals of the education. For advanced degrees, though, I'm not so sure. I expe…

This is why PhD's typically have comprehensive exams. It varies (a lot!) by university and even department, but panel led oral exams are pretty common. At some places they are even open to the public (no pressure). These are distinct from a thesis defence, which happens at the end of your degree; they are likely to happen about a year in and are meant to ensure you have a solid background. Overall I think an oral exa…

As you say, different departments and institutions have different approaches, but I'd definitely emphasise that outside of Europe (where I've seen more ceremonial "exams"), there are still robust and vigorous "final" oral viva exams used.

In some European universities, the outcome is such a foregone conclusion that the candidate's family is laying out the buffet, peeling the cling film off the plates, and unboxing the champagne bottles as the "defense" begins.

In the UK however, I've never seen this. It's generally a small room with candidate, an internal examiner, an external examiner from another university who is an expert in the field, and a convenor to record the minutes of the examination. The outcome is by far from a foregone conclusion.

A good student who is an expert in their field, is well-read and up to date on their work and the surrounding literature will perform well and have little to fear. Someone who hasn't written their own thesis, or didn't really have an understanding of the area, and thus isn't really an expert, will have a very unpleasant time, and will likely be failed, or be sent away with major corrections to be completed to the satisfaction of the examiners, possibly including a full oral re-examination.

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#457

Post by whimsicalism » Thu, Apr 22, 2021, 10:47 PM UTC

steelframe wrote: post #439

whimsicalism wrote: post #274

Earlier quoted context omitted.

Perhaps people get stressed in interviews? For instance, I don't code in C that often, and if asked, I'm not sure I could remember off the top of my head if the function signature is `memset(dst, value, n)` or `memset(dst, n, value)`. My guess from intuition is the first, but I would be hemming and hawing a bit if asked in an interview something like that. Also, many CS programs are easy enough that you'll never even…

> Perhaps people get stressed in interviews? Like I said, "Of course I recognize that some people may have been severely impacted by the technical interview process to the point that they were intellectually paralyzed." > I'm not sure I could remember off the top of my head if the function signature That would be a terrible question. Of course I wouldn't ask that.

Fair enough, I missed that while skimming your comment, and if you agree that would be a terrible question, then I generally agree with your sentiment.

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#458

Post by g_p » Thu, Apr 22, 2021, 10:53 PM UTC

twic wrote: post #449

ska wrote: post #443

Earlier quoted context omitted.

> America is leaking again. Ok, from a terminology point of view that's fair - it isn't the same everywhere by any means. However, most if not all of the graduate programs I know if internationally have something roughly equivalent, whether they are comps or prelims or qualifying or whatever. The basic idea is that a department (and university, generally) has an interest in maintaining the quality of their programs,…

No UK PhD programme has exams like this, that i have heard of.

Some UK programmes differ in that there isn't the same formal concept of "candidacy" like you see in other places. The end-of-first-year review is often a written report, sometimes with a "mini-viva".

The common factor among all UK PhDs I know of is that there is a rigorous viva at the end, where the outcome for the sudent is not a foregone conclusion. Despite the shorter overall duration of the PhD (~3 to 4 years typically), the oral examination can (rightly) cover material far beyond the scope of your thesis - if you are an expert in your field, you will be able to have a knowledgeable and informed discussion as a peer with your external examiner, who will be a recognised expert in the field. I'm definitely a big believer in the importance of being able to have a well-informed discussion around the area of your work, and actually found the whole viva process very enjoyable and cordial - a nice chat about the wider field, my and the examiners' own previous work, some debate of the merits of different approaches, and then onto a run-through of the thesis, chapter-by-chapter, skipping any pages where there were no points for discussion or contention.

Unlike European vivas though, there's no family or friends, no champagne corks being popped mid-defense, and no foregone conclusion of the outcome. I've been at European vivas with the family of the candidate preparing the celebratory buffet at the back of the auditorium while the questioning continues!

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#459

Post by g_p » Thu, Apr 22, 2021, 10:59 PM UTC

ska wrote: post #452

twic wrote: post #449

Earlier quoted context omitted.

No UK PhD programme has exams like this, that i have heard of.

Oh, good point, the UK programs I know of are notably ligher in this regard (and PhD shorter) though they do generally require a 1st class honors (honours, i guess!) degree in subject, which includes "tripos" which is roughly equivalent. So in some ways a higher bar for undergrad matched with a lower bar for grad. At least that's the theory - I don't know if in practice it holds up; most of the grad students and late…

The difference I tend to see is that the outcome of the UK viva (oral) examination is far less of a foregone conclusion, and can still be quite traditional in that students are expected to be able to have a broad and well-informed discussion about their wider field and the context of their work. The thinking is that they will (if meritous of a PhD) have a certain level of expertise, and thus be able to have a discussion with their external examiner (a distinguished and recognised expert) as a peer.

My experience of it was that if you are genuinely knowledgeable and approaching being an expert in your field, it is an enjoyable experience, and just like having a (longer than usual, but not uncomfortably long) conversation with someone about a topic that you both share a deep interest in. There is nothing to worry about, as you can have a nice discussion about an interesting topic, and share interesting ideas etc.

I'm not sure if it's a lower bar for grad as such - I think it's got fewer "formal" requirements, and far more informal requirements. The most common way to "fail" is to simply not submit the thesis.

Re: EFF sues Proctorio on behalf of student falsely DMCA'd

#460

Post by davrosthedalek » Thu, Apr 22, 2021, 10:59 PM UTC

raegis wrote: post #450

davrosthedalek wrote: post #414

Earlier quoted context omitted.

One of my colleagues had an online exam. The exam was uploaded to chegg within minutes, by multiple people. Unfortunately, the tests were personalized, so the people could be identified. I think cheating is quite common in the undergrad years, but gets less later.

How did you get access to Chegg? I thought of buying a subscription to check for cheaters, but there were complaints on the net about Chegg charging credit cards on canceled accounts.

I didn't, my colleague did. Not sure how. He asked them to took down the tests, and they did. Withing a couple of days though. Answers to questions are up within minutes!