Tips, Tricks and Advice for Reviewing

[Dr_Jeff (Guennadi Maximov C)]

Tips, Tricks and Advice for Reviewing

THIS LIST IS EVER-CHANGING, SO I RECOMMEND YOU TO BOOKMARK THIS TOPIC AND ADD A CUSTOM REMINDER.

REMINDER: Questions for Tips are 100% valid!
If you have any questions regarding this document you can:

• Ask in the Curators Chat —which is integrated to Curator Lounge.
• Reach out through our Discord server.
• Send a DM to Staff.
• Emailing Staff at team@tosdr.org.

- Dr_Jeff, from the ToS;DR Staff

---

Table of Contents

• Introduction
• Reviewing Points
• Managing Services

---

Introduction

The following is a growing list of tips for Curators to keep in mind when reviewing on Phoenix.

You should revisit this topic every now and then, in case of changes!

---

Reviewing Points

Here are some useful tips for reviewing user Points:

You are not required to provide an explanation for declining points when they are...

• Points marked as “Quote not found” — The Phoenix UI for Points allow users to view the previous state of the Point in the Point history.

• Outdated and/or incorrect Points from user accounts that have been suspended and/or deleted — Those accounts will no longer respond to your feedback, so no need to provide it.

• Invalid DocBot Points — DocBot is not human, so feedback is not needed.

You are free to provide an explanation if you deem it necessary, though.

If you find duplicated pending Points...

Follow these steps in order:

1. Use your browser’s “Find and Search” utility to locate all Pending/Approved duplicates.

   • CTRL+f is the hotkey for Gecko-based browsers (Firefox, LibreWolf, etc.) and Chromium-based ones (Chromium, Google Chrome, Edge, Vivaldi, Opera, Brave, etc.).

2. Purge any invalid duplicate Points.

   • If there are no duplicates left after purging, you can skip the remaining steps.

3. Compare the remaining duplicates based on the quoted text.

4. Approve the Point that is more appropriate.

   • If there are leftover duplicates other than the one you selected, decline them all.

---

Managing Services

For managing a Service, take the following into account:

When can I mark a Service as "Comprehensively Reviewed"?

To do so, a Service MUST fit the following criteria:

• All the RELEVANT documents have been reviewed — Some Services have a lot of documents crawled; most of the text WILL NOT BE REVIEWED either because
  we don’t have a Case for certain quotes, or because a Point related the potential quote has been created.

• There shouldn’t be any Points pending for review — You COULD get away with at most TWO pending Points, but ALL OF THEM MUST BE “Neutral” ()!
  If there are pending Points with a rating of either “Good” , “Bad” or “Blocker” ,
  then DO NOT MARK AS COMPREHENSIVELY REVIEWED UNTIL THESE GET REVIEWED!

• All the relevant information across all documents has been covered — If there’s still relevant information that hasn’t been reviewed, review it first, then wait for them to be reviewed.
  Please make sure any Service you come across with has the fewest missing Points as possible!

---

NOTE: Work In Progress!

[Clippy]

@Dr_Jeff

3. Compare the remaining duplicates based on the quoted text.

4. Approve the Point that is more appropriate.

   • If there are leftover duplicates other than the one you selected, decline them all.

What if there are 2 docbot points, and they are both correct, does declining the extra points effect it’s training?

It kind of does, but should be negligible. @evan is aware of this (hopefully).

In that case the “better” Point is really up to your judgement.

[Clippy]

Sometimes though (quite rarely) there are 2 points by docbot (from 2 different documents) that have the same quote, I guess I just accept one and decline the other?

Exercise caution. If there’s, say one Point for a document called “Terms of Service” and another one called “Terms of Service for …”, the first one should take precedence.

If a point that usually belongs to ToS is found in the Privacy Policy, but there is a similar quote in the ToS, stick with the appropriate document.

---

TL;DR

Pick the Point linked to the most appropriate document.

[Dr_Jeff (Guennadi Maximov C)]

By the way @curators, you can edit the post if you have a piece of advice.
Before doing so, however, consult with your fellow Curators. Also notify me and/or other Staff members.