Our development environment incorporates a chat-based AI coding assistant known as Cody, powered by popular state-of-the-art LLM models.
Cody serves two primary purposes:
- Facilitating AI assistance within our IDE, simulating access to a modern development tool while upholding anti-plagiarism and cheating standards.
- Providing the opportunity to assess prompt-writing, output validation, and other AI collaboration skills.
Cody plays a crucial role in helping candidates understand the task at hand. For instance, candidates can seek clarification on task requirements, inquire about specific details related to elements of the task, or request examples illustrating task requirements.
Candidates also have the option to ask Cody for publicly available information, such as references from documentation (e.g., "Demonstrate how a Lambda function works in Python").
Clients have the ability to review the complete transcript of the candidate's interaction with Cody, enabling hiring teams to evaluate the candidate's prompt-writing skills and proficiency in utilizing AI for development. The assessment of prompt-writing skills holds significant importance for chat-based generative AI tools.
We believe in an AI-assisted future, where engineers do their job in collaboration with AI assistants. Today’s examples are CoPilot and ChatGPT.
More realistic assessment - Cody within the candidate assessment makes the evaluation more realistic. In most real-life cases, engineers would do their job with AI assistant(s) available.
Insights on Engineer-AI interaction - The transcript of a candidate and Cody’s conversation allows the hiring manager to gain more insights on whether the candidate asks reasonable questions (i.e. knows the field) or attempts to get a complete solution from the AI.
AI tailored for assessment - Cody is specifically trained to bounce back a candidate’s requests to solve the solution completely. Even if a candidate tries to trick Cody through various known methods, it will not solve the full problem for the candidate.
Cody can be used within both CodeCheck and CodeLive:
- CodeCheck Test - Create a new test and under Test Settings step, toggle AI Assistant switch.
This can also be done in Settings after the test is created prior to sending invites to candidates.
- CodeLive Interview - Create an interview. Interviewer will need to toggle on AI Assistant in the interview settings:
During CodeLive sessions, tasks will feature the AI Assistant button. Once the interview begins, candidates can utilize the assistant, while the interviewer has access to a read-only version of the conversation.
If you have any additional questions about Cody or would like to leave your feedback, please reach out to your Customer Success Manager or contact us at firstname.lastname@example.org.