Getting Started

Prompting and Troubleshooting Guide

Workflow Prompting Guide

Choose the Best Block Type

  • Most deterministic: Action block
    • single action like click, input text, etc
  • Happy medium: Navigation block
    • single goal, like your goal is to fill out the form
  • Most flexible: Navigation V2 block
    • multi-goal, like your goal is to login, then fill the form, etc.

Crafting a Prompt

Anatomy

  1. [Required] Main goal
  2. Guardrails/details
  3. Payload, or information that Skyvern uses in input fields
  4. [Required] Completion/Termination criteria

Dos

  • Be very clear about your main goal and completion criteria. A good rule of thumb is if someone were to read it with no visual context, would they be able to understand?
    • Use key words like “complete” or “terminate” to let Skyvern know when the task is done or should be ended, respectively
  • Start general and get more specific with testing.
    • Making the workflow more general first means there’s a higher chance of it being generalizable across different websites
    • Use parameters in goal fields to add strings of texts that can make the workflow adjustable across different unique sites + their quirks
  • Use clear visual indicators and phrases to capture the main goal and completion criteria.
    • For instance, if your goal is to navigate to a certain page, you can say “using the sales dropdown at the top of the page” or “to navigate to invoices, the button has a receipt icon”

Don’ts

  • Be unclear. Make sure your goals are clear, that details are rooted in visuals. If you’re confused or a random person would be confused, the LLM will definitely be confused
  • Provide too little information. If you’re logging in, you need to provide login credentials, a URL to go to, etc. Think, “what would I need if I were starting this task from scratch?” You can always overprovide information and Skyvern will use what it needs
  • Make an action list unless absolutely necessary. If needed as extra details, make sure it’s accompanied by a main goal
  • Name the HTML element for items. Some HTML element items are accessible, but not all. Wouldn’t recommend this as a first course of action!

Examples

Example Navigation/Task Block prompt:

Your goal is to fill out the form. Only fill out required fields that you have information for. To add address information, you will need to select the Add button and input the information in a popup modal.

Here is the information you need to get through the form:

{{complete_payload}}

Your goal is complete when you have filled out the form and submitted. You will know your goal is complete when you are viewing a confirmation message that it was submitted.

Example Navigation V2 Block prompt:

Your first goal is to search for an image using Google Lens. To do so, select the Google Lens camera icon. You will know you’re looking at Google Lens when you see the option to drag and drop an image. Then search using the provided image link: {{image_URL}}

Now that you’ve searched google for the image, your goal is to identify the location. If you’re unsure about the location, default to the location that is most represented in the search results.

Now that you have the location, your goal is to go to google maps and extract the address. Provide the address in the following format: {{schema}}

Example Action Block prompt:

Click the continue button

Workflow Troubleshooting Guide

Process

Sometimes we test things and they run perfectly. Huzzah! Other times, issues crop up. The process of overcoming these issues by identifying errors is troubleshooting

There are a two main reasons why a workflow might fail

  • prompt issues/configuration issues
  • bugs

When you get a Failure or Termination

When you need to make a prompt change

Common signs that indicate a prompt change

  • Completed too soon: the task wasn’t finished, but the agent thought it was finished and moved on to the next block too early
  • Didn’t complete soon enough: the task finished but the agent didn’t realize it was finished and kept taking actions on that block
  • Payload issue: The information provided may be incorrectly structured or incomplete. This could mean:
    • The format needs to change: JSON parameter, string parameter, breaking a parameter into smaller parameters
    • Content is missing or invalid: check the payload to see if the parameter is missing or mislabeled
    • Parameters might not be configured properly: does each parameter have a double bracket {{sample_parameter}}? Does the label in the bracket match the label in the parameters tab?

When you’re unsure

What do you do when you’re unsure:

  1. Identify where the problem is happening
    • Check the screenshots with every step: is everything happening as you expect it to?
    • Check the recording: is there something missing in the screenshots that can explain why it failed?
  2. Identify the problem
    • Check the parameters of the problem task: Are the inputs what you expected them to be? Do they match what’s in the workflow?
    • Check diagnostics for the problem task: Is the navigation goal correct? Were elements mapped properly? Do they “exist” according to the bot? What actions in the action list did it say it took?

When you need to file a bug

  • Usually, you’ve tried making multiple prompt changes and the error persists
  • Sometimes it’s clear from the gate that there is a bug
    • Issues with common bugs like dropdowns, like checkboxes, etc.
    • Action lists referencing the wrong element or bounding boxes not mapped on properly

When you get a Complete

If it ain’t broke don’t fix it…

  • First, celebrate
  • Check reliability: run again and again, are any errors popping up on subsequent runs? Add guardrails, change payloads, refine completion criteria
  • Check accuracy: is the content correct? Improve prompting and add validation blocks to check the content

…or maybe fix it

  • If you have a content-sensitive workflow, plan for edge cases ahead of time
    • What happens if I end up on a different screen? Set termination criteria preemptively
    • Add validation blocks to check a review page and make sure the content matches. Make sure to pass in the payload to check!
  • In ForLoops, it is especially important to add guardrails since each block builds on the last
    • For instance, if you are clicking into each item in a table, consider passing in termination criteria that stops the task in the case you’re not looking at the correct page. Pro Tip: set continue on failure and a first block so it continues even if it messes up one!