Bash tool

[gladyshcodes]

Issue #226

This PR introduces a Bash tool that enables using Shortest for API testing

[vercel]

@gladyshcodes is attempting to deploy a commit to the Antiwork Team on Vercel.

A member of the Team first needs to authorize it.

[gladyshcodes]

A couple of things:

[gladyshcodes]

Page extraction. To extract data such as bearers, I propose three solutions

1. Add extract method, providing it with schema of data we want to extract

const apiKey = await shortest.extract({
  instruction: "extract API key“,
  schema: z.object({
    username: z.string(),
    url: z.string(),
  }),
});

2. Leverage shortest directly with an ExtractSchema object, keeps things simple:

const apiKey = await shortest(“extract API key”, new ExtractSchema{
  schema: z.object({
    username: z.string(),
    url: z.string(),
  }),
});

3. Third, use FormData to create structured requests

const keys = new FormData(); // uses shortest behind the hood
keys.set('bearer', ‘extract bearer token);

This way we are embedding values directly in the test file memory rather than relying on conversational context

Why:
-Reduced dependence on AI Memory
-Streamlined code, allows us to access vars in a native way (instead of using <>)
-Potentially easier to debug
-More scalable (e.g what if we need to save those values somewhere in the future?)

@slavingia @m2rads

[gladyshcodes]

@slavingia Test cases with API requests that we would like to fully migrate to Shortest, in my opinion, have one drawback: API queries like that are super inconsistent and unstructured

In this PR, I propose an ApiRequest class (inspired by Playwright's API design). This class standardizes the format of requests to LLMs by aligning them with the RequestInit structure from fetchAPI

Why:
-API request queries are inconsistent and unstructured
-Inconsistency potentially leads to decrease in precision. Since RequestInit is the most widely used format, there's a good chance LLM will be more precise when working with it
-Lack of structure makes tool less readable and harder to use

p.s This does not impact flexibility, as you can still create API test without using it. Also, hybrid approach is possible (first prompt argument will be in priority when assembling final command):

const req = new APIRequest({
  baseURL: "https://66ae07d8b18f3614e3b69165.mockapi.io/api",
  extraHTTPHeaders: {
    "Content-Type": "application/json",
  },
  timeout: 50000,
});

shortest(
  "Ensure the API request is successful and returns 50 user objects. **Also, turn off cache control**",
  req.fetch(
    {
      method: "GET",
      headers: {
        "Content-Type": "application/json",
        Authorization: "<Bearer API KEY>",
      },
    },
    {
      maxRetries: 3,
    }
  )
);

[gladyshcodes]

Had a chance to refactor packages/shortest/src/ai/client.ts, now it's far more readable

[gladyshcodes]

I am unable to test the functionality in the final test file of #226. I'm encountering this issue when trying to create an organization at Iffy:
Email: [email protected]

@slavingia Could you please assist me with that?

[slavingia]

It may be good to add an endpoint within shortest and test for that so we can have it as part of this repo's tests/CI?

[gladyshcodes]

It may be good to add an endpoint within shortest and test for that so we can have it as part of this repo's tests/CI

Done!

[gladyshcodes]

So this thing is R4R

Please take a look @slavingia @m2rads

[slavingia]

Minor comments. We should update the README.

[gladyshcodes]

@slavingia README updated

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
shortest	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 2, 2025 5:56pm

[slavingia]

Would be good to have an example within this repo itself, somewhere in here: https://github.com/anti-work/shortest/tree/main/app/api

Of a test. I'd also think we'd want to take advantage of the "after" async page block thingy so we can verify the output more specifically, than just "valid users" – how does the QA AI/person know that the specific users were listed appropriately, and none that should be hidden, aren't shown (e.g. the app allows one to have a deleted user, that may still show up)?

[gladyshcodes]

Would be good to have an example within this repo itself, somewhere in here: https://github.com/anti-work/shortest/tree/main/app/api

Not sure what you mean by 'putting examples into the repo itself'. We now have three more working tests in /examples folder:

examples/api-failure.test.ts
examples/api-assert-bearer.test.ts
examples/api-assert-bearer-2.test.ts

Have you seen them?
You mean you want those tests to be near to test API endpoint they test (route.test.ts)?

README example is made up, we do not have such test in our codebase

[slavingia]

My bad, I missed those.

[m2rads]

Looks solid. Let's have these tests in the api/test/assert-bearer token so that we can have them run in the CI.