Skip to main content

Hello Vecto!

This tutorial covers the basic usage of the Vecto AI client for indexing. It includes examples for both text and image data. You can follow along the tutorial or refer to the tutorial repository we've setup for you.

Setting Up a Node.js Environment with TypeScript for Vecto Application​

This guide assumes that you do not have Node.js installed on your machine. If you already have Node.js installed, you can skip the installation steps.

1. Install Node.js​

First, you need to install Node.js. Download it from the Node.js official website. It is recommended to download the LTS (Long Term Support) version.

2. Create a Working Directory​

Create a new directory on your system where you'll be developing your Vecto application. For example, let's call it hello_vecto_app.

3. Initialize Your Node.js Project​

Open your system's terminal or command prompt, navigate to the hello_vecto_app directory, and run the following command:

npm init -y

This command will create a package.json file in your directory, which is used to manage the project dependencies.

4. Install TypeScript​

To add TypeScript to your project, run the following command:

npm install typescript --save-dev

This command installs TypeScript as a development dependency in your project.

5. Create a TypeScript Configuration File​

Create a tsconfig.json file in your project root to configure TypeScript options. You can generate a default tsconfig.json file with this command:

npx tsc --init

6. Install Required Packages​

Finally, install the necessary Node.js packages. In the hello_vecto_app directory, run the following commands:

npm install @xpressai/vecto-client

If you've followed up to this point, you should have successfully set up your workspace for your Vecto application!

Creating a Vector Space and Usage Level Token​

To start ingesting data to a vector space, you will need a new vector space and vector space token.

  1. Launch the Vecto login page at Vecto Login. Enter your Username and Password then proceed by clicking Sign In.

  2. Next, click on Vector Spaces in the menu bar and select the New Vector Space option. Give it a name, for example ts-ingest-tutorial. Next, we get to select a vectorization model. Given our intent to work with both images and text, the CLIP model is an ideal choice. Wrap it up by clicking the Create Vector Space button. To view the specifics of your Vector Space, simply click on its name in the Vector Spaces list. Remember to jot down your Vector Space ID; we'll be needing it soon.

  3. To interact with our vector space, we need a unique Vector Space authentication token. Start by clicking on your username to expose the Tokens tab. Give it the token name as ts-ingest-tutorial-token. For our initial activities with this vector space, a USAGE access token will suffice. It grants us read-write privileges for our specific Vector Space. Having selected the ts-ingest-tutorial Vector Space we previously crafted, proceed by clicking Create User Token.

Remember, the token will only be displayed once, so keep it safe! We'll need it for the upcoming steps.

As always, it is important to keep your token safe. A common practice is to set the token in an .env file or export it as a variable.

Setup​

Let's start creating the application. Create a .tsx TypeScript file that will contain the ingest script. Let's call it ingest.tsx. Inside, add the following code:

import { Configuration } from '@xpressai/vecto-client';

const config = new Configuration({
    accessToken: 'your-vecto-token'
});

This will configure your Vecto client with the access token.

Data Ingesting​

To index text data into your vector space, update the previous snippet with the following section:

import {
    Configuration,
    IndexApi,
    IndexDataRequest,
  } from '@xpressai/vecto-client';
  
const config = new Configuration({
    accessToken: '', // Replace with your actual token
});

async function indexTextData() {
    const indexApi = new IndexApi(config);
    const textBlob = new Blob(['Hello Vecto']);

    const textDataParams: IndexDataRequest = {
        vectorSpaceId: '', // Replace with your actual vector space id
        modality: 'TEXT',
        attributes: [JSON.stringify('sample metadata')],
        input: [textBlob],
    };

    try {
        const result = await indexApi.indexData(textDataParams);
        console.log('Text data indexed successfully:', result);
    } catch (error) {
        console.error('Error indexing data:', error);
    }
}

indexTextData();

Finally, compile your TypeScript:

npx tsc

Then execute the script.

node ingest.js

You should see the IDs of the items you've just ingested in your console!

{ "status": "OK", "ids": [ 1 ] }

Congratulations, have successfully vectorized and ingested your first data into Vecto! You can view the ingested data attributes by going to app.vecto.ai and to your vector space.

Indexing image data is very similar, all you need to do is prepare your image data into a blob. Here's a modified code snippet of how to do that:

import fs from 'fs';

async function indexImageData() {
  const indexApi = new IndexApi(config);
  const fileContent = fs.readFileSync('bread.png');
  const imageBlob = new Blob([fileContent]);

  const ImageDataParams: IndexDataRequest = {
    vectorSpaceId: '', // Replace with your actual vector space id
    modality: 'IMAGE',
    attributes: [JSON.stringify('Bread: A staple food made from flour and water.')],
    input: [imageBlob],
  };

  try {
    const result = await indexApi.indexData(ImageDataParams);
    console.log('Image data indexed successfully:', result);
  } catch (error) {
    console.error('Error indexing data:', error);
  }
}

indexImageData();

You should get a similar message as the text ingest.

Key Points to Remember​

  • The data to be ingested should be in the form of a Blob.
  • Once ingested, the data is not viewable, hence the need of attributes. Attribute can be understood as the metadata associated with that data. You may append as much metadata as needed to the attribute. Don't forget to stringify it before ingestion.

Batching Data Ingest​

You can also ingest multiple data at the time. To do that, simply format your data to be an array of data Blobs and stringified attributes. In the following example, we're ingesting common food and their description.

import {
    Configuration,
    IndexApi,
    IndexDataRequest,
  } from '@xpressai/vecto-client';

const config = new Configuration({
    accessToken: '', // Replace with your actual token
});

async function indexTextData() {
    const indexApi = new IndexApi(config);

    const foodItems = [
        { name: 'Bread', description: 'A staple food made from flour and water.' },
        { name: 'Apple', description: 'A sweet, edible fruit.' },
        { name: 'Cheese', description: 'A dairy product derived from milk.' },
        { name: 'Tomato', description: 'A red or yellowish fruit with a juicy pulp.' },
        { name: 'Chicken', description: 'A domestic fowl kept for its eggs or meat.' },
        { name: 'Rice', description: 'A cereal grain, staple food in many parts of the world.' },
        { name: 'Pasta', description: 'A type of Italian noodle dish.' },
        { name: 'Fish', description: 'An aquatic animal, important source of protein.' },
        { name: 'Banana', description: 'A long curved fruit with a yellow skin.' },
        { name: 'Potato', description: 'A starchy plant tuber, a staple food.' }
    ];

    // Create arrays for blobs and attributes
    const blobs = foodItems.map(item => new Blob([item.name]));
    const attributes = foodItems.map(item => JSON.stringify({ name: item.name, description: item.description }));

    const textDataParams: IndexDataRequest = {
        vectorSpaceId: '', // Replace with your actual vector space id
        modality: 'TEXT',
        attributes: attributes,
        input: blobs,
    };

    try {
        const result = await indexApi.indexData(textDataParams);
        console.log('Food items indexed successfully:', result);
    } catch (error) {
        console.error('Error indexing food items:', error);
    }
}

indexTextData();

On completion, you will see multiple ids ingested.

{
  status: 'OK',
  ids: [
    340, 341, 342, 343,
    344, 345, 346, 347,
    348, 349
  ]
}

Data Lookup​

Once ingested, you can perform Vector Search lookup on them. Let's create a new file and name it lookup.tsx.

import {
  Configuration,
  LookupApi,
  LookupRequest,
} from '@xpressai/vecto-client';

const config = new Configuration({
    accessToken: '', // Replace with your actual token
});

async function lookupTextData() {
  const lookupApi = new LookupApi(config);

  const textParams: LookupRequest = {
    vectorSpaceId: '', // Replace with your actual vector space id
    modality: 'TEXT',
    topK: 3, // How many search results you would like to have 
    query: 'text query',
  };

  try {
    const results = await lookupApi.lookup(textParams);
    console.log("Text lookup results: ", JSON.stringify(results, null, 2));
  } catch (error) {
    console.error('Error lookup data:', error);
  }
}

lookupTextData();

You should get the an output similar to this:

❯ node dist/lookup.js
{
  results: [
    {
      id: 11,
      similarity: 0.8623678684234619,
      attributes: 'sample metadata'
    },
    {
      id: 9,
      similarity: 0.8623678684234619,
      attributes: 'sample metadata'
    },
    {
      id: 10,
      similarity: 0.8623678684234619,
      attributes: 'sample metadata'
    }
  ]
}

If you've selected a model that supports the IMAGE modality, you can also perform image search! Here's an example of how you can do that:

import {
  Configuration,
  LookupApi,
  LookupRequest,
} from '@xpressai/vecto-client';

const config = new Configuration({
    accessToken: '', // Replace with your actual token
});

import fs from 'fs';

async function lookupImageData() {
  const lookupApi = new LookupApi(config);
  const fileContent = fs.readFileSync('bread.png');
  const imageBlob = new Blob([fileContent]);
  
  const ImageParams: LookupRequest = {
    vectorSpaceId: '', // Replace with your actual vector space id
    modality: 'IMAGE',
    topK: 3,
    query: imageBlob,
  };

  try {
    const results = await lookupApi.lookup(ImageParams);
    console.log("Image lookup results: ", JSON.stringify(results, null, 2));
  } catch (error) {
    console.error('Error lookup data:', error);
  }
}

lookupImageData();

And that's it! You have successfully created an application capable of ingesting data and performing vector search.