Package com.hazelcast.vector.jet

Class VectorSources

java.lang.Object

com.hazelcast.vector.jet.VectorSources

@Beta
public class VectorSources
extends Object

Sources for loading and processing standard vector data formats.

Since:

5.5

Method Summary

Modifier and Type	Method	Description
static com.hazelcast.jet.pipeline.BatchSource<Map.Entry<Integer,com.hazelcast.vector.VectorValues>>	fvecs(String directory, String glob)	Creates legacy file source in fvecs format.
static FvecsFileFormat	fvecsFormat()	Returns fvecs file format definition for use with Jet Unified File Connector (FileSources).
static com.hazelcast.jet.pipeline.BatchSource<Map.Entry<Integer,int[]>>	ivecs(String directory, String glob)	Creates legacy file source in ivecs format.
static IvecsFileFormat	ivecsFormat()	Returns ivecs file format definition for use with Jet Unified File Connector (FileSources).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

fvecs

@Nonnull
public static com.hazelcast.jet.pipeline.BatchSource<Map.Entry<Integer,com.hazelcast.vector.VectorValues>> fvecs(@Nonnull
 String directory,
 @Nonnull
 String glob)

Creates legacy file source in fvecs format. Provided for convenience, if simpler API is sufficient. Assumes that the files are not shared by the members.

Parameters:

directory - directory containing files to read

glob - mask for file names

Returns:

fvecs file source

See Also:

• fvecsFormat()

ivecs

@Nonnull
public static com.hazelcast.jet.pipeline.BatchSource<Map.Entry<Integer,int[]>> ivecs(@Nonnull
 String directory,
 @Nonnull
 String glob)

Creates legacy file source in ivecs format. Provided for convenience, if simpler API is sufficient. Assumes that the files are not shared by the members.

Parameters:

directory - directory containing files to read

glob - mask for file names

Returns:

ivecs file source

See Also:

• ivecsFormat()

fvecsFormat

public static FvecsFileFormat fvecsFormat()

Returns fvecs file format definition for use with Jet Unified File Connector (FileSources). Example usage - load vectors to vector collection where key and metadata is vector index in the fvecs file:

 Pipeline p = Pipeline.create();
 p.readFrom(FileSources.files("/path/to/directory")
                         .glob("*.fvecs")
                         .format(VectorSources.fvecsFormat())
                         .build())
   .writeTo(VectorSinks.vectorCollection("collection-name", Map.Entry::getKey, Map.Entry::getKey, Map.Entry::getValue));
 

Returns:

fvecs format definition for use with Jet Unified File Connector

See Also:

• FileSources.files(java.lang.String)

ivecsFormat

public static IvecsFileFormat ivecsFormat()

Returns ivecs file format definition for use with Jet Unified File Connector (FileSources).

Returns:

ivecs format definition for use with Jet Unified File Connector

See Also:

• FileSources.files(java.lang.String)