Documentation Index
Fetch the complete documentation index at: https://wb-21fd5541-john-wbdocs-2044-rename-serverless-products.mintlify.app/llms.txt
Use this file to discover all available pages before exploring further.
weave / Evaluation
Sets up an evaluation which includes a set of scorers and a dataset.
Calling evaluation.evaluate(model) will pass in rows form a dataset into a model matching
the names of the columns of the dataset to the argument names in model.predict.
Then it will call all of the scorers and save the results in weave.
Example
// Collect your examples into a dataset
const dataset = new weave.Dataset({
id: 'my-dataset',
rows: [
{ question: 'What is the capital of France?', expected: 'Paris' },
{ question: 'Who wrote "To Kill a Mockingbird"?', expected: 'Harper Lee' },
{ question: 'What is the square root of 64?', expected: '8' },
],
});
// Define any custom scoring function
const scoringFunction = weave.op(function isEqual({ modelOutput, datasetRow }) {
return modelOutput == datasetRow.expected;
});
// Define the function to evaluate
const model = weave.op(async function alwaysParisModel({ question }) {
return 'Paris';
});
// Start evaluating
const evaluation = new weave.Evaluation({
id: 'my-evaluation',
dataset: dataset,
scorers: [scoringFunction],
});
const results = await evaluation.evaluate({ model });
Type parameters
| Name | Type |
|---|
R | extends DatasetRow |
E | extends DatasetRow |
M | M |
Hierarchy
Table of contents
Constructors
Properties
Accessors
Methods
Constructors
constructor
• new Evaluation<R, E, M>(parameters): Evaluation<R, E, M>
Type parameters
| Name | Type |
|---|
R | extends DatasetRow |
E | extends DatasetRow |
M | M |
Parameters
| Name | Type |
|---|
parameters | EvaluationParameters<R, E, M> |
Returns
Evaluation<R, E, M>
Overrides
WeaveObject.constructor
Defined in
evaluation.ts:148
Properties
__savedRef
• Optional __savedRef: ObjectRef | Promise<ObjectRef>
Inherited from
WeaveObject.__savedRef
Defined in
weaveObject.ts:73
Accessors
description
• get description(): undefined | string
Returns
undefined | string
Inherited from
WeaveObject.description
Defined in
weaveObject.ts:100
name
• get name(): string
Returns
string
Inherited from
WeaveObject.name
Defined in
weaveObject.ts:96
Methods
evaluate
▸ evaluate(«destructured»): Promise<Record<string, any>>
Parameters
| Name | Type | Default value |
|---|
«destructured» | Object | undefined |
› maxConcurrency? | number | 5 |
› model | WeaveCallable<(…args: [{ datasetRow: R }]) => Promise<M>> | undefined |
› nTrials? | number | 1 |
Returns
Promise<Record<string, any>>
Defined in
evaluation.ts:163
predictAndScore
▸ predictAndScore(«destructured»): Promise<{ model_latency: number = modelLatency; model_output: any = modelOutput; model_success: boolean = !modelError; scores: { [key: string]: any; } }>
Parameters
| Name | Type |
|---|
«destructured» | Object |
› columnMapping? | ColumnMapping<R, E> |
› example | R |
› model | WeaveCallable<(…args: [{ datasetRow: E }]) => Promise<M>> |
Returns
Promise<{ model_latency: number = modelLatency; model_output: any = modelOutput; model_success: boolean = !modelError; scores: { [key: string]: any; } }>
Defined in
evaluation.ts:231
saveAttrs
▸ saveAttrs(): Object
Returns
Object
Inherited from
WeaveObject.saveAttrs
Defined in
weaveObject.ts:77
